Datomic Knowledgebase - Recent questions and answers in Ions

Answered: 500 errors from api gateway for large Ion responses

Thu, 15 May 2025 05:04:46 +0000

For anyone getting opaque 500 errors for large responses through IONs in the future, in other words the problem that I was reporting here, in my case the problem was a response header was > 10k which exceeds this documented limit here...

https://docs.aws.amazon.com/general/latest/gr/apigateway.html#limits_apigateway

I hope this saves somebody some time.

Answered: http-direct configuation not working as per tutorials

Tue, 07 May 2024 19:51:47 +0000

Ugh! @phil Sorry that the screens have changed. I am going to need some time to reproduce what you are looking at to try to figure out what has changed and where you need to go. I think I probably have enough to go on here, but if there is any information you have not yet shared (perhaps because its not appropriate for this discourse) please feel free to forward into a support case at support@datomic.com.

Cheers,
Jaret

How to change the default 1 minute jetty async servlet timeout?

Thu, 10 Aug 2023 16:01:07 +0000

We have some ring routes in our HTTP direct ring handler, which returns some Excel files, which are constructed on the fly.

This handler sometimes runs for more than a minute, resulting in the following error:
HTTP ERROR 503 Async servlet timeout URI: /... STATUS: 503 MESSAGE: Async servlet timeout SERVLET: cognitect.http_endpoint.jetty.FnServlet-49bf6f8c Powered by Jetty:// 9.4.48.v20220622

How can we raise this timeout to ~5minutes, which is the maximum timeout of AWS NLBs?

(We are exposing the HTTP direct ion endpoint to the public internet via an AWS NLB, to avoid the 30s timeout and 10MB response payload limit of the AWS API Gateway)

Multiple sources on the internet - eg. https://github.com/ring-clojure/ring/issues/299 - mention a org.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT Java property which is supposed to affect Jetty's async context timeout.

Since the change of this property at runtime is not picked up by a running jetty instance, we have added provided it as a startup option, via the OverrideSettings CloudFormation template parameter:

`
{:OverrideSettings
(str "export JVM_FLAGS=\"$JVM_FLAGS"

  " -Dclojure.server.repl="
  "{:address,\\\"0.0.0.0\\\",:port,50505,:accept,clojure.core.server/repl}"
  " -Dorg.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT=300000"
  "\"")}

It is indeed observable within the ion JVM process:
(Long/getLong "org.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT") => 300000

Yet we are getting a 503 timeout response from the cognitect.http_endpoint.jetty.FnServlet-49bf6f8c servlet after 1 minute still.

Request, which take less then a minute work as expected.

Answered: Is there only one deployment target for ions?

Fri, 19 May 2023 13:35:34 +0000

Hi Phil!

When you create a primary compute group or query group, you associate it with a Code Deploy application. The :app-name in your ion-config.edn connects your code to a Code Deploy application and determines which compute groups you can deploy to.

If you are interested in deploying multiple Ions to the same "system" you could spin up a dedicated query group per app-name and deploy to those targets.

Cheers,
Jaret

Answered: Is there documentation for the AWS Permissions required for push and deploy ?

Tue, 04 Apr 2023 11:09:08 +0000

For now there is no such a thing, sadly. This is also why this feature request was issued a while ago.

For now you'll have to use a dedicated IAM role/user with the built-in Administrator and PowerUser policies (as per Access Control docs section).

Assigning just the Datomic Administrator Policy (datomic-admin-...) and datomic-code-... permissions will result in the following error upon Push:

{:message
 "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2YP8M87C0385R93H; S3 Extended Request ID: +dHQtObBUaq/T7BH7lWKNALPOCNULLd1uYy0PkXoVCeQ9A8hmilVo3KpNhtnGSOxbXB/NhPzQCI=; Proxy: null)",
 :class AmazonS3Exception}

That said, you are forced to leak access keys with a full DB administrator control to all your system developers and CI/CD operators, which is not that "least privileged" as one would probably like. Or I may be missing something here...

Answered: Support Multi-value parameters with Ions

Mon, 31 Oct 2022 22:44:55 +0000

I was just bitten by this too.

I reduced my problem to defining a [:http-direct :handler-fn], like this:

  (constantly
    {:status  200
     :headers {"x" ["1" "2"]}
     :body    ""})

the response I get is just

HTTP/2 500 
date: Mon, 31 Oct 2022 22:34:23 GMT
content-type: text/plain
content-length: 12
server: Jetty(9.4.44.v20210927)
apigw-requestid: a5EJajRaSQ0EMtQ=

Server Error

and the error in the datomic-{SystemName} log group is simply:

{
    "Msg": "http-endpoint respond failed: http://<custom-api-gw-domain>:8184/",
    "Ex": {
        "Via": [
            {
                "Type": "java.lang.ClassCastException"
            }
        ],
        "Trace": []
    },
    "Type": "Alert",
    "Tid": 18,
    "Timestamp": 1667255663263
}

without any stack trace, so it took awhile to narrow down to this minimal repro-case :(

I think it would be really helpful to mention this limitation in the documentation, at least.

Maybe adding a note about "Server Error" to https://docs.datomic.com/cloud/troubleshooting.html would help too.

Answered: Error on ion {:op :push} for redef of abs

Fri, 22 Apr 2022 22:22:03 +0000

I updated the ~/.clojure/deps.edn version and this seems to resolve this issue.

:ion-dev

{:deps {com.datomic/ion-dev {:mvn/version "1.0.306"}}
 :main-opts ["-m" "datomic.ion.dev"]}

from https://docs.datomic.com/cloud/operation/howto.html#ion-dev

Troubleshooting :deploy-status "FAILED", :code-deploy-status "SUCCEEDED"

Mon, 21 Feb 2022 13:33:43 +0000

After a few days of attempting to launch Datomic Ion on AWS following the Ion Tutorial [0], this appears to be the best I can do:

{:deploy-status "FAILED", :code-deploy-status "SUCCEEDED"}

Searching CloudWatch for “Alert - Alert” reveals no events.

I did not modify the ion-starter codebase except for configuring the correct :endpoint and :app-name. (I believe this was done correctly because I completed all steps in “Develop at the REPL”[1] without a hitch.)

[0]: https://docs.datomic.com/cloud/ions/ions-tutorial.html
[1]: https://docs.datomic.com/cloud/ions/ions-tutorial.html#repl

Thank you for any pointers.

Answered: Mismatch between expected release dependencies and "dependency conflicts"

Fri, 11 Feb 2022 20:07:52 +0000

Hi Jacob!

The cloud-deps.edn in ion-dev does not match the cloud-deps.edn that is actually running in your version of Datomic cloud. We understand this is a problem and we are working on it but you can ignore the dependency conflict map reported from a push.

Hope you are well!

How datomic.client.api/connect work?

Mon, 15 Nov 2021 14:46:44 +0000

I interested how connection work.

When connection is open? (all time or when we call him)
Can we hold connection opened all time ? Is this good or bad?
When we execute any transaction is it updated connection?

maybe the questions are simple, but I would like to better understand how it works.

Answered: Is it possible to use Leiningen when deploying with ions>

Thu, 30 Sep 2021 20:57:55 +0000

For ion deployment it is expected that your dependencies are managed in a deps.edn file.

Answered: How to properly configure access to IonApiGateway?

Fri, 23 Jul 2021 13:19:37 +0000

Can you provide details about what problem you are trying to solve. What do you mean by "properly configure access to IonApiGateway"?

Answered: Can I perform blocking IO during an Ion request?

Thu, 20 May 2021 01:04:11 +0000

Kind of opposite answer, pedestal ions do not support "async requests"

Once you can't return an async channel, the only other way to do IO is blocking IO

https://github.com/pedestal/pedestal.ions#caveats

http://pedestal.io/reference/request-map#_description

I agree with you that it should be explicit on docs

Answered: Ion deploy says Function not found

Fri, 05 Mar 2021 14:51:55 +0000

Finally I tested a hunch that if I just used different lambda names, I'd be in the situation where the system will no longer expect this particular lambda to exist so bad that it fails deploying my app. This hunch turned out to be true.

My old name still appears in each deployment's list of lambdas to delete, but since I no more use the name, my deployments won't fail because of it.

I still wonder, though, how I could make it so that the system understands once for all that this lambda no more exists.

Answered: How do I use datomic.ion/cast to log on my development machine?

Fri, 08 Jan 2021 16:17:23 +0000

You can redirect to a file

(cast/initialize-redirect cast-file)

or to stdout

(cast/initialize-redirect :stdout)

See https://docs.datomic.com/cloud/ions/ions-monitoring.html#local-workflow

Answered: Is there a way in Datomic Cloud Ions to know what git hash is being run?

Fri, 08 Jan 2021 16:12:54 +0000

It is in the description of the CodeDeploy deployment. You should be able to get it from the AWS CodeDeploy API.

How do I initialize core.async connections in Datomic Cloud?

Sat, 28 Nov 2020 23:53:01 +0000

I need to "hook up" a core.async channel to a thread on system initialization. How might I go about doing this, or doing any initialization tasks?

Thanks,
Hadil

Answered: Quote string values in result of Ions `{:op :deploy}`

Thu, 26 Nov 2020 20:37:35 +0000

Yup, ran into this today, killed my poor babashka script

How can I get a ring-like request in ionized web handlers?

Thu, 26 Nov 2020 19:04:57 +0000

Hello! After hearing the talks and reading the tutorial and documentation, I was under the impression that ions receive their requests in a ring-like format, and that we can "wire a web handler to a single function, using a routing library such as Compojure or pedestal."1

On one hand, this seems to be true with standard ion lambdas, but these are not exposed to the web. OTOH if we want to expose our web service, we need to write a web handler that we'll ionize. When we do that, the requests coming in lose their ring-like shape, and only expose the following keys:

 :headers
 :server-name
 :body
 :datomic.ion.edn.api-gateway/json
 :datomic.ion.edn.api-gateway/data

With that, our router (reitit backed by ring) fails.

I have seen different examples on the web where some router is exposed behind a single web handler, the most promising of which being probably Replion (look at the code) (and he uses reitit and ring and seems to have more success than us) and Pedestal Ions Sample (but Pedestal works in a relaxed ring context).

So my question is: is there something obvious that I don't understand? Are there more working examples? Some people seem to write adapters to transform the requests to make them more ring-like. Are they doing the right thing, or are we all missing something?

For example, here's a very naive adapter:

(defn get-path [req]
  (let [s (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :stage] "stg")
        p (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :http :path] "/")]
    (get (str/split p (re-pattern (str "/" s))) 1 "/"))) ; get rid of stage leading path!

(defn req->ring-req [req]
  (assoc req
         :protocol (get-in req
                           [:datomic.ion.edn.api-gateway/data :requestContext :http :protocol] "HTTP/1.1")
         :remote-addr (get-in req [:headers "host"] "example.com")
         :request-method (keyword
                          (str/lower-case
                           (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :http :method] "GET")))
         :uri (get-path req)))

(defn app
  "Web handler that returns the app's responses for each web request."
  [arg]
  (let [sta (start! arg)
        req (req->ring-req arg)
        ret ((handler/app) req)]
    ret))

Another related question is: what is the :integration key on lambda configurations? Why does "nobody" use it, except Stu's slack event handler? What does it do? We tried setting it and didn't seem to observe an impact.

Thanks!

Answered: How can I get rid of reflection warnings in cognitect/hmac_authn ?

Mon, 16 Nov 2020 20:14:38 +0000

Hi Matt,

First, I'd confirm you are on the latest clj tools deps:

brew upgrade clojure/tools/clojure

Then can you let me know what version you are on with:

clojure -Sdescribe

If you still see it on the latest, could I see your java version:

(System/getProperty "java.version")

Answered: Is there an example of using something like ring with ions http-direct?

Wed, 11 Nov 2020 17:23:25 +0000

I found https://github.com/pedestal/pedestal-ions-sample that might be worth looking into.

Bind Ions to IAM roles separate from their ambient compute role

Sun, 25 Oct 2020 19:58:49 +0000

Right now Ions run under the role of the underlying compute resources that they are deployed to. It would be nice to be able to give Ions a role independent of the ambient compute. This can be managed by the Ion runtime machinery by calling AssumeRole in the background, and supplying those creds to the Ions

This is important so that Ions don't have to run with the permissions of all the other ions running in the same compute.

Bind Ions to IAM roles separate from their ambient compute role

Thu, 22 Oct 2020 17:28:02 +0000

This is important so that Ions don't have to run with the permissions of all the other ions running in the same compute.

Allow users to redirect each type of Datomic.ion.cast message to different targets in local dev environment

Thu, 22 Oct 2020 17:06:23 +0000

Ability to direct each ion.cast type (Event, alert, dev, metric) to specific targets in local dev.