Datomic Knowledgebase - Recent questions without answers

Ion push failing with error message about (missing?, unreadable?) maven artifact ion-1.0.47.jar

Tue, 19 May 2026 19:40:21 +0000

Attempting to push an ion and get an error about not finding (or unable to read) ion-1.0.47.jar. I tried some troubleshooting and I don't think it is a permissions issue, because I am able to use the aws cli and copy ion-0.9.7.jar. So why is 1.0.47 not working?

deps.edn:

{:mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}

 :aliases {
           :ion-dev {:deps {com.datomic/ion-dev {:mvn/version "1.0.352"}
                            com.datomic/client-cloud {:mvn/version "1.0.131"}
                            com.datomic/ion {:mvn/version "1.0.71"}}
                     :main-opts ["-m" "datomic.ion.dev"]}}}

command I'm trying to run:

clojure -M:ion-dev '{:op :push :creds-profile "byt-wrought-test" :region "us-west-2"}'

error:

{:command-failed
 "{:op :push :creds-profile \"byt-wrought-test\" :region \"us-west-2\"}",
 :causes
 ({:message
   "Failed to read artifact descriptor for com.datomic:ion-http-direct:jar:1.0.47",
   :class ArtifactDescriptorException}
  {:message
   "Could not transfer artifact com.datomic:ion-http-direct:pom:1.0.47 from/to datomic-cloud (s3://datomic-releases-1fc2183a/maven/releases): Unexpected error downloading artifact from datomic-releases-1fc2183a",
   :class ArtifactResolutionException}
  {:message
   "Could not transfer artifact com.datomic:ion-http-direct:pom:1.0.47 from/to datomic-cloud (s3://datomic-releases-1fc2183a/maven/releases): Unexpected error downloading artifact from datomic-releases-1fc2183a",
   :class ArtifactTransferException}
  {:message
   "Unexpected error downloading artifact from datomic-releases-1fc2183a",
   :class ExceptionInfo,
   :data
   {:bucket "datomic-releases-1fc2183a",
    :path
    "maven/releases/com/datomic/ion-http-direct/1.0.47/ion-http-direct-1.0.47.pom",
    :reason :cognitect.anomalies/fault}})}

Based on some conversations I found on slack and datomic dev forum, I tried this as a troubleshooting step:

aws --profile byt-wrangle-test s3 cp s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/0.9.7/ion-0.9.7.jar .

download: s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/0.9.7/ion-0.9.7.jar to ./ion-0.9.7.jar

and that works. But the same attempt with 1.0.47 does not:

aws --profile byt-wrangle-test s3 cp s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/1.0.47/ion-1.0.47.jar .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Using a collection of datoms as input to a query

Thu, 24 Apr 2025 12:51:36 +0000

I was wondering if there is a way to use a collection of datoms obtained from d/tx-range + get a single tx or received from a listener that subscribed to a tx queue as input to d/q? The nature of this collection looks very similar to history db which can be used with d/q, however passing this collection as input does not produce the expected result. Neither there is any error.

Using Valcache via Docker bind mounts

Thu, 11 Apr 2024 09:50:54 +0000

I'm evaluating caching options for Datomic Pro. Valcache sounds like a great fit, but there is one thing that might be a showstopper.

The Prerequisites docs says that "Valcache relies on an SSD with the strictatime and lazytime flags set". This requirement I can meet pretty easily by remounting the disk on the hosts running the peers.

Now, the applications that are Datomic peers, who would be the users of these Valcaches, run as Docker containers in a Docker Swarm. The idea is to mount the Valcache as a Docker bind mount.

Would this work? The underlaying SSD disk would be mounted with strictatime and lazytime, but would whatever mechanism that requires these flags to be set work through a Docker bind mount?

Possibly security issue with the dev H2 storage?

Wed, 13 Mar 2024 21:58:26 +0000

While trying to understand more about the behaviour of Datomic dev (mainly from a security/encryption point of view), I attempted to connect to datomic.mv.db from an instance of the H2 console. I accidentally tried to log in as "admin" with a blank password and was surprised when I connected successfully. Connecting as "admin" with any password fails, so it seems explicitly to be blank. As far as I can tell I only have public schema access and can't see any transacted data, but I am able to run SQL queries in the console.

Clearly a dev transactor isn't intended to be high security, but admin with a blank password seems risky even for a dev environment. In most SQL environments I work with, the sa/admin account is just disabled if not used. Am I missing something?

As an aside, does the dev transactor use H2's AES encryption or is the resulting data file unencrypted?

How to change the default 1 minute jetty async servlet timeout?

Thu, 10 Aug 2023 16:01:07 +0000

We have some ring routes in our HTTP direct ring handler, which returns some Excel files, which are constructed on the fly.

This handler sometimes runs for more than a minute, resulting in the following error:
HTTP ERROR 503 Async servlet timeout URI: /... STATUS: 503 MESSAGE: Async servlet timeout SERVLET: cognitect.http_endpoint.jetty.FnServlet-49bf6f8c Powered by Jetty:// 9.4.48.v20220622

How can we raise this timeout to ~5minutes, which is the maximum timeout of AWS NLBs?

(We are exposing the HTTP direct ion endpoint to the public internet via an AWS NLB, to avoid the 30s timeout and 10MB response payload limit of the AWS API Gateway)

Multiple sources on the internet - eg. https://github.com/ring-clojure/ring/issues/299 - mention a org.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT Java property which is supposed to affect Jetty's async context timeout.

Since the change of this property at runtime is not picked up by a running jetty instance, we have added provided it as a startup option, via the OverrideSettings CloudFormation template parameter:

`
{:OverrideSettings
(str "export JVM_FLAGS=\"$JVM_FLAGS"

  " -Dclojure.server.repl="
  "{:address,\\\"0.0.0.0\\\",:port,50505,:accept,clojure.core.server/repl}"
  " -Dorg.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT=300000"
  "\"")}

It is indeed observable within the ion JVM process:
(Long/getLong "org.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT") => 300000

Yet we are getting a 503 timeout response from the cognitect.http_endpoint.jetty.FnServlet-49bf6f8c servlet after 1 minute still.

Request, which take less then a minute work as expected.

Is `(pull $db2 ?e [*])` supported officially?

Sat, 15 Jul 2023 04:42:45 +0000

The documentation only mentions the syntax without a src-var:

pull-expr = ['pull' variable pattern]

but found that this syntax also works:

pull-expr = ['pull' src-var variable pattern]

I have a self-contained code example here, which demonstrates a use-case, where an entity is split between 2 Datomic databases, for privacy reasons:

https://forum.datomic.com/t/pull-db-entity-pattern-is-undocumented/2299

Is this variant of the pull-expr in Datalog queries are not supported officially?

Should I use it in production or does it have some undefined behaviour, compared to its binary arity variant?

Or the documentation is just incomplete?

Datomic Cloud IAM role drift on setup - is this normal?

Wed, 12 Jul 2023 09:34:33 +0000

hi, just set up DC for the first time (ran the two AWS cloudformation templates). The storage stack is showing drift on the IAM managed policy datomic-admin-..., shows 'MODIFIED', in red. Is this normal upon setup? (checking the changes in the 'versions' tab, the difference is addition of autoscaling and s3 actions to the policy; the second version, presumably causing the 'drift', was added during the provisioning). Is this just a setup artifact nothing to worry about?

Request Entity Too Large - clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

Tue, 21 Mar 2023 12:26:20 +0000

There seems to be a bug in Datomic 990-9202 that breaks the d/tx-range function.

Earlier I could use the tx-range function the fetch data from my database.

But after upgrading to Datomic 990-9202 I get and exception when a transaction has too much data.

`
clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

cognitect.anomalies/category: :cognitect.anomalies/incorrect
 cognitect.anomalies/message: "Response body did not conform to Datomic client protocol"
                 http-result: {:status 413,
                               :headers
                               {"apigw-requestid" "Vg4UxisCDoEEJsQ=",
                                "server" "Jetty(9.4.44.v20210927)",
                                "connection" "keep-alive",
                                "content-length" "38",
                                "date" "Tue, 19 Jul 2022 12:17:47 GMT",
                                "content-type" "application/json"},
                               :body "{\"message\":\"Request Entity Too Large\"}"}

Is it possible for Datomic On-prem to get the storage credentials from Secrets Manager or similar software?

Thu, 02 Mar 2023 10:50:19 +0000

Is there a way to run the transactor service without storing the credentials to disk?

What's the state of the art in distributed Datomic?

Sun, 18 Dec 2022 00:37:33 +0000

I'd like to get an idea of the state of the art in distributed Datomic. FWIW, my wish list includes the ability for selected processing nodes to have local images and/or participate in federated global images, linked by CRDTs, etc. And a pony.

-r

Querying 2000+ entities, by passing in uuids (Really slow)

Wed, 14 Dec 2022 09:24:29 +0000

Passing in over 2000 uuids to mysql for a fairly simple query takes milliseconds to retrieve relevant data. However we're running into an issue with datomic where it takes noticeable longer. Is there a specific syntax to use when processing such long lists, or is the linked example the only way?

https://docs.datomic.com/on-prem/query/query.html#collection-binding

Inconsistency between d/pull and d/q + pull for non-existent entities

Sat, 10 Dec 2022 15:55:29 +0000

Inconsistency between d/pull and d/q + pull for non-existent entities

Datomic returns a map with the :db/id of the entity being pulled, even if the entity doesn't exist in the database. For example:

(d/pull (d/db conn) '[*] 9999999999)
=> #:db{:id 9999999999}

This behavior is somewhat confusing, especially when using d/pull with time filters. I was expecting d/pull to return nil or #:db{:id nil} if the entity wasn't created at the given point in time. However, Datomic always returns a map with the entity ID:

(d/pull (as-of (d/db conn) 1) '[*] 9999999999)
=> #:db{:id 9999999999}

Surprisingly, I see the expected behavior if I pull the entity information from a query using d/q:

(d/q '{:find [(pull ?e [*])]
        :in   [?e]}
      (d/db conn) 9999999999)
=> [[#:db{:id nil}]]

Is that difference in behavior expected? How can I determine if an entity existed at any point in time using d/pull?

How do you add relationship properties, e.g. the time a person met another person?

Mon, 07 Nov 2022 08:35:28 +0000

I have some experience with Neo4j / cypher. In cypher relationships are first-class citizens, so they can have properties, which I really like.

How do I do something similar with datalog? Do I have to have a "mapping entity" in between two entities that have a relationship?

Is using entid function to resolve lookup refs a supported behaviour?

Wed, 08 Jun 2022 13:30:29 +0000

I'm using datomic.api/entid to resolve a lookup ref to an entity ID, like
$ (entid db [:user/id "test@test.com"]) 17592186045432
This works fine, but a colleague pointed out this is not a documented behaviour.
>Usage: (entid db ident)
>Returns the entity id associated with a symbolic keyword, or the id
itself if passed.

Is this a gap in documentation, or is this behaviour accidental and not recommended to rely on?

java.lang.IllegalArgumentException: Cannot resolve key: 24a96e20-f526-4f7f-ba38-4f684caa5607

Mon, 06 Jun 2022 19:37:30 +0000

I am trying to implement resursive rule (loglcal OR) by following this video - Lucas Cavalcanti & Edward Wible - Exploring four hidden superpowers of Datomic

Here the code

(defn owns? [cid pid db]
(d/q '{:find [?pur]

     :in [$ ?cus-id ?pur %]
     :where
     [(owns? ?cus-id ?pur)]}
db cid [:purchase/id pid] owner-rules))

(comment
(owns?

#uuid "0fb7ea94-44af-46fa-98ca-0ddb5eb23123"
#uuid "24a96e20-f526-4f7f-ba38-4f684caa5607"
(d/db bank-conn)))

While running the above example I am getting this error:
java.lang.IllegalArgumentException Cannot resolve key: 24a96e20-f526-4f7f-ba38-4f684caa5607

Here is the full code:

(def purchase-schema
[{:db/ident :purchase/id

:db/doc "The purchase ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :purchase/account

:db/doc "Purchase Account"
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}])

(def account-schema
[{:db/ident :account/id

:db/doc "The Account ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :account/customer

:db/doc "Customer who own's this account"
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}])

(def customer-schema
[{:db/ident :customer/id

:db/doc "Customer ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :customer/name

:db/doc "Customer name"
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one}])

;;
;; Create client
(def client (d/client {:server-type :dev-local

                   :system "dev"}))

;; Create connections
(def bank-conn (d/connect client {:db-name "bank"}))

;; Transact schemas
(comment
(d/transact bank-conn {:tx-data purchase-schema})
(d/transact bank-conn {:tx-data account-schema})
(d/transact bank-conn {:tx-data customer-schema})
(d/tx-range bank-conn {:start 6}))

(comment
(java.util.UUID/randomUUID))

;; Transact fake data
(def cid #uuid "0fb7ea94-44af-46fa-98ca-0ddb5eb23123")
(def sample-customer {:customer/id cid

                  :customer/name "Jon Snow"})

(def aid #uuid "c579a9f6-f6c3-4718-8ecf-c3003021ee9a")
(def sample-account {:account/id aid

                 :account/customer [:customer/id cid]})

(def pid #uuid "24a96e20-f526-4f7f-ba38-4f684caa5607")
(def sample-purchase {:purchase/id pid

                  :purchase/account [:account/id aid]})

(comment
(d/transact bank-conn {:tx-data [sample-customer]})
(d/transact bank-conn {:tx-data [sample-account]})
(d/transact bank-conn {:tx-data [sample-purchase]}))

(comment
(d/q '[:find ?cus

     :in $ ?cid ?pid
     :where
     [?pur :purchase/id ?pid]
     [?pur :purchase/account ?acc]
     [?acc :account/customer ?cus]
     [?cus :customer/id ?cid]]
  (d/db bank-conn)
  (:customer/id sample-customer)
  (:purchase/id sample-purchase)))

;; owns v1
(defn ownsv1? [cid pid db]
(d/q '[:find ?cus

     :in $ ?customer-id ?purchase-id
     :where
     [?pur :purchase/id ?purchase-id]
     [?pur :purchase/account ?acc]
     [?acc :account/customer ?cus]
     [?cus :customer/id ?customer-id]]
db cid pid))

(comment
(ownsv1?

(:customer/id sample-customer)
(:purchase/id sample-purchase)
(d/db bank-conn)))

;; recursive rule (logical OR)
(def owner-rules
'[[(owns? ?cus-id ?e)

 [?e :customer/id ?cus-id]]
[(owns? ?cus-id ?e)
 [?e ?ref-attr ?r]
 (owns? ?cus-id ?r)]])

(defn owns? [cid pid db]
(d/q '{:find [?pur]

     :in [$ ?cus-id ?pur %]
     :where
     [(owns? ?cus-id ?pur)]}
db cid [:purchase/id pid] owner-rules))

;;
(comment
(owns?

(:customer/id sample-customer)
(:purchase/id sample-purchase)
(d/db bank-conn)))

Notifying clients of database changes in Datomic Cloud?

Sat, 04 Jun 2022 15:18:15 +0000

In your 2013 blog post The Transaction Report Queue, you write:

In Datomic, you can monitor all transactions. Any peer process in the
system can request a transaction report queue of every transaction
against a particular database.

Is there a change data capture or related mechanism in Datomic Cloud, where changes to the database can be pushed to a client?

Thank you,

Andrew

Troubleshooting :deploy-status "FAILED", :code-deploy-status "SUCCEEDED"

Mon, 21 Feb 2022 13:33:43 +0000

After a few days of attempting to launch Datomic Ion on AWS following the Ion Tutorial [0], this appears to be the best I can do:

{:deploy-status "FAILED", :code-deploy-status "SUCCEEDED"}

Searching CloudWatch for “Alert - Alert” reveals no events.

I did not modify the ion-starter codebase except for configuring the correct :endpoint and :app-name. (I believe this was done correctly because I completed all steps in “Develop at the REPL”[1] without a hitch.)

[0]: https://docs.datomic.com/cloud/ions/ions-tutorial.html
[1]: https://docs.datomic.com/cloud/ions/ions-tutorial.html#repl

Thank you for any pointers.

client api: clear cache when credentials change

Tue, 14 Dec 2021 08:30:00 +0000

I have my development program running and connect to datomic proper. How do I clear the caches because it brakes when I need new aws creds.

How datomic.client.api/connect work?

Mon, 15 Nov 2021 14:46:44 +0000

I interested how connection work.

When connection is open? (all time or when we call him)
Can we hold connection opened all time ? Is this good or bad?
When we execute any transaction is it updated connection?

maybe the questions are simple, but I would like to better understand how it works.

How to install transaction function using the [com.datomic/client-pro "0.9.63"]

Wed, 13 Oct 2021 16:27:12 +0000

How to install transaction function using the [com.datomic/client-pro "0.9.63"] ?
I am following this example https://docs.datomic.com/on-prem/reference/database-functions.html

However there is no d/function to install the transaction function... And looking at the documentation there is only datomic dev local and datomic client pro libraries

How can I create a transaction function using the client pro lib?

How to integrate library using datomic.client.api when using the peer library?

Fri, 03 Sep 2021 17:33:57 +0000

I am using the peer library for on-prem Datomic. Now I want to use a library - ragtime.datomic - that uses the datomic.client.api. How can I do that?

Based on https://docs.datomic.com/client-api/datomic.client.api.html#var-client it seems the only option is to run a separate peer process, exposed at a port and set up authorization between the two. But it seems wasteful to run a separate server when I already have the peer library. Is there a simpler way?

query with pull and more attributes count max min etc

Tue, 31 Aug 2021 20:42:03 +0000

When I try pull attributes together with max, min or count I received only 2 attributes.

I want the same output from Example 1, but instead a new query to :where, I want use (pull).

My problem its on example 2, I'm missing or a did something wrong?

Exemple 1, happens correctly:

(defn resume [db]

  (d/q '[:find ?name-cat (min ?price) (max ?price) (count ?price)
         :keys category min max qtd
         :with ?product
         :where [?product :product/price ?price]
         [?product :product/category ?category]
         [?category :category/nome ?name-cat]]

       db))

[{:category "Eletronics",

  :min 15M,

  :max 15M,

  :qtd 10}]

Exemplo2, using (pull)

(defn resume [db]

  (d/q '[:find (pull ?category [:category/nome])
         (min ?price)
         (max ?price)
         (count ?price)
         ;:keys category min max qtd ;cant use kays because Indexoutbound happen

        :with ?product ;using with because it is a conj, so now it allow if a product has same price
         :where [?product :product/price ?price]
         [?product :product/category ?category]]

       db))

output

[{:category/id 34 :category/name "test"} 10

Retracting entities, including tuples with refs

Tue, 31 Aug 2021 13:36:44 +0000

In my schema I have attributes of type tuple that has :db.type/ref as one of the tuple's types.
My usecase is maintaining a set of ordered to-many relations.

When I retract an entity using :db/retractEntity, the tuple attrs that refer to the retracted entity still remain, continuing to refer to a now non-existent entity.
I'm not sure whether this is the intended behaviour, and if so, what would be a standard approach to clean up these tuples? My intention would be for these tuples to be completely redacted. As I'm using this in a cardinality-many setup, I'd expect the attribute to retain all tuple values, except the one that does include the ref to the retracted entity in any position.
Should I be writing a custom retractEntity-like transaction function?
Thanks!

How to authenticate and authorize access to API Gateway endpoints for datomic?

Thu, 19 Aug 2021 17:05:29 +0000

I'm currently evaluating Datomic and I've deployed a production topology using a split stack. DatomicCloudVersion is 9095 and DatomicCFTVersion is 884. The CloudFormation stack created two API Gateway endpoints: datomic-[system]-client-api and datomic-[system]-ions.

From my local computer I can get a client by using datomic-[system]-client-api endpoint:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

Above, https://[endpoint-id].execute-api.us-east-1.amazonaws.com/ is the client-api API Gateway endpoint. It works, but the endpoint is open to the world. I then added an IAM authorizer to datomic-[system]-client-api and used an authorized IAM user to make a request to the now authenticated endpoint:

awscurl --service execute-api -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com --profile datomic
{:s3-auth-path "long s3 path"}

awscurl is just a tool to make authenticated calls to AWS services. As you can see, I can get a response from the endpoint by using my local profile named "datomic". If I try using regular curl without the v4 AWS signature it fails:

curl -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com
{"message":"Forbidden"}

Works as expected. The problem is now I can't get a client using:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :creds-profile "datomic"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

The response I get is:

{:status 403,
:headers
{"apigw-requestid" "EUsyNi9aIAMEPlg=",
"connection" "keep-alive",
"content-length" "23",
"date" "Thu, 19 Aug 2021 17:01:40 GMT",
"content-type" "application/json"},
:body "{\"message\":\"Forbidden\"}"}

It is as if the d/client call is ignoring the profile or not generating the proper signature. I tried setting the AWS_PROFILE environment variable as well, to no avail.

Is using an IAM Authorizer not the proper way to go? How can I protect both the client-api and ions endpoints? Finally, what's the difference between the client-api and ions endpoints?

It would be nice to have something like sync-excise to coordinate with background history removal

Mon, 26 Jul 2021 15:58:21 +0000

According to the docs: "The effect of :db/noHistory happens in the background, and some amount of history may be visible even for attributes with :db/noHistory set to true." It would be useful to have something like sync-excise to know when the db value is aware of all history removal of a specific (noHistory) attribute.

Datomic returns empty results where "Insufficient bindings, will cause db scan" exception is expected

Wed, 07 Jul 2021 14:30:05 +0000

Hello

If we change #uuid"..." to "" (from uuid to string), we get the exception (desired behavior)

clj -Sdeps '{:deps {com.datomic/dev-local {:mvn/version "0.9.232"}}}' -M \
  -e "(require '[datomic.client.api :as d])" \
  -e '(def client (d/client {:server-type :dev-local :system "hello"}))' \
  -e '(d/create-database client {:db-name "hello"})' \
  -e '(def conn (d/connect client {:db-name "hello"}))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
=> #'user/client
=> true
=> #'user/conn
=> []

Occour on (old) versions of datomic-free too

clj -Sdeps '{:deps {com.datomic/datomic-free {:mvn/version "0.9.5697"}}}' \
  -M --report stderr \
  -e "(require '[datomic.api :as d])" \
  -e '(d/create-database "datomic:mem://hello")' \
  -e '(def conn (d/connect "datomic:mem://hello"))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
WARNING: requiring-resolve already refers to: #'clojure.core/requiring-resolve in namespace: datomic.common, being replaced by: #'datomic.common/requiring-resolve
=> true
=> #'user/conn
=> #{}

ClassCastException, d/with inside transact function

Thu, 24 Jun 2021 14:57:51 +0000

Calling a transaction function that uses d/with in a dev-local, filesystem (i.e., not :storage-dir :mem) throws a ClassCastException on internal Datomic code.

Environment

com.datomic/dev-local {:mvn/version "0.9.232"}
client-cloud 0.8.105

History

Example with exception using file system dev-local.

(def client (d/client {:server-type :dev-local
                         :storage-dir "/tmp/cce-bug3"
                         :system      "test"}))
=> #'user/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'user/conn
(d/transact conn {:tx-data [{:db/ident       :user/name
                               :db/valueType   :db.type/string
                               :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "test",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "test",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T00:49:46.400-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(defn my-tx-fn
    [db argm]
    (let [tx-report (d/with db {:tx-data [{:user/name "asd"}]})]
      []))
=> #'user/my-tx-fn
(d/transact conn {:tx-data [(list 'user/my-tx-fn {})]})
Execution error (ClassCastException) at datomic.dev-local.tx/datom-lookup-valfn (tx.clj:397).
class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')

Working memdb example.

(def client-memdb (d/client {:server-type :dev-local
                               :storage-dir :mem
                               :system      "test"}))
=> #'user/client-memdb
(d/create-database client-memdb {:db-name "test"})
=> true
(def conn-memdb (d/connect client-memdb {:db-name "test"}))
=> #'user/conn-memdb
(d/transact conn-memdb {:tx-data [{:db/ident       :user/name
                                     :db/valueType   :db.type/string
                                     :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:02:57.965-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(d/transact conn-memdb {:tx-data [(list 'user/my-tx-fn {})]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 6,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 7,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533319 50 #inst"2021-05-09T01:03:08.042-00:00" 13194139533319 true]],
 :tempids {}}

Expectation

Transaction functions that use d/with do not throw a ClassCastException. Additionally, the behavior of d/with in all Datomic environments (Cloud, File system, Memdb) is consistent. Given a previous Datomic Cloud release fixed a problem with using with-db from an Ion, it seems using d/with in a transaction function is a supported behavior.

Actual

The final d/transact transaction function throws a ClassCastException when using d/with on some internal Datomic code. This only occurs when using a dev-local, file system environment.

Full stacktrace is pasted below.

#error{:cause "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
       :via [{:type java.lang.ClassCastException,
              :message "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
              :at [datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]}],
       :trace [[datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]
               [datomic.dev_local.tx$datom_lookup_valfn invoke "tx.clj" 397]
               [datomic.dev_local.local_log.LocalLog valAt "local_log.clj" 56]
               [clojure.lang.RT get "RT.java" 760]
               [datomic.dev_local.btindex.BTIndex cons "btindex.clj" 281]
               [clojure.lang.RT conj "RT.java" 677]
               [clojure.core$conj__5390 invokeStatic "core.clj" 85]
               [clojure.core$conj__5390 invoke "core.clj" 82]
               [datomic.core.db.Db addData "db.clj" 2322]
               [datomic.core.db$add_ensured_data invokeStatic "db.clj" 3353]
               [datomic.core.db$add_ensured_data invoke "db.clj" 3351]
               [datomic.core.db$with_tx invokeStatic "db.clj" 3370]
               [datomic.core.db$with_tx invoke "db.clj" 3357]
               [datomic.core.db.Db with "db.clj" 2164]
               [datomic.core.local_db$fn__25633 invokeStatic "local_db.clj" 67]
               [datomic.core.local_db$fn__25633 invoke "local_db.clj" 24]
               [datomic.client.api.protocols$fn__11959$G__11877__11966 invoke "protocols.clj" 126]
               [datomic.client.api$with invokeStatic "api.clj" 363]
               [datomic.client.api$with invoke "api.clj" 353]
               [user$my_tx_fn invokeStatic "user.clj" 3]
               [user$my_tx_fn invoke "user.clj" 20]
               [clojure.lang.AFn applyToHelper "AFn.java" 156]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.lang.Var applyTo "Var.java" 705]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$apply invoke "core.clj" 660]
               [datomic.core.db.ProcessExpander inject "db.clj" 3229]
               [datomic.core.db.ProcessInpoint inject "db.clj" 2950]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586$fn__18587 invoke "btindex_db.clj" 440]
               [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
               [clojure.core$reduce invokeStatic "core.clj" 6827]
               [clojure.core$reduce invoke "core.clj" 6810]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586 invoke "btindex_db.clj" 440]
               [datomic.dev_local.btindex_db$expand_tx invokeStatic "btindex_db.clj" 444]
               [datomic.dev_local.btindex_db$expand_tx invoke "btindex_db.clj" 429]
               [datomic.dev_local.btindex_db$with_tx invokeStatic "btindex_db.clj" 468]
               [datomic.dev_local.btindex_db$with_tx invoke "btindex_db.clj" 461]
               [datomic.dev_local.impl.DurableConnection transact "impl.clj" 161]
               [datomic.client.api$transact invokeStatic "api.clj" 200]
               [datomic.client.api$transact invoke "api.clj" 183]
               [user$eval124260 invokeStatic "user.clj" 25]
               [user$eval124260 invoke "user.clj" 25]
               [clojure.lang.Compiler eval "Compiler.java" 7177]
               [clojure.lang.Compiler eval "Compiler.java" 7132]
               [clojure.core$eval invokeStatic "core.clj" 3214]
               [clojure.core$eval invoke "core.clj" 3210]
               [nrepl.middleware.interruptible_eval$evaluate$fn__959 invoke "interruptible_eval.clj" 91]
               [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
               [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
               [clojure.main$repl$fn__9095 invoke "main.clj" 458]
               [clojure.main$repl invokeStatic "main.clj" 458]
               [clojure.main$repl doInvoke "main.clj" 368]
               [clojure.lang.RestFn invoke "RestFn.java" 1523]
               [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
               [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
               [nrepl.middleware.interruptible_eval$interruptible_eval$fn__985$fn__989
                invoke
                "interruptible_eval.clj"
                155]
               [clojure.lang.AFn run "AFn.java" 22]
               [nrepl.middleware.session$session_exec$main_loop__1086$fn__1090 invoke "session.clj" 190]
               [nrepl.middleware.session$session_exec$main_loop__1086 invoke "session.clj" 189]
               [clojure.lang.AFn run "AFn.java" 22]
               [java.lang.Thread run "Thread.java" 829]]}

Evidence

The previously pasted example is reproducible and always throws a ClassCastException.

Impact

We are unable to test functionality in a consistent manner. Local setups using dev-local file systems will not function as expected. Developers are unable to test features in the same way they will work in production.

d/client external server communication

Thu, 24 Jun 2021 14:55:59 +0000

Environment

com.datomic/client-cloud 0.8.105

History

Get the docstring for d/client.

(clojure.repl/doc d/client)
-------------------------
datomic.client.api/client
([arg-map])
  Create a client for a Datomic system. This function does not
communicate with a server and returns immediately.

For a cloud system, arg-map requires:

  :server-type   - :cloud
  :region        - AWS region, e.g. "us-east-1"
  :system        - your system name
  :endpoint      - IP address of your system or query group

Optionally, a cloud system arg-map accepts:

  :creds-provider  - instance of com.amazonaws.auth.AWSCredentialsProvider. Defaults to DefaultAWSCredentialsProviderChain
  :creds-profile   - name of an AWS Named Profile. See http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
  :proxy-port      - local port for SSH tunnel to bastion 

  Note: :creds-provider and :creds-profile are mutually exclusive, providing both will result in an error.

For a dev-local system, arg-map comprises:

  :server-type   - :dev-local (required)
  :system        - a system name (required)
  :storage-dir   - optional, overrides :storage-dir in ~/.datomic/dev-local.edn

dev-local stores databases under ${storage-dir}/${system}/${db-name}.

For a peer-server system, arg-map requires:

  :server-type   - :peer-server
  :access-key    - access-key from peer server launch
  :secret        - secret from peer server launch
  :endpoint      - peer server host:port
  :validate-hostnames  - false

Returns a client object.

Note the second sentence in the docstring: "This function does not communicate with a server and returns immediately." Next call d/client with an endpoint that does not exist.

(d/client {:server-type :ion
             :system      "my-system"
             :endpoint    "http://i-dont-exist.datomic.net:8182"
             :region      "us-west-2"})
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to http://i-dont-exist.datomic.net:8182

Expectation

Calling d/client does not communicate with a server.

Actual

Calling d/client does communicate with a server.

Evidence

The above reproducible example indicates the d/client function call is attempted to communicate the the passing in :endpoint string given the exception thrown.

Impact

The d/client docstring misleads the user by saying no server communication will occur when it does occur. The result is calls to d/client can fail in unexpected ways when an external server communication is not accounted for. For example, we spin up Datomic Cloud query groups dynamically and attempt to communicate with the newly spun up cluster. Until the cluster is up and healthy, we expect calls to fail at d/connect with an unavailable anomaly, and instead the calls would fail at d/client with an unavailable anomaly.

Add query group ASG metrics to query group CW dashboard

Thu, 27 May 2021 17:22:27 +0000

When looking at the query group dashboard, it is helpful to know what "state" the ASG was in when referencing the various dashboard metrics. Currently, I must do one of the following: 1) reference these metrics in the ASG monitoring tab (e.g., https://us-west-2.console.aws.amazon.com/ec2autoscaling/home?region=us-west-2#/details/my-query-group?view=monitoring), 2) create a CW dashboard of my own, 3) modify the existing Datomic CW dashboard to add this chart. The ASG metrics answer questions like: 1) how many instances was my query group running? 2) was my query group trying to scale out or in? 3) why did my query group not scale out or in?

I would find the following ASG metrics (in order of importance) useful additions to the current query group dashboard.

In Service Instances (Count)
Minimum Group Size (Count), Maximum Group Size (Count), and Desired Capacity (Count)
Pending Instances (Count)

How might multiple systems serve the same database in different contexts?

Mon, 24 May 2021 15:16:46 +0000

From the guide on naming there is the following quote:

Multiple systems might serve the same primary database in different development stages. So you might name systems via the convention "[db]-[stage]", e.g."inventory-dev", "inventory-staging" and "inventory-prod".

I'm trying to understand how this would work, and how I should think about my system, application, and database name, but I'm not really sure how this would work.

My understanding was that you have one System per Storage stack, plus some number of Compute stacks, and that databases are "owned" by the Storage stack, but that doesn't make sense with the quote above.

Can anyone give some examples of how this would work? I've read the planning guide but almost certainly don't fully understand it.

Thank you!
Evan

What is a performant way to do "and" matching on attribute values?

Fri, 21 May 2021 18:04:41 +0000

So I can use

[?entity1 ?attrname1 ?attrval]
[?entity2 ?attrname2 ?attrval]

in a :where clause to get ?entity1 and ?entity2 where there exists an ?attrval that matches

I'm using

  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity1 ?attrname1) [[?attrvals1]]]
  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity2 ?attrname2) [[?attrvals2]]]
  (not-join [?attrvals1 ?attrvals2]
               [(seq ?attrvals1) [?element ...]]
               (not [(contains? ?attrvals2 ?element)]))

to get ?entity1 and ?entity2 where all ?attrvals for ?entity1 exist for ?entity2. Is there a more performant way to do this??
(This feels like a directional "and" to the implicit "or" being applied to each attrval matching in the first case)

AssertionError shows up as an Event?

Tue, 18 May 2021 15:33:51 +0000

This surprised me. In a lambda invocation, a failed assert shows up as an Event. I would have expected Alert:

"Msg": "IonLambdaException",
"Ex": {
    "Via": [
        {
            "Type": "java.lang.AssertionError",
            "Message": "Assert failed: (= (count s) 16)",
        }
    ],
    "Cause": "Assert failed: (= (count s) 16)"
},
"Type": "Event",
"Tid": 338,
"Timestamp": 162134781093

Additional keys on anomaly part of the API?

Sun, 09 May 2021 01:33:17 +0000

Datomic sometimes attaches additional keys onto the anomaly in ex-data on the ExceptionInfos thrown. Is this additional data part of the API and able to be depended on? Further, could all keys returned be documented in the api docs? The returned keys are typically quite helpful in dispatching on different Datomic error "types."

Here's an example.

(def client (d/client {:server-type :dev-local
                       :storage-dir :mem
                       :system      "aa"}))
=> #'compute.http-api.public.task/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'compute.http-api.public.task/conn
(d/transact conn {:tx-data [{:db/ident       :user/id
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string
                             :db/unique      :db.unique/identity}
                            {:db/ident       :user/name
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string
                             :db/unique      :db.unique/identity}]})
=>
{:db-before #datomic.core.db.Db{:id "47c8f6f7-26f9-4575-97ec-7fc0948823c1",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "47c8f6f7-26f9-4575-97ec-7fc0948823c1",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:32:25.677-00:00" 13194139533318 true]
           #datom[73 10 :user/id 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 42 38 13194139533318 true]
           #datom[74 10 :user/name 13194139533318 true]
           #datom[74 41 35 13194139533318 true]
           #datom[74 40 23 13194139533318 true]
           #datom[74 42 38 13194139533318 true]
           #datom[0 13 73 13194139533318 true]
           #datom[0 13 74 13194139533318 true]],
 :tempids {}}
(d/transact conn {:tx-data [{:db/id     [:user/id "a"]
                             :user/name "a"}]})
Execution error (ExceptionInfo) at datomic.core.error/raise (error.clj:55).
:db.error/not-an-entity Unable to resolve entity: [:user/id "a"] in datom [[:user/id "a"] :user/name "a"]
(ex-data *e)
=>
{:cognitect.anomalies/category :cognitect.anomalies/incorrect,
 :cognitect.anomalies/message "Unable to resolve entity: [:user/id \"a\"] in datom [[:user/id \"a\"] :user/name \"a\"]",
 :entity [:user/id "a"],
 :datom [[:user/id "a"] :user/name "a"],
 :db/error :db.error/not-an-entity}

Use Launch Template instead of Launch Configuration

Fri, 07 May 2021 23:38:51 +0000

The compute Datomic CloudFormation templates (primary and query groups) create ASGs using launch configurations. Launch configurations do not support the full set of Auto Scaling group features. For example, you cannot create an Auto Scaling group that launches both Spot and On-Demand Instances or that specifies multiple instance types or multiple launch templates. In fact, AWS recommends creating ASGs using launch templates, not launch configurations.

We are interested in launch templates for its support for using Spot and On-Demand instances.

EDIT: As of 2021-10-20, launch templates will not receive any new EC2 ASG features.

Does the transactor eagerly fill memcached with indexed segments?

Wed, 07 Apr 2021 15:21:18 +0000

The datomic transactor memcached documentation says this:

When configured to used memcached, a Datomic process will automatically write into memcached any segment it needs that is not already present in memcached. In addition, the transactor (only) will write index and log segments to memcached as the segments are produced.

(Emphasis mine.)

Based on this sentence and assuming a memcached memory pool larger than the storage size that is shared by transactor and peer, my expectation is the following:

As the transactor is building an index, it writes newly-created index segments to storage and to memcached at roughly the same time.
When the index is finished, it commits the new index root and informs all peers of the new index.
Any peers using the same memcached cluster should, at the moment the index is released:
- have a decreasing ObjectCache hitrate because newly-indexed segments are not cached
- but an increasing memcached GET count
- and a flat StorageGet and MemcachedPut count because the segments are already in memcached
- For an overall flat or increasing memcached hitrate.

Instead, what I actually see is that when peers accept a new index,

there's a dramatic decrease in Memcached hitrate
and increase in MemcachedPut counts and Storage Get counts.
The IndexWrite count reported by the transactor is very similar to the MemcachedPut count reported by the peer.

This behavior is consistent with the transactor not writing newly-indexed segments into memcached eagerly while indexing, which is contrary to the documentation.

So the question: Does the transactor really write index segments to memcached as it is indexing?

Support for recent AWS instance types

Wed, 24 Mar 2021 18:36:03 +0000

AWS has released several new instance types that are much better price per performance. It'd be great to have the Datomic query group template updated to support these instance types. The Datomic query group template currently supports the following instance types: t3.medium, m5.large, i3.large, i3.xlarge. Based on this, I propose the following additions:

t3a.medium (~10% cheaper than t3.medium)
m5a.large (~10% cheaper than m5.large)

If Datomic can run on ARM based processors then further additions can be made with even better price per performance (the new AWS Graviton2 processors are fantastic. Read about Honeycomb's success).

t4g.medium (perhaps also large and xlarge, ~20% cheaper than t3.medium)
m6g.large (~20% cheaper than m5.large)
r6gd.large (has NVMe based SSD, 26% cheaper than i3.large)
I4i (released 2022-04-28)

The Datomic Solo instance type could also be configurable for some inconsequential savings. See monthly prices below:

(Current) t3.small primary + t3.nano bastion: $18.90
t3a.small + t3a.nano: $17.15
t4g.small + t4g.nano: $15.33

An example Datomic query group monthly prices. Take a query group with 2 nodes running for the full month.

(Current) m5.large: $140.16
m5a.large: $125.56 ($14.6/month savings)
m6g.large: $112.42 ($27.74/month savings)

Make DesiredCapacity an optional parameter in the query group CF template

Wed, 24 Mar 2021 17:24:25 +0000

Currently (March 2021) the Datomic query group CF template has the DesiredCapacity parameter marked as required. Because of this, during a CF template update, CF will always set the DesiredCapacity to the CF parameter for DesiredCapacity. This behavior does not work well in the following situation.

In the query group template I have the DesiredCapacity at 2, and MaxCount at 4. I have an ASG scaling policy to target track 50% average CPU utilization. My query group is currently at the max, because the ASG scaling policy scaled up the group DesiredCapacity to 4 (to match my load). My load has increased to the point where 4 nodes is no longer sufficient. I must update the query group CF template to increase the MacCount. The problem is that when you update the MacCount from 4 to 6 via a CF template update and leave the DesiredCapacity at 2, CF will actually set the DesiredCapacity on the ASG to 2 during the high load situation. This worsens the high load situation.

One option is to query the ASG for its current DesiredCapacity prior to running a CF update and set the CF parameter to the value you retrieved. This approach complicates the update procedure. It also has a "read-modify-write" race condition -- the DesiredCapacity could have changed from the time you read the DesiredCapacity to the time you are performing the CF update.

A second option is to make the DesiredCapacity parameter optional. As per the CF docs, DesiredCapacity is not a required parameter. When it is not present, it defaults to the minimum size of the group. Leaving DesiredCapacity unset will make it so during a CF update, CF will not modify the DesiredCapacity at all -- leaving the ASG "in control."

I am now seeing Iterator+offset is not available doing a an index-pull

Mon, 08 Mar 2021 16:46:35 +0000

I am doing an index-pull in Datomic Cloud. The request was timing out so I addedd :timeout 300000 and not "Iterator+offset is not available" exception starts popping up

We are on 781-9041

Reverse order browsing for raw indexes

Mon, 25 Jan 2021 11:39:25 +0000

Hello,
The index-pull function has the :reverse option to go through the index in reverse order.
Would it be possible to add it to the raw index browsing functions index-range and seek-datoms ?

Dev-local query does not support :timeout

Wed, 30 Dec 2020 21:25:39 +0000

Dev-local 0.9.229 does not appear to support :timeout when querying. Here is a repro.

(def client (d/client {:server-type :dev-local
                       :system      "example"}))
(d/create-database client {:db-name "example"})
(def conn (d/connect client {:db-name "example"}))
(d/transact conn {:tx-data [{:db/ident       :string
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string}]})
(dotimes [n 1000]
  (d/transact conn {:tx-data (map (fn [i]
                                    {:string (str n i)}) (range 1000))}))
(first
  (d/q {:query   '[:find (pull ?e [*])
                   :where
                   [?e :string]]
        :timeout 1000
        :args    [(d/db conn)]}))

The query takes longer than 1s and does not throw. I would have expected an exception thrown.

Add default :timeout to client api ns docstring

Thu, 24 Dec 2020 00:12:44 +0000

It'd be great to have the default value for :timeout (60000) included in the datomic.client.api ns docstring. Currently this value can only be found in the docs on the website. As of client-cloud 0.8.102, the ns docstring specifies the default value for :limit.

Datomic Client API equivalent to on-prem tx->t, t->tx

Mon, 21 Dec 2020 00:25:17 +0000

The Datomic client API does not provider tx->t or t->tx. These functions can be useful for very specific scenarios (e.g., you have a tx id and want to know if the database has "moved forward").

This was also requested on the forums. A workaround was provided for the user's specific case but the request is still relevant.

Allow dev-local import-cloud :source and :dest to work with any :server-type

Sun, 20 Dec 2020 00:37:33 +0000

Currently import-cloud requires the :source client :server-type to be :cloud and the :dest :server-type to be :dev-local. It'd be great if this function allowed any :server-type to be passed for :source and :dest. A possible use case would be copying a dev-local database to a cloud system for testing in a staging environment.

How do I initialize core.async connections in Datomic Cloud?

Sat, 28 Nov 2020 23:53:01 +0000

I need to "hook up" a core.async channel to a thread on system initialization. How might I go about doing this, or doing any initialization tasks?

Thanks,
Hadil

How can I get a ring-like request in ionized web handlers?

Thu, 26 Nov 2020 19:04:57 +0000

Hello! After hearing the talks and reading the tutorial and documentation, I was under the impression that ions receive their requests in a ring-like format, and that we can "wire a web handler to a single function, using a routing library such as Compojure or pedestal."1

On one hand, this seems to be true with standard ion lambdas, but these are not exposed to the web. OTOH if we want to expose our web service, we need to write a web handler that we'll ionize. When we do that, the requests coming in lose their ring-like shape, and only expose the following keys:

 :headers
 :server-name
 :body
 :datomic.ion.edn.api-gateway/json
 :datomic.ion.edn.api-gateway/data

With that, our router (reitit backed by ring) fails.

I have seen different examples on the web where some router is exposed behind a single web handler, the most promising of which being probably Replion (look at the code) (and he uses reitit and ring and seems to have more success than us) and Pedestal Ions Sample (but Pedestal works in a relaxed ring context).

So my question is: is there something obvious that I don't understand? Are there more working examples? Some people seem to write adapters to transform the requests to make them more ring-like. Are they doing the right thing, or are we all missing something?

For example, here's a very naive adapter:

(defn get-path [req]
  (let [s (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :stage] "stg")
        p (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :http :path] "/")]
    (get (str/split p (re-pattern (str "/" s))) 1 "/"))) ; get rid of stage leading path!

(defn req->ring-req [req]
  (assoc req
         :protocol (get-in req
                           [:datomic.ion.edn.api-gateway/data :requestContext :http :protocol] "HTTP/1.1")
         :remote-addr (get-in req [:headers "host"] "example.com")
         :request-method (keyword
                          (str/lower-case
                           (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :http :method] "GET")))
         :uri (get-path req)))

(defn app
  "Web handler that returns the app's responses for each web request."
  [arg]
  (let [sta (start! arg)
        req (req->ring-req arg)
        ret ((handler/app) req)]
    ret))

Another related question is: what is the :integration key on lambda configurations? Why does "nobody" use it, except Stu's slack event handler? What does it do? We tried setting it and didn't seem to observe an impact.

Thanks!

What to do about "syncContext completed and\/or Request lifecycle recycled"?

Mon, 16 Nov 2020 22:03:14 +0000

We get into a state where we see a lot of syncContext completed and\/or Request lifecycle recycled

What does it mean?
What can be done about it?

The stack trace is:

"Trace": [
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "state",
                "AsyncContextState.java",
                52
            ],
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "getResponse",
                "AsyncContextState.java",
                118
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invokeStatic",
                "jetty.clj",
                326
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invoke",
                "jetty.clj",
                323
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invokeStatic",
                "jetty.clj",
                351
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invoke",
                "jetty.clj",
                348
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "fail_request",
                "http_endpoint.clj",
                126
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "respond",
                "http_endpoint.clj",
                147
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127$fn__13129",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invokeStatic",
                "ioc_macros.clj",
                978
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invoke",
                "ioc_macros.clj",
                977
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invokeStatic",
                "ioc_macros.clj",
                982
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invoke",
                "ioc_macros.clj",
                980
            ],
            [
                "clojure.core.async.impl.ioc_macros$take_BANG_$fn__5769",
                "invoke",
                "ioc_macros.clj",
                991
            ],
            [
                "clojure.core.async.impl.channels.ManyToManyChannel$fn__589$fn__590",
                "invoke",
                "channels.clj",
                95
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor",
                "runWorker",
                "ThreadPoolExecutor.java",
                1149
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor$Worker",
                "run",
                "ThreadPoolExecutor.java",
                624
            ],
            [
                "clojure.core.async.impl.concurrent$counted_thread_factory$reify__492$fn__493",
                "invoke",
                "concurrent.clj",
                29
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.lang.Thread",
                "run",
                "Thread.java",
                748
            ]
        ]

Provide way to interact with a datomic database before making the system available

Mon, 02 Nov 2020 15:45:42 +0000

Currently in Cloud there is no way to interact with a datomic database before making the system available for external requests.

Potential uses:

deploy the same code everywhere (dev & prod) and it does db-ensure
actions
load data and/or schema prior to the system being live
start components or other system-management tooling
provide (easy) way to integrate startup (e.g. mount/system-component)
provide status and ability to know when pieces are started so you can finish app initialization

Does dev-local/import-cloud handle failure in the middle of an import?

Fri, 30 Oct 2020 16:21:53 +0000

When running an import-cloud, does dev-local gracefully handle import-cloud failing in the middle of the import? i.e., it fails (gets killed or throws) and I can call import-cloud again without worrying about corrupt data, missing data, etc.

Bind Ions to IAM roles separate from their ambient compute role

Sun, 25 Oct 2020 19:58:49 +0000

Right now Ions run under the role of the underlying compute resources that they are deployed to. It would be nice to be able to give Ions a role independent of the ambient compute. This can be managed by the Ion runtime machinery by calling AssumeRole in the background, and supplying those creds to the Ions

This is important so that Ions don't have to run with the permissions of all the other ions running in the same compute.

Expose an alt-port transactor configuration property

Sun, 25 Oct 2020 19:45:14 +0000

The alt-host configuration property is great and helps in many scenarios, but not having an equivalent for the port (alt-port?) is a blocker in some other scenarios. It would be great to have such property, which would default to the same port as the transactor is listening.