Datomic Knowledgebase - Questions without answers in Datomic

Using a collection of datoms as input to a query

Thu, 24 Apr 2025 12:51:36 +0000

I was wondering if there is a way to use a collection of datoms obtained from d/tx-range + get a single tx or received from a listener that subscribed to a tx queue as input to d/q? The nature of this collection looks very similar to history db which can be used with d/q, however passing this collection as input does not produce the expected result. Neither there is any error.

Using Valcache via Docker bind mounts

Thu, 11 Apr 2024 09:50:54 +0000

I'm evaluating caching options for Datomic Pro. Valcache sounds like a great fit, but there is one thing that might be a showstopper.

The Prerequisites docs says that "Valcache relies on an SSD with the strictatime and lazytime flags set". This requirement I can meet pretty easily by remounting the disk on the hosts running the peers.

Now, the applications that are Datomic peers, who would be the users of these Valcaches, run as Docker containers in a Docker Swarm. The idea is to mount the Valcache as a Docker bind mount.

Would this work? The underlaying SSD disk would be mounted with strictatime and lazytime, but would whatever mechanism that requires these flags to be set work through a Docker bind mount?

Possibly security issue with the dev H2 storage?

Wed, 13 Mar 2024 21:58:26 +0000

While trying to understand more about the behaviour of Datomic dev (mainly from a security/encryption point of view), I attempted to connect to datomic.mv.db from an instance of the H2 console. I accidentally tried to log in as "admin" with a blank password and was surprised when I connected successfully. Connecting as "admin" with any password fails, so it seems explicitly to be blank. As far as I can tell I only have public schema access and can't see any transacted data, but I am able to run SQL queries in the console.

Clearly a dev transactor isn't intended to be high security, but admin with a blank password seems risky even for a dev environment. In most SQL environments I work with, the sa/admin account is just disabled if not used. Am I missing something?

As an aside, does the dev transactor use H2's AES encryption or is the resulting data file unencrypted?

Is `(pull $db2 ?e [*])` supported officially?

Sat, 15 Jul 2023 04:42:45 +0000

The documentation only mentions the syntax without a src-var:

pull-expr = ['pull' variable pattern]

but found that this syntax also works:

pull-expr = ['pull' src-var variable pattern]

I have a self-contained code example here, which demonstrates a use-case, where an entity is split between 2 Datomic databases, for privacy reasons:

https://forum.datomic.com/t/pull-db-entity-pattern-is-undocumented/2299

Is this variant of the pull-expr in Datalog queries are not supported officially?

Should I use it in production or does it have some undefined behaviour, compared to its binary arity variant?

Or the documentation is just incomplete?

Datomic Cloud IAM role drift on setup - is this normal?

Wed, 12 Jul 2023 09:34:33 +0000

hi, just set up DC for the first time (ran the two AWS cloudformation templates). The storage stack is showing drift on the IAM managed policy datomic-admin-..., shows 'MODIFIED', in red. Is this normal upon setup? (checking the changes in the 'versions' tab, the difference is addition of autoscaling and s3 actions to the policy; the second version, presumably causing the 'drift', was added during the provisioning). Is this just a setup artifact nothing to worry about?

Request Entity Too Large - clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

Tue, 21 Mar 2023 12:26:20 +0000

There seems to be a bug in Datomic 990-9202 that breaks the d/tx-range function.

Earlier I could use the tx-range function the fetch data from my database.

But after upgrading to Datomic 990-9202 I get and exception when a transaction has too much data.

`
clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

cognitect.anomalies/category: :cognitect.anomalies/incorrect
 cognitect.anomalies/message: "Response body did not conform to Datomic client protocol"
                 http-result: {:status 413,
                               :headers
                               {"apigw-requestid" "Vg4UxisCDoEEJsQ=",
                                "server" "Jetty(9.4.44.v20210927)",
                                "connection" "keep-alive",
                                "content-length" "38",
                                "date" "Tue, 19 Jul 2022 12:17:47 GMT",
                                "content-type" "application/json"},
                               :body "{\"message\":\"Request Entity Too Large\"}"}

Is it possible for Datomic On-prem to get the storage credentials from Secrets Manager or similar software?

Thu, 02 Mar 2023 10:50:19 +0000

Is there a way to run the transactor service without storing the credentials to disk?

What's the state of the art in distributed Datomic?

Sun, 18 Dec 2022 00:37:33 +0000

I'd like to get an idea of the state of the art in distributed Datomic. FWIW, my wish list includes the ability for selected processing nodes to have local images and/or participate in federated global images, linked by CRDTs, etc. And a pony.

-r

Querying 2000+ entities, by passing in uuids (Really slow)

Wed, 14 Dec 2022 09:24:29 +0000

Passing in over 2000 uuids to mysql for a fairly simple query takes milliseconds to retrieve relevant data. However we're running into an issue with datomic where it takes noticeable longer. Is there a specific syntax to use when processing such long lists, or is the linked example the only way?

https://docs.datomic.com/on-prem/query/query.html#collection-binding

Inconsistency between d/pull and d/q + pull for non-existent entities

Sat, 10 Dec 2022 15:55:29 +0000

Inconsistency between d/pull and d/q + pull for non-existent entities

Datomic returns a map with the :db/id of the entity being pulled, even if the entity doesn't exist in the database. For example:

(d/pull (d/db conn) '[*] 9999999999)
=> #:db{:id 9999999999}

This behavior is somewhat confusing, especially when using d/pull with time filters. I was expecting d/pull to return nil or #:db{:id nil} if the entity wasn't created at the given point in time. However, Datomic always returns a map with the entity ID:

(d/pull (as-of (d/db conn) 1) '[*] 9999999999)
=> #:db{:id 9999999999}

Surprisingly, I see the expected behavior if I pull the entity information from a query using d/q:

(d/q '{:find [(pull ?e [*])]
        :in   [?e]}
      (d/db conn) 9999999999)
=> [[#:db{:id nil}]]

Is that difference in behavior expected? How can I determine if an entity existed at any point in time using d/pull?

How do you add relationship properties, e.g. the time a person met another person?

Mon, 07 Nov 2022 08:35:28 +0000

I have some experience with Neo4j / cypher. In cypher relationships are first-class citizens, so they can have properties, which I really like.

How do I do something similar with datalog? Do I have to have a "mapping entity" in between two entities that have a relationship?

Is using entid function to resolve lookup refs a supported behaviour?

Wed, 08 Jun 2022 13:30:29 +0000

I'm using datomic.api/entid to resolve a lookup ref to an entity ID, like
$ (entid db [:user/id "test@test.com"]) 17592186045432
This works fine, but a colleague pointed out this is not a documented behaviour.
>Usage: (entid db ident)
>Returns the entity id associated with a symbolic keyword, or the id
itself if passed.

Is this a gap in documentation, or is this behaviour accidental and not recommended to rely on?

Notifying clients of database changes in Datomic Cloud?

Sat, 04 Jun 2022 15:18:15 +0000

In your 2013 blog post The Transaction Report Queue, you write:

In Datomic, you can monitor all transactions. Any peer process in the
system can request a transaction report queue of every transaction
against a particular database.

Is there a change data capture or related mechanism in Datomic Cloud, where changes to the database can be pushed to a client?

Thank you,

Andrew

Retracting entities, including tuples with refs

Tue, 31 Aug 2021 13:36:44 +0000

In my schema I have attributes of type tuple that has :db.type/ref as one of the tuple's types.
My usecase is maintaining a set of ordered to-many relations.

When I retract an entity using :db/retractEntity, the tuple attrs that refer to the retracted entity still remain, continuing to refer to a now non-existent entity.
I'm not sure whether this is the intended behaviour, and if so, what would be a standard approach to clean up these tuples? My intention would be for these tuples to be completely redacted. As I'm using this in a cardinality-many setup, I'd expect the attribute to retain all tuple values, except the one that does include the ref to the retracted entity in any position.
Should I be writing a custom retractEntity-like transaction function?
Thanks!

How to authenticate and authorize access to API Gateway endpoints for datomic?

Thu, 19 Aug 2021 17:05:29 +0000

I'm currently evaluating Datomic and I've deployed a production topology using a split stack. DatomicCloudVersion is 9095 and DatomicCFTVersion is 884. The CloudFormation stack created two API Gateway endpoints: datomic-[system]-client-api and datomic-[system]-ions.

From my local computer I can get a client by using datomic-[system]-client-api endpoint:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

Above, https://[endpoint-id].execute-api.us-east-1.amazonaws.com/ is the client-api API Gateway endpoint. It works, but the endpoint is open to the world. I then added an IAM authorizer to datomic-[system]-client-api and used an authorized IAM user to make a request to the now authenticated endpoint:

awscurl --service execute-api -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com --profile datomic
{:s3-auth-path "long s3 path"}

awscurl is just a tool to make authenticated calls to AWS services. As you can see, I can get a response from the endpoint by using my local profile named "datomic". If I try using regular curl without the v4 AWS signature it fails:

curl -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com
{"message":"Forbidden"}

Works as expected. The problem is now I can't get a client using:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :creds-profile "datomic"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

The response I get is:

{:status 403,
:headers
{"apigw-requestid" "EUsyNi9aIAMEPlg=",
"connection" "keep-alive",
"content-length" "23",
"date" "Thu, 19 Aug 2021 17:01:40 GMT",
"content-type" "application/json"},
:body "{\"message\":\"Forbidden\"}"}

It is as if the d/client call is ignoring the profile or not generating the proper signature. I tried setting the AWS_PROFILE environment variable as well, to no avail.

Is using an IAM Authorizer not the proper way to go? How can I protect both the client-api and ions endpoints? Finally, what's the difference between the client-api and ions endpoints?

Datomic returns empty results where "Insufficient bindings, will cause db scan" exception is expected

Wed, 07 Jul 2021 14:30:05 +0000

Hello

If we change #uuid"..." to "" (from uuid to string), we get the exception (desired behavior)

clj -Sdeps '{:deps {com.datomic/dev-local {:mvn/version "0.9.232"}}}' -M \
  -e "(require '[datomic.client.api :as d])" \
  -e '(def client (d/client {:server-type :dev-local :system "hello"}))' \
  -e '(d/create-database client {:db-name "hello"})' \
  -e '(def conn (d/connect client {:db-name "hello"}))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
=> #'user/client
=> true
=> #'user/conn
=> []

Occour on (old) versions of datomic-free too

clj -Sdeps '{:deps {com.datomic/datomic-free {:mvn/version "0.9.5697"}}}' \
  -M --report stderr \
  -e "(require '[datomic.api :as d])" \
  -e '(d/create-database "datomic:mem://hello")' \
  -e '(def conn (d/connect "datomic:mem://hello"))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
WARNING: requiring-resolve already refers to: #'clojure.core/requiring-resolve in namespace: datomic.common, being replaced by: #'datomic.common/requiring-resolve
=> true
=> #'user/conn
=> #{}

ClassCastException, d/with inside transact function

Thu, 24 Jun 2021 14:57:51 +0000

Calling a transaction function that uses d/with in a dev-local, filesystem (i.e., not :storage-dir :mem) throws a ClassCastException on internal Datomic code.

Environment

com.datomic/dev-local {:mvn/version "0.9.232"}
client-cloud 0.8.105

History

Example with exception using file system dev-local.

(def client (d/client {:server-type :dev-local
                         :storage-dir "/tmp/cce-bug3"
                         :system      "test"}))
=> #'user/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'user/conn
(d/transact conn {:tx-data [{:db/ident       :user/name
                               :db/valueType   :db.type/string
                               :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "test",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "test",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T00:49:46.400-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(defn my-tx-fn
    [db argm]
    (let [tx-report (d/with db {:tx-data [{:user/name "asd"}]})]
      []))
=> #'user/my-tx-fn
(d/transact conn {:tx-data [(list 'user/my-tx-fn {})]})
Execution error (ClassCastException) at datomic.dev-local.tx/datom-lookup-valfn (tx.clj:397).
class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')

Working memdb example.

(def client-memdb (d/client {:server-type :dev-local
                               :storage-dir :mem
                               :system      "test"}))
=> #'user/client-memdb
(d/create-database client-memdb {:db-name "test"})
=> true
(def conn-memdb (d/connect client-memdb {:db-name "test"}))
=> #'user/conn-memdb
(d/transact conn-memdb {:tx-data [{:db/ident       :user/name
                                     :db/valueType   :db.type/string
                                     :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:02:57.965-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(d/transact conn-memdb {:tx-data [(list 'user/my-tx-fn {})]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 6,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 7,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533319 50 #inst"2021-05-09T01:03:08.042-00:00" 13194139533319 true]],
 :tempids {}}

Expectation

Transaction functions that use d/with do not throw a ClassCastException. Additionally, the behavior of d/with in all Datomic environments (Cloud, File system, Memdb) is consistent. Given a previous Datomic Cloud release fixed a problem with using with-db from an Ion, it seems using d/with in a transaction function is a supported behavior.

Actual

The final d/transact transaction function throws a ClassCastException when using d/with on some internal Datomic code. This only occurs when using a dev-local, file system environment.

Full stacktrace is pasted below.

#error{:cause "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
       :via [{:type java.lang.ClassCastException,
              :message "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
              :at [datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]}],
       :trace [[datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]
               [datomic.dev_local.tx$datom_lookup_valfn invoke "tx.clj" 397]
               [datomic.dev_local.local_log.LocalLog valAt "local_log.clj" 56]
               [clojure.lang.RT get "RT.java" 760]
               [datomic.dev_local.btindex.BTIndex cons "btindex.clj" 281]
               [clojure.lang.RT conj "RT.java" 677]
               [clojure.core$conj__5390 invokeStatic "core.clj" 85]
               [clojure.core$conj__5390 invoke "core.clj" 82]
               [datomic.core.db.Db addData "db.clj" 2322]
               [datomic.core.db$add_ensured_data invokeStatic "db.clj" 3353]
               [datomic.core.db$add_ensured_data invoke "db.clj" 3351]
               [datomic.core.db$with_tx invokeStatic "db.clj" 3370]
               [datomic.core.db$with_tx invoke "db.clj" 3357]
               [datomic.core.db.Db with "db.clj" 2164]
               [datomic.core.local_db$fn__25633 invokeStatic "local_db.clj" 67]
               [datomic.core.local_db$fn__25633 invoke "local_db.clj" 24]
               [datomic.client.api.protocols$fn__11959$G__11877__11966 invoke "protocols.clj" 126]
               [datomic.client.api$with invokeStatic "api.clj" 363]
               [datomic.client.api$with invoke "api.clj" 353]
               [user$my_tx_fn invokeStatic "user.clj" 3]
               [user$my_tx_fn invoke "user.clj" 20]
               [clojure.lang.AFn applyToHelper "AFn.java" 156]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.lang.Var applyTo "Var.java" 705]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$apply invoke "core.clj" 660]
               [datomic.core.db.ProcessExpander inject "db.clj" 3229]
               [datomic.core.db.ProcessInpoint inject "db.clj" 2950]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586$fn__18587 invoke "btindex_db.clj" 440]
               [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
               [clojure.core$reduce invokeStatic "core.clj" 6827]
               [clojure.core$reduce invoke "core.clj" 6810]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586 invoke "btindex_db.clj" 440]
               [datomic.dev_local.btindex_db$expand_tx invokeStatic "btindex_db.clj" 444]
               [datomic.dev_local.btindex_db$expand_tx invoke "btindex_db.clj" 429]
               [datomic.dev_local.btindex_db$with_tx invokeStatic "btindex_db.clj" 468]
               [datomic.dev_local.btindex_db$with_tx invoke "btindex_db.clj" 461]
               [datomic.dev_local.impl.DurableConnection transact "impl.clj" 161]
               [datomic.client.api$transact invokeStatic "api.clj" 200]
               [datomic.client.api$transact invoke "api.clj" 183]
               [user$eval124260 invokeStatic "user.clj" 25]
               [user$eval124260 invoke "user.clj" 25]
               [clojure.lang.Compiler eval "Compiler.java" 7177]
               [clojure.lang.Compiler eval "Compiler.java" 7132]
               [clojure.core$eval invokeStatic "core.clj" 3214]
               [clojure.core$eval invoke "core.clj" 3210]
               [nrepl.middleware.interruptible_eval$evaluate$fn__959 invoke "interruptible_eval.clj" 91]
               [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
               [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
               [clojure.main$repl$fn__9095 invoke "main.clj" 458]
               [clojure.main$repl invokeStatic "main.clj" 458]
               [clojure.main$repl doInvoke "main.clj" 368]
               [clojure.lang.RestFn invoke "RestFn.java" 1523]
               [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
               [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
               [nrepl.middleware.interruptible_eval$interruptible_eval$fn__985$fn__989
                invoke
                "interruptible_eval.clj"
                155]
               [clojure.lang.AFn run "AFn.java" 22]
               [nrepl.middleware.session$session_exec$main_loop__1086$fn__1090 invoke "session.clj" 190]
               [nrepl.middleware.session$session_exec$main_loop__1086 invoke "session.clj" 189]
               [clojure.lang.AFn run "AFn.java" 22]
               [java.lang.Thread run "Thread.java" 829]]}

Evidence

The previously pasted example is reproducible and always throws a ClassCastException.

Impact

We are unable to test functionality in a consistent manner. Local setups using dev-local file systems will not function as expected. Developers are unable to test features in the same way they will work in production.

d/client external server communication

Thu, 24 Jun 2021 14:55:59 +0000

Environment

com.datomic/client-cloud 0.8.105

History

Get the docstring for d/client.

(clojure.repl/doc d/client)
-------------------------
datomic.client.api/client
([arg-map])
  Create a client for a Datomic system. This function does not
communicate with a server and returns immediately.

For a cloud system, arg-map requires:

  :server-type   - :cloud
  :region        - AWS region, e.g. "us-east-1"
  :system        - your system name
  :endpoint      - IP address of your system or query group

Optionally, a cloud system arg-map accepts:

  :creds-provider  - instance of com.amazonaws.auth.AWSCredentialsProvider. Defaults to DefaultAWSCredentialsProviderChain
  :creds-profile   - name of an AWS Named Profile. See http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
  :proxy-port      - local port for SSH tunnel to bastion 

  Note: :creds-provider and :creds-profile are mutually exclusive, providing both will result in an error.

For a dev-local system, arg-map comprises:

  :server-type   - :dev-local (required)
  :system        - a system name (required)
  :storage-dir   - optional, overrides :storage-dir in ~/.datomic/dev-local.edn

dev-local stores databases under ${storage-dir}/${system}/${db-name}.

For a peer-server system, arg-map requires:

  :server-type   - :peer-server
  :access-key    - access-key from peer server launch
  :secret        - secret from peer server launch
  :endpoint      - peer server host:port
  :validate-hostnames  - false

Returns a client object.

Note the second sentence in the docstring: "This function does not communicate with a server and returns immediately." Next call d/client with an endpoint that does not exist.

(d/client {:server-type :ion
             :system      "my-system"
             :endpoint    "http://i-dont-exist.datomic.net:8182"
             :region      "us-west-2"})
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to http://i-dont-exist.datomic.net:8182

Expectation

Calling d/client does not communicate with a server.

Actual

Calling d/client does communicate with a server.

Evidence

The above reproducible example indicates the d/client function call is attempted to communicate the the passing in :endpoint string given the exception thrown.

Impact

The d/client docstring misleads the user by saying no server communication will occur when it does occur. The result is calls to d/client can fail in unexpected ways when an external server communication is not accounted for. For example, we spin up Datomic Cloud query groups dynamically and attempt to communicate with the newly spun up cluster. Until the cluster is up and healthy, we expect calls to fail at d/connect with an unavailable anomaly, and instead the calls would fail at d/client with an unavailable anomaly.

Add query group ASG metrics to query group CW dashboard

Thu, 27 May 2021 17:22:27 +0000

When looking at the query group dashboard, it is helpful to know what "state" the ASG was in when referencing the various dashboard metrics. Currently, I must do one of the following: 1) reference these metrics in the ASG monitoring tab (e.g., https://us-west-2.console.aws.amazon.com/ec2autoscaling/home?region=us-west-2#/details/my-query-group?view=monitoring), 2) create a CW dashboard of my own, 3) modify the existing Datomic CW dashboard to add this chart. The ASG metrics answer questions like: 1) how many instances was my query group running? 2) was my query group trying to scale out or in? 3) why did my query group not scale out or in?

I would find the following ASG metrics (in order of importance) useful additions to the current query group dashboard.

In Service Instances (Count)
Minimum Group Size (Count), Maximum Group Size (Count), and Desired Capacity (Count)
Pending Instances (Count)

How might multiple systems serve the same database in different contexts?

Mon, 24 May 2021 15:16:46 +0000

From the guide on naming there is the following quote:

Multiple systems might serve the same primary database in different development stages. So you might name systems via the convention "[db]-[stage]", e.g."inventory-dev", "inventory-staging" and "inventory-prod".

I'm trying to understand how this would work, and how I should think about my system, application, and database name, but I'm not really sure how this would work.

My understanding was that you have one System per Storage stack, plus some number of Compute stacks, and that databases are "owned" by the Storage stack, but that doesn't make sense with the quote above.

Can anyone give some examples of how this would work? I've read the planning guide but almost certainly don't fully understand it.

Thank you!
Evan

What is a performant way to do "and" matching on attribute values?

Fri, 21 May 2021 18:04:41 +0000

So I can use

[?entity1 ?attrname1 ?attrval]
[?entity2 ?attrname2 ?attrval]

in a :where clause to get ?entity1 and ?entity2 where there exists an ?attrval that matches

I'm using

  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity1 ?attrname1) [[?attrvals1]]]
  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity2 ?attrname2) [[?attrvals2]]]
  (not-join [?attrvals1 ?attrvals2]
               [(seq ?attrvals1) [?element ...]]
               (not [(contains? ?attrvals2 ?element)]))

to get ?entity1 and ?entity2 where all ?attrvals for ?entity1 exist for ?entity2. Is there a more performant way to do this??
(This feels like a directional "and" to the implicit "or" being applied to each attrval matching in the first case)

AssertionError shows up as an Event?

Tue, 18 May 2021 15:33:51 +0000

This surprised me. In a lambda invocation, a failed assert shows up as an Event. I would have expected Alert:

"Msg": "IonLambdaException",
"Ex": {
    "Via": [
        {
            "Type": "java.lang.AssertionError",
            "Message": "Assert failed: (= (count s) 16)",
        }
    ],
    "Cause": "Assert failed: (= (count s) 16)"
},
"Type": "Event",
"Tid": 338,
"Timestamp": 162134781093

Use Launch Template instead of Launch Configuration

Fri, 07 May 2021 23:38:51 +0000

The compute Datomic CloudFormation templates (primary and query groups) create ASGs using launch configurations. Launch configurations do not support the full set of Auto Scaling group features. For example, you cannot create an Auto Scaling group that launches both Spot and On-Demand Instances or that specifies multiple instance types or multiple launch templates. In fact, AWS recommends creating ASGs using launch templates, not launch configurations.

We are interested in launch templates for its support for using Spot and On-Demand instances.

EDIT: As of 2021-10-20, launch templates will not receive any new EC2 ASG features.

Does the transactor eagerly fill memcached with indexed segments?

Wed, 07 Apr 2021 15:21:18 +0000

The datomic transactor memcached documentation says this:

When configured to used memcached, a Datomic process will automatically write into memcached any segment it needs that is not already present in memcached. In addition, the transactor (only) will write index and log segments to memcached as the segments are produced.

(Emphasis mine.)

Based on this sentence and assuming a memcached memory pool larger than the storage size that is shared by transactor and peer, my expectation is the following:

As the transactor is building an index, it writes newly-created index segments to storage and to memcached at roughly the same time.
When the index is finished, it commits the new index root and informs all peers of the new index.
Any peers using the same memcached cluster should, at the moment the index is released:
- have a decreasing ObjectCache hitrate because newly-indexed segments are not cached
- but an increasing memcached GET count
- and a flat StorageGet and MemcachedPut count because the segments are already in memcached
- For an overall flat or increasing memcached hitrate.

Instead, what I actually see is that when peers accept a new index,

there's a dramatic decrease in Memcached hitrate
and increase in MemcachedPut counts and Storage Get counts.
The IndexWrite count reported by the transactor is very similar to the MemcachedPut count reported by the peer.

This behavior is consistent with the transactor not writing newly-indexed segments into memcached eagerly while indexing, which is contrary to the documentation.

So the question: Does the transactor really write index segments to memcached as it is indexing?

Support for recent AWS instance types

Wed, 24 Mar 2021 18:36:03 +0000

AWS has released several new instance types that are much better price per performance. It'd be great to have the Datomic query group template updated to support these instance types. The Datomic query group template currently supports the following instance types: t3.medium, m5.large, i3.large, i3.xlarge. Based on this, I propose the following additions:

t3a.medium (~10% cheaper than t3.medium)
m5a.large (~10% cheaper than m5.large)

If Datomic can run on ARM based processors then further additions can be made with even better price per performance (the new AWS Graviton2 processors are fantastic. Read about Honeycomb's success).

t4g.medium (perhaps also large and xlarge, ~20% cheaper than t3.medium)
m6g.large (~20% cheaper than m5.large)
r6gd.large (has NVMe based SSD, 26% cheaper than i3.large)
I4i (released 2022-04-28)

The Datomic Solo instance type could also be configurable for some inconsequential savings. See monthly prices below:

(Current) t3.small primary + t3.nano bastion: $18.90
t3a.small + t3a.nano: $17.15
t4g.small + t4g.nano: $15.33

An example Datomic query group monthly prices. Take a query group with 2 nodes running for the full month.

(Current) m5.large: $140.16
m5a.large: $125.56 ($14.6/month savings)
m6g.large: $112.42 ($27.74/month savings)

Make DesiredCapacity an optional parameter in the query group CF template

Wed, 24 Mar 2021 17:24:25 +0000

Currently (March 2021) the Datomic query group CF template has the DesiredCapacity parameter marked as required. Because of this, during a CF template update, CF will always set the DesiredCapacity to the CF parameter for DesiredCapacity. This behavior does not work well in the following situation.

In the query group template I have the DesiredCapacity at 2, and MaxCount at 4. I have an ASG scaling policy to target track 50% average CPU utilization. My query group is currently at the max, because the ASG scaling policy scaled up the group DesiredCapacity to 4 (to match my load). My load has increased to the point where 4 nodes is no longer sufficient. I must update the query group CF template to increase the MacCount. The problem is that when you update the MacCount from 4 to 6 via a CF template update and leave the DesiredCapacity at 2, CF will actually set the DesiredCapacity on the ASG to 2 during the high load situation. This worsens the high load situation.

One option is to query the ASG for its current DesiredCapacity prior to running a CF update and set the CF parameter to the value you retrieved. This approach complicates the update procedure. It also has a "read-modify-write" race condition -- the DesiredCapacity could have changed from the time you read the DesiredCapacity to the time you are performing the CF update.

A second option is to make the DesiredCapacity parameter optional. As per the CF docs, DesiredCapacity is not a required parameter. When it is not present, it defaults to the minimum size of the group. Leaving DesiredCapacity unset will make it so during a CF update, CF will not modify the DesiredCapacity at all -- leaving the ASG "in control."

I am now seeing Iterator+offset is not available doing a an index-pull

Mon, 08 Mar 2021 16:46:35 +0000

I am doing an index-pull in Datomic Cloud. The request was timing out so I addedd :timeout 300000 and not "Iterator+offset is not available" exception starts popping up

We are on 781-9041

Dev-local query does not support :timeout

Wed, 30 Dec 2020 21:25:39 +0000

Dev-local 0.9.229 does not appear to support :timeout when querying. Here is a repro.

(def client (d/client {:server-type :dev-local
                       :system      "example"}))
(d/create-database client {:db-name "example"})
(def conn (d/connect client {:db-name "example"}))
(d/transact conn {:tx-data [{:db/ident       :string
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string}]})
(dotimes [n 1000]
  (d/transact conn {:tx-data (map (fn [i]
                                    {:string (str n i)}) (range 1000))}))
(first
  (d/q {:query   '[:find (pull ?e [*])
                   :where
                   [?e :string]]
        :timeout 1000
        :args    [(d/db conn)]}))

The query takes longer than 1s and does not throw. I would have expected an exception thrown.

Allow dev-local import-cloud :source and :dest to work with any :server-type

Sun, 20 Dec 2020 00:37:33 +0000

Currently import-cloud requires the :source client :server-type to be :cloud and the :dest :server-type to be :dev-local. It'd be great if this function allowed any :server-type to be passed for :source and :dest. A possible use case would be copying a dev-local database to a cloud system for testing in a staging environment.

What to do about "syncContext completed and\/or Request lifecycle recycled"?

Mon, 16 Nov 2020 22:03:14 +0000

We get into a state where we see a lot of syncContext completed and\/or Request lifecycle recycled

What does it mean?
What can be done about it?

The stack trace is:

"Trace": [
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "state",
                "AsyncContextState.java",
                52
            ],
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "getResponse",
                "AsyncContextState.java",
                118
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invokeStatic",
                "jetty.clj",
                326
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invoke",
                "jetty.clj",
                323
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invokeStatic",
                "jetty.clj",
                351
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invoke",
                "jetty.clj",
                348
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "fail_request",
                "http_endpoint.clj",
                126
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "respond",
                "http_endpoint.clj",
                147
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127$fn__13129",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invokeStatic",
                "ioc_macros.clj",
                978
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invoke",
                "ioc_macros.clj",
                977
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invokeStatic",
                "ioc_macros.clj",
                982
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invoke",
                "ioc_macros.clj",
                980
            ],
            [
                "clojure.core.async.impl.ioc_macros$take_BANG_$fn__5769",
                "invoke",
                "ioc_macros.clj",
                991
            ],
            [
                "clojure.core.async.impl.channels.ManyToManyChannel$fn__589$fn__590",
                "invoke",
                "channels.clj",
                95
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor",
                "runWorker",
                "ThreadPoolExecutor.java",
                1149
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor$Worker",
                "run",
                "ThreadPoolExecutor.java",
                624
            ],
            [
                "clojure.core.async.impl.concurrent$counted_thread_factory$reify__492$fn__493",
                "invoke",
                "concurrent.clj",
                29
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.lang.Thread",
                "run",
                "Thread.java",
                748
            ]
        ]

Provide way to interact with a datomic database before making the system available

Mon, 02 Nov 2020 15:45:42 +0000

Currently in Cloud there is no way to interact with a datomic database before making the system available for external requests.

Potential uses:

deploy the same code everywhere (dev & prod) and it does db-ensure
actions
load data and/or schema prior to the system being live
start components or other system-management tooling
provide (easy) way to integrate startup (e.g. mount/system-component)
provide status and ability to know when pieces are started so you can finish app initialization

Does dev-local/import-cloud handle failure in the middle of an import?

Fri, 30 Oct 2020 16:21:53 +0000

When running an import-cloud, does dev-local gracefully handle import-cloud failing in the middle of the import? i.e., it fails (gets killed or throws) and I can call import-cloud again without worrying about corrupt data, missing data, etc.

Expose an alt-port transactor configuration property

Sun, 25 Oct 2020 19:45:14 +0000

The alt-host configuration property is great and helps in many scenarios, but not having an equivalent for the port (alt-port?) is a blocker in some other scenarios. It would be great to have such property, which would default to the same port as the transactor is listening.

Allow for datomic cloud installs into pre-existing VPCs

Sun, 25 Oct 2020 19:29:15 +0000

The default Marketplace/CF config is great for getting started, more greenfield usage, etc. But it doesn't accomodate more complex (but common) situations. For instance, we've just completed a complete overhaul of our AWS infrastructure. The project had kind of shuffled into AWS. Started with one account, with lifecycle environments only segregated via security groups. After accepting that that was a VeryBadThing, went to VPC's per environment, with AWS account segregation (one for production and pre-production, another for all dev lifecycle environments), we've now moved to something a bit more sophisticated with accounts/vpcs per lifecycle environments, dedicated shared accounts/VPCs for ingress, data vault/logging, management, etc with IPSec tunnels between them all, as opposed to VPC peering.

Sorry about the excruciating detail, but just wanted to provide context. So, in this scenario, shoehorning in Datomic Cloud becomes a bit difficult. Especially in conjunction with the current app/system assumptions (bring on those multiple query groups please! :)). For instance, my plan was to give each of my devs their own sandbox Solo instance, this won't scale well (there are 40 of them ;)) of they each need their own VPC, never mind the other issues. And I guess that might be basically a separate but related request, where one could stand up multiple 'systems' in a single VPC

Store values encrypted, allowing key rotation.

Sun, 25 Oct 2020 19:14:02 +0000

In our application we have a requirement to store certain fields encrypted. This encryption is done in the application. However, we would like to be able to rotate the encryption key should it become compromised, and retain the historical query features that you would get with unencrypted storage. In order to rotate the key, we have to excise the data and re-transact it with the new key, or store the encrypted values outside of Datomic. It would be nice to have an encryption API that enabled this use case.

Create DB from sequence of transactions

Sun, 25 Oct 2020 18:35:40 +0000

Given an input sequence of transactions expressed as EDN or some other format, provide a process to create a new Datomic database "offline" as quickly as possible, either in a supported storage backend or as a backup suitable for restore-db.

The input transactions will use real Entity IDs, not tempids. Provide public APIs for constructing Entity IDs correctly.

No other transactions or queries can be performed while the new database is being built. The input transactions cannot call any transaction functions.

Similarly, provide an "export" process to write out the transaction log of an existing database to a file in the same format.

This is essentially an optimization of the current process for bulk imports with a Transactor, with the added benefit of being able to control the creation of Entity IDs. This might be used, for example, to make a copy of a Datomic database with a filter or transformation applied to the transactions.

Datomic CLI should support AWS SSO

Sun, 25 Oct 2020 18:24:06 +0000

Datomic CLI should be able to load SSO profile from ~/.aws/config. It should be able to authorize Datomic users from AWS SSO console.

Allow for setting transactor properties using environment variables

Sun, 25 Oct 2020 18:08:09 +0000

Datomic should be able to read it's settings from environment variables, following a naming standard like what Spring Boot does. E.g.:

DATOMIC_LICENSE_KEY
DATOMIC_PROTOCOL
DATOMIC_PORT
…

Support AWS dynamo point in time backup in on-prem.

Sun, 25 Oct 2020 16:57:02 +0000

https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/

Currently Datomic supplies backup/restore capability here:

https://docs.datomic.com/on-prem/backup.html

Dynamo's PITR remains untested for use with Datomic DBs. Implementation of this feature would ensure that using Dynamo's PITR causes no issues on a Datomic DB.

Pass along DDB request IDs with transactions

Sun, 25 Oct 2020 16:41:05 +0000

Include the Dynamo DB request IDs with transactions.

In order to provide more information for troubleshooting, the DDB team has indicated that passing along the DDB request ID with transactions would enable further troubleshooting, down to the specific problematic transactions.

New engine for executing java scripts on the transactor

Fri, 23 Oct 2020 13:52:54 +0000

Is it planned to consider another script executing engines than janino? There is script evaluator provided by groovy.lang.GroovyShell which can execute java scripts too. I have compared the time to evaluate the same java script by both engines and groovy engine executes significantly faster. Another positive side effect: support of groovy scripts. I believe using of new script evaluator will make improvements to the time of executing transactor functions written in java

https://docs.groovy-lang.org/latest/html/api/groovy/lang/GroovyShell.html

UPD. How much are java transactor scripts executted longer than clojure transactor scripts?

Can datomic.api/q be used inside or/or-join to query data from other DB?

Fri, 23 Oct 2020 10:53:08 +0000

Hi, when I'm executing query in repl in such way:

(d/q "[:find
 [?Post ...]
 :in
 $
 $comment
 [?Post_value_0 ...]
 :where

[?Post :IModel/typeName ?Post_value_0]
(or-join 
 [?Post ?Comment_text_value_3]

 [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

  [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

)
 [?Post :Post/comments ?Comment_uid_2]
]
" (d/db conn) (d/db connComment) #{"Post"})

The next error is returned:
:db.error/not-an-entity Unable to resolve entity: :Comment/text

My supposing is that query inside datomic.api/q is executed for database returned by (d/db conn) but not (d/db connComment) because if I create :Comment/text for the first database from input list then query is executed without errors.

If to avoid or-join and execute the query in the next way:

(d/q "[:find
 [?Post ...]
 :in
 $
 $comment
 [?Post_value_0 ...]
 :where

[?Post :IModel/typeName ?Post_value_0]
 [?Post ?Comment_text_value_3]

 [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

  [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]


 [?Post :Post/comments ?Comment_uid_2]
]
" (d/db conn) (d/db connComment) #{"Post"})

then query is executed without errors.

Datamic on prem 0.9.6.* version

Delete all except the latest point in time within a backup URI

Thu, 22 Oct 2020 21:46:45 +0000

Also deleting a single point in time might be useful

Provide integrity check on Datomic Backups

Thu, 22 Oct 2020 18:03:10 +0000

A tool to confirm the validity and completeness of a Datomic backup.

Custom Value Types

Thu, 22 Oct 2020 17:58:01 +0000

An attribute could accept a custom comparator and validator (as installed functions) and any fressian/edn-compatible value. We often have a complex type which datomic either forces us to serialize to a string (slow, hacky, space-inefficient), or encode into a long (hacky, tricky), or use a full entity (lost value semantics, need to declare more attributes). Custom value types would solve all this.

Add Cassandra Cluster callback to the Peer

Thu, 22 Oct 2020 17:52:42 +0000

The transactor provides a configurable cassandra cluster callback, but the peers do not. The Cassandra driver depends on old versions of metrics libraries that are incompatible with newer versions used in (for example) clojure metrics. Datastax solution for this requires use of the cluster callback https://docs.datastax.com/en/developer/java-driver/3.5/manual/metrics/#metrics-4-compatibility, but that isn't available in Peers.

Primary-Replica Database Replication

Thu, 22 Oct 2020 17:44:38 +0000

Provide a turnkey Primary-Replica feature for disaster recover. Alternatively provide API such that a "Replication Peer Instance" can: connect to a "primary" database simultaneously connect to a "replica" database (in a different data center) * read transactions from primary sequentially and write those transactions to the replica.

Per-db metrics

Thu, 22 Oct 2020 17:33:06 +0000

Add a "database" dimension to the CloudWatch metric to distinguish which Database is being reported on, including an "all databases" value.

Provide a progress bar/percentage on backup/restore process.

Thu, 22 Oct 2020 17:10:52 +0000

It would be nice to be able to see the percentage complete when the restoring progress is not clear.

Ability to start peer without transactor.

Thu, 22 Oct 2020 16:48:41 +0000

If all transactors are down, no peers can start. Add the ability to start a peer in a read-only mode while continuing to try to connect to the transactor.