Datomic Knowledgebase - Recent questions tagged datomic

Datomic peer - Support SSO

Fri, 13 Feb 2026 17:36:46 +0000

Datomic currently does not support AWS SSO out-of-the-box

This issue is probably because it uses an old version of dynamodb SDK

I believe that this issue can be fixed by upgrading the AWS SDK to v2

Here a workaround:

https://gist.github.com/souenzzo/42060567479ab2dd916769f1d73042fa

Datomic peer on dynamodb - misleading error message

Fri, 13 Feb 2026 16:19:56 +0000

Starting from a simple setup:

{:deps {com.datomic/peer       {:mvn/version "1.0.7491"}
        org.slf4j/slf4j-simple {:mvn/version "2.0.17"}}}

And trying to connect into a dynamodb connection, it fails with a "s3" error:

user=> (require '[datomic.api :as d])
nil
user=> (d/connect "datomic:ddb://us-east-1/prd/db")
[main] INFO datomic.process-monitor - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :phase :begin, :pid 311, :tid 3}
[main] INFO datomic.process-monitor - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :msec 0.696, :phase :end, :pid 311, :tid 3}
[main] INFO datomic.process-monitor - {:metrics/started clojure.core/identity, :pid 311, :tid 3}
[clojure-agent-send-off-pool-0] INFO datomic.domain - {:event :cache/create, :cache-bytes 502267904, :pid 311, :tid 25}
[clojure-agent-send-off-pool-0] INFO datomic.process-monitor - {:AvailableMB 895.0, :ObjectCacheCount 0, :event :metrics, :pid 311, :tid 25}
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:580).
com.amazonaws.services.s3.model.MultiObjectDeleteException

From this error message, it is pretty hard to find that datomic actually requires an extra dynamodb dependency to work.

https://docs.datomic.com/operation/storage.html#dynamodb-aws-peer-dependency

Possible fixes:

A better error message
Adding this error message to datomic docs, as a "common errors" - do google will be able to find.

Datomic on JDK25 - WARNING: A restricted method in java.lang.System has been called

Fri, 13 Feb 2026 16:19:55 +0000

Using this dependencies

{:deps {com.datomic/peer                    {:mvn/version "1.0.7491"}
        org.slf4j/slf4j-nop                 {:mvn/version "2.0.17"}
        com.amazonaws/aws-java-sdk-dynamodb {:mvn/version "1.12.564"}}}

Note: aws-java-sdk-dynamodb has an updated version, but 1.12.564 is the recommended

Generate these warnings:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by io.netty.util.internal.PlatformDependent0$4 (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$4
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by io.netty.util.internal.NativeLibraryUtil in an unnamed module (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

This issue report focus in the first part:

WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by io.netty.util.internal.NativeLibraryUtil in an unnamed module (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)

Trace:
- io.netty/netty-common 4.1.100
- required by org.apache.activemq/artemis-commons 2.31.2
- required by org.apache.activemq/artemis-core-client 2.31.2
- required by com.datomic/peer.

Datomic on JDK25 - WARNING: A terminally deprecated method in sun.misc.Unsafe has been called

Fri, 13 Feb 2026 16:19:51 +0000

Using this dependencies

{:deps {com.datomic/peer                    {:mvn/version "1.0.7491"}
        org.slf4j/slf4j-nop                 {:mvn/version "2.0.17"}
        com.amazonaws/aws-java-sdk-dynamodb {:mvn/version "1.12.564"}}}

Note: aws-java-sdk-dynamodb has an updated version, but 1.12.564 is the recommended

Generate these warnings:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by io.netty.util.internal.PlatformDependent0$4 (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$4
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by io.netty.util.internal.NativeLibraryUtil in an unnamed module (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

This issue report focus in the first part:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by io.netty.util.internal.PlatformDependent0$4 (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)

Trace:
- io.netty/netty-common 4.1.100
- required by org.apache.activemq/artemis-commons 2.31.2
- required by org.apache.activemq/artemis-core-client 2.31.2
- required by com.datomic/peer.

Why does changing the #inst reader make my query fail?

Mon, 20 Oct 2025 17:21:00 +0000

does datomic pro print and read queries during datomic.api/q, maybe as part of its query cache? we’ve changed the #inst reader in our project to return java.time.Instant and now a java.util.Date we pass to a query comes out as a java.time.Instant. Is that expected & documented somewhere?

(require '[datomic.api])

;; this is demonstrating an issue we ran into where overwriting the
;; `#inst` reader broke `datomic.api/q` in surprising ways.

(def schema
  [{:db/ident :exit/datetime
    :db/valueType :db.type/instant
    :db/cardinality :db.cardinality/one}])

(def conn
  (datomic.api/connect (doto "datomic:mem://inst-java-time-readers" datomic.api/create-database)))

@(datomic.api/transact conn schema)

(def db+data
  (:db-after
   @(datomic.api/transact conn [{:exit/datetime #inst "2024"}
                                {:exit/datetime #inst "2025"}
                                {:exit/datetime #inst "2026"}])))

;; this works
(count (datomic.api/q [:find '?f
                       :where
                       '[?f :exit/datetime ?exit]
                       [(list '< '?exit (java.util.Date.))]]
                      db+data))


;; fails silently (returns incorrect number of results)
(binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
  (count (datomic.api/q [:find '?f
                         :where
                         '[?f :exit/datetime ?exit]
                         [(list '< '?exit (java.util.Date.))]]
                        db+data)))

;; fails loudly with processing rule: (q__124130 ?f), message:
;; processing clause: [?f :exit/datetime ?exit], message:
;; java.lang.ClassCastException: class java.time.Instant cannot be
;; cast to class java.util.Date (java.time.Instant and java.util.Date
;; are in module java.base of loader 'bootstrap')
(binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
  (count (datomic.api/q [:find '?f
                         :where
                         '[?f :exit/datetime ?exit]
                         [(list java.util.Date/.before '?exit (java.util.Date.))]]
                        db+data)))

;; My guess is that this is related to datomic's [query
;; caching](https://docs.datomic.com/query/query-executing.html#query-caching)
;; because once a query has been cached, a sucessive call with the
;; same query value and a bound #inst reader succeeds:
(let [query [:find '?f
             :where
             '[?f :exit/datetime ?exit]
             [(list java.util.Date/.before '?exit (java.util.Date.))]]]
  (= (count (datomic.api/q query db+data))
     (binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
       (count (datomic.api/q query db+data)))))

Why does Datomic store all databases in a single datomic_kvs table instead of having a separate table per database?

Mon, 03 Mar 2025 17:06:33 +0000

I am curious about the reasoning behind keeping all Datomic databases in a single datomic_kvs table rather than creating a separate table for each database.

Some disadvantages of the current approach that I see:

More complex maintenance: For example, if VACUUM FULL is needed because one database is highly active and generates a lot of garbage, it will block all Datomic databases, not just the affected one.
Difficult to determine per-database segment size: Since the table contains entries for all databases, simply counting elements in datomic_kvs does not provide a per-database segment count.
Database deletion requires a special utility: Instead of simply dropping a table, removing a database requires a dedicated tool.
Disk space is not immediately freed: Because of the previous point, space is not reclaimed until VACUUM FULL is run, which again locks all Datomic databases.

Given these limitations, what are the key advantages that justify using a single datomic_kvs table instead of separate tables per database?

Also, is there any plan to introduce a mode where each Datomic database has its own separate table?

datomic's spymemcached fork collides with original

Mon, 24 Feb 2025 19:02:52 +0000

Datomic seems to be shading spymemcached. The logger implementation of spymemcached uses the property net.spy.log.LoggerImpl.

Unfortunately that property wasn't shaded properly in datomic - the fork uses the naked property without a prefix. This means setting that property crashes either datomic startup or spymemcached startup, because the same class cannot be used in both places.

"No suitable driver" when starting peer

Fri, 24 Jan 2025 01:10:09 +0000

In the peer container:

root@3249b8857f6f:/opt/datomic# bin/repl
Clojure 1.11.4
user=> (Class/forName "org.postgresql.Driver")
org.postgresql.Driver
user=> (require 'datomic.peer-server)
nil
user=> (datomic.peer-server/-main
"-h" "0.0.0.0"
"-p" "8998"
"-a" "none,none"
"-d" "datomic,datomic:sql://mydb?jdbc=postgresql://catalyst-postgres-1:5432/datomic?user=datomic&password=datomic&jdbc-class=org.postgresql.Driver")
Execution error (SQLException) at java.sql.DriverManager/getDriver (DriverManager.java:299).
No suitable driver
user=>

http-direct configuation not working as per tutorials

Tue, 07 May 2024 19:17:52 +0000

I cannot get http-direct working. The AWS configuration consoles have changed and the tutorial does not appear to work.

Specifically: in the API Gateway->APIs->Create API creation I appear to have to choose between HTTPapi or RESTapi. Between the two, the tutorial appears to follow the REST Api route (before HTTP was an option apparently). If I do that, the load balancer does not show up as an option. The vpc links are created for either HTTP or REST. There is not a LoadBalancer for REST.

I have been able to work through getting a request routed to the LoadBalancer with the preconfigured 8184 listener but I get a Ion excution failed and I'm too far off the tutorial to understand what I need.

How to write the Metaschema file?

Mon, 22 Apr 2024 09:54:04 +0000

I'm learning the Analytics Support of Datomic Pro. I have configured the Transactor, Peer Server, and Presto Server and ensured that they can run properly.
I am trying to use PrestoJDBC to query the content in the Datomic database through SQL statements. By reading the Datomic Pro documentation, I realized that I need to write a Metaschema file to implement this operation. But I didn't find any examples of Metasahema files on the internet.
So I would like to seek help information related to Metaschema file examples.If I misunderstood, please point out my mistake!
I really need your help in the short term!
Thank you!

Possibly security issue with the dev H2 storage?

Wed, 13 Mar 2024 21:58:26 +0000

While trying to understand more about the behaviour of Datomic dev (mainly from a security/encryption point of view), I attempted to connect to datomic.mv.db from an instance of the H2 console. I accidentally tried to log in as "admin" with a blank password and was surprised when I connected successfully. Connecting as "admin" with any password fails, so it seems explicitly to be blank. As far as I can tell I only have public schema access and can't see any transacted data, but I am able to run SQL queries in the console.

Clearly a dev transactor isn't intended to be high security, but admin with a blank password seems risky even for a dev environment. In most SQL environments I work with, the sa/admin account is just disabled if not used. Am I missing something?

As an aside, does the dev transactor use H2's AES encryption or is the resulting data file unencrypted?

import-cloud fails with "No reader function for tag object"

Wed, 14 Feb 2024 18:08:30 +0000

The recent release of version 1.0.277 of datomic-local allowed me to run import-cloud to completion successfully. Unfortunately, the resulting database is unusable. When I try to connect, I get this error message:

{:type java.lang.RuntimeException
:message "No reader function for tag object"
:at [clojure.lang.EdnReader$TaggedReader readTagged "EdnReader.java" 801]}]
:trace
[[clojure.lang.EdnReader$TaggedReader readTagged "EdnReader.java" 801]
[clojure.lang.EdnReader$TaggedReader invoke "EdnReader.java" 783]
[clojure.lang.EdnReader$DispatchReader invoke "EdnReader.java" 549]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$VectorReader invoke "EdnReader.java" 672]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader read "EdnReader.java" 145]
[clojure.lang.EdnReader read "EdnReader.java" 111]
[clojure.lang.EdnReader readString "EdnReader.java" 67]
[clojure.edn$read_string invokeStatic "edn.clj" 46]
[clojure.edn$read_string invokeStatic "edn.clj" 37]
[clojure.edn$read_string invoke "edn.clj" 37]
[datomic.dev_local.btindex_set$read_latest_root invokeStatic "btindex_set.clj" 99]
[datomic.dev_local.btindex_set$read_latest_root invoke "btindex_set.clj" 95]
[datomic.dev_local.btindex_set$current_index_files invokeStatic "btindex_set.clj" 149]
[datomic.dev_local.btindex_set$current_index_files invoke "btindex_set.clj" 146]
[datomic.dev_local.btindex_set$old_index_files invokeStatic "btindex_set.clj" 159]
[datomic.dev_local.btindex_set$old_index_files invoke "btindex_set.clj" 156]
[datomic.dev_local.btindex_set$delete_old_index_files invokeStatic "btindex_set.clj" 176]
[datomic.dev_local.btindex_set$delete_old_index_files invoke "btindex_set.clj" 169]
[datomic.dev_local.btindex_db$delete_unused_files invokeStatic "btindex_db.clj" 604]
[datomic.dev_local.btindex_db$delete_unused_files invoke "btindex_db.clj" 602]
[datomic.dev_local.impl$create_durable_connection invokeStatic "impl.clj" 189]
[datomic.dev_local.impl$create_durable_connection invoke "impl.clj" 187]
[datomic.dev_local.impl$create_connection invokeStatic "impl.clj" 205]
[datomic.dev_local.impl$create_connection invoke "impl.clj" 201]
[datomic.dev_local.impl.DurableClient connect "impl.clj" 271]
[datomic.client.api$connect invokeStatic "api.clj" 151]
[datomic.client.api$connect invoke "api.clj" 138]
[st.system$ensure_db_BANG_ invokeStatic "system.clj" 117]
[st.system$ensure_db_BANG_ invoke "system.clj" 114]
[st.system$eval57307$fn__57309$fn__57311 invoke "system.clj" 122]
[clojure.core$mapv$fn__8535 invoke "core.clj" 6979]
...

Note that I use a custom credentials provider and it appears as an opaque java object in the resulting db-root-00000000 file.

Are DB values obtained from a single conn guaranteed to include transactions submitted against the same connection?

Wed, 31 Jan 2024 05:03:55 +0000

It stands to reason that all transactions submitted (via datomic.api/transact) and transacted (demonstrated via deref) on a single peer would be included in all subsequent DB values obtained (via datomic.api/db) on the Peer, but I'd love to know if that is guaranteed.

For example, if a test transacts data resulting in some T, and the rest of the test only cares that it runs against a DB value with T' >= T, is the following sufficient?:

@(datomic.api/transact conn tx) ; T (let [db' (datomic.api/db conn)] ; T' >= T? ,,(do-something-with db')

Or is it strictly necessary to do:

(let [{:keys [db-after]} @(datomic.api/transact conn tx)] ,,(do-something-with db-after))

I know that the former tends to work, but I'm particularly interested if it's guaranteed to work.

I do understand that additional transactions from other threads / clients / etc. may have been included in db' in the first scenario, so I realize the code is not equivalent in the general case. I'm only concerned about tx having been transacted and included in the fetched db' value.

Thank you!

(Apologies for formatting—I can't seem to convince this thing to care about my whitespace.)

Equivalents for d/with; transaction function test scaffold story when using cloud

Wed, 01 Nov 2023 11:24:08 +0000

Is there absolutely no way to access the peer api on datomic cloud?

My thinking is that, conceptually, since it uses clients, that implies that peer servers do exist.

I realise that light horizontal scaling is the point in DC, but the peer api, and in particular d/with I'm finding important to allow test scaffolds for transaction functions.

So the thought here is it would be nice to be able to access the richer peer api just for one off tasks, such as managing schema or even here making speculative databases so one can create a test scaffold to test a more complex transaction function.

I don't know what the test scaffold story is with DC or if there is one. Bit of a dark area on the map for me at the moment so any suggestions would be appreciated.

I guess one last-resort option is to develop using Pro and test transaction functions that way. Then copy them over to DC. I have found though that in a previous iteration trying to think about Pro and Cloud at the same time as designing for your domain is something I would want to avoid where possible.

Update: I've just realised that the client api does actually include d/with, but I'll leave the question up anyway, in case anyone has any related thoughts they'd like to share.

Further Update: looking back at my initial inept attempts to build a datomic system and schema which utilised transaction functions to avoid race conditions given a complex content hierarchy in the domain I am modelling, it looks like the reason I felt forced to use the Peer API and not Client, for those Database Functions, was the absence of d/function in the Client API. I wonder if Client API actually is fully capable in terms of installing and testing database functions, but that wasn't clear to me at the time. So much is required to be able to develop in Datomic. It might be that my missing a little detail of Clojure/Java design (an easy syntactic alternative to d/function?) ended up pushing my design decision in an unnecessary direction.
I suppose the answer may be, re. DB functions - "Client API yes", "Cloud no"?
(Illustrating the point that it is best not to try to think about Pro and Cloud at the same time...)

Remaining question: I'm finding it very difficult to find out what the difference is between d/function on the peer API, vs. use of the #db/fn literal. Cannot locate an explanation in the docs. GPT is telling me the latter is for an inline database function which will not be installed, whereas d/function is for installing the fn into the database, but it I know it could be hallucinating in such a case.

Why do values of :db.type/uri return as com.cognitect.transit.impl.URIImpl?

Mon, 02 Oct 2023 03:56:45 +0000

I've recently worked with dev-local try out Datomic and when I moved to Cloud I noticed the following scenario, whenever I transact a fact that is of the java.net.URI type and try to query it back I get com.cognitect.transit.impl.URIImpl back as the type, I have found no documentation about this, I was expecting that the types remain the same, is there any way to disable this behavior or at least convert back to java.net.URI?

This matters because I had some code break because it was expecting java.net.URI and because this behavior is undocumented (or if documented inaccessible by searching the Datomic documentation and search engines)

Thank you for your time

deps.edn

org.clojure/clojure {:mvn/version "1.11.1"}
com.datomic/client-cloud {:mvn/version "1.0.123"}

Why is this error message saying the entity can't be resolved?

Thu, 04 May 2023 22:53:42 +0000

Here is the error message:

cognitect.anomalies/message: "Unable to resolve entity: 1.7592186046217E13 in datom [\"shape_VVVVVVVVVVVVVVVV_0_1\" :shape/source-shape 1.7592186046217E13]"
                           datom: ["shape_VVVVVVVVVVVVVVVV_0_1" :shape/source-shape 1.7592186046217E13]
                        db/error: :db.error/not-an-entity
                             dbs: [{:database-id
                                    "datomic:dev://datomic-transactor:4334/blah",
                                    :t 1818,
                                    :next-t 1819,
                                    :history false}]
                          entity: 1.7592186046217E13

I can query for the :db/id 17592186046217, so I know it exists. My question is why can't Datomic find it? There's only one transaction and it failed, so it is guaranteed that entity 17592186046217 existed before the transaction as well.

This is the schema for :shape/source-shape:

{:db/ident :shape/source-shape
 :db/valueType :db.type/ref  
 :db/cardinality :db.cardinality/one}

The version of Datomic is datomic-pro-1.0.6269.

What defines a datomic on-prem system?

Wed, 29 Mar 2023 20:55:33 +0000

The price for an on-prem pro license is for one system. What defines a system? Does a single system include a transactor, a stand-by transactor and multiple peers?

Request Entity Too Large - clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

Tue, 21 Mar 2023 12:26:20 +0000

There seems to be a bug in Datomic 990-9202 that breaks the d/tx-range function.

Earlier I could use the tx-range function the fetch data from my database.

But after upgrading to Datomic 990-9202 I get and exception when a transaction has too much data.

`
clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

cognitect.anomalies/category: :cognitect.anomalies/incorrect
 cognitect.anomalies/message: "Response body did not conform to Datomic client protocol"
                 http-result: {:status 413,
                               :headers
                               {"apigw-requestid" "Vg4UxisCDoEEJsQ=",
                                "server" "Jetty(9.4.44.v20210927)",
                                "connection" "keep-alive",
                                "content-length" "38",
                                "date" "Tue, 19 Jul 2022 12:17:47 GMT",
                                "content-type" "application/json"},
                               :body "{\"message\":\"Request Entity Too Large\"}"}

Error connecting to datomic cloud

Mon, 12 Dec 2022 00:30:29 +0000

I'm trying to connect with the datomic cloud, but when I connect with the client I get the following error:
Execution error (NoClassDefFoundError) at com.amazonaws.services.s3.AmazonS3ClientBuilder/standard (AmazonS3ClientBuilder.java:46).
Could not initialize class com.amazonaws.services.s3.S3CredentialsProviderChain

How do you add relationship properties, e.g. the time a person met another person?

Mon, 07 Nov 2022 08:35:28 +0000

I have some experience with Neo4j / cypher. In cypher relationships are first-class citizens, so they can have properties, which I really like.

How do I do something similar with datalog? Do I have to have a "mapping entity" in between two entities that have a relationship?

Hi guys, I need to retrieve id's from all users in database that are older then 30 days, every user has creation date.

Mon, 01 Aug 2022 22:13:08 +0000

Hi guys,
I need to retrieve id's from all users in database that are older then 30 days, every user has creation date.,
(let [conn (client/get-conn)

  db (client/db)
  last-month (t/minus (t/now) (t/month 1))
  q '{:find  [(pull ?eid [*])]
      :where [[?eid :user/created]]}]

;;.............
)
I have last-month #inst date, I just dont know how to get all users "older" then 30 days.
I am new in datomic so,
if someone can help, thank you in advance...

How to add new entities (as references) to the many attribute of an existing entity?

Wed, 27 Jul 2022 19:01:04 +0000

By way of example I'm going to use a drawing as the existing entity which is to have two new shapes added to it. This is the tx that is failing:
[{:shape/id "BKF806TXXTAFWII0", :db/id "22NF08ZVGH9N7QGG_0"} {:shape/id "YKIZU4CJC0JXJKVQ", :db/id "22NF08ZVGH9N7QGG_1"} [:db/add 17592186047451 :drawing/shapes ["22NF08ZVGH9N7QGG_0" "22NF08ZVGH9N7QGG_1"]]]

As you can see the existing drawing has :db/id of 17592186047451. I'm trying to create two new shapes and add them to the many cardinality attribute :drawing/shapes. "22NF08ZVGH9N7QGG_0" and "22NF08ZVGH9N7QGG_1" are tempids that are supposed to be translated to the same new :db/ids wherever they appear.

I should say that the drawing doesn't have any existing shapes. If it did I would expect them to be orphaned.

This is the error message I'm getting:

Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
Cannot interpret as a keyword: 22NF08ZVGH9N7QGG_0, no leading :

Something similar I've also tried:
[{:shape/id "9HTL5BMMHT6QUJM6", :db/id "22NF08ZVGH9N7QGG_0"} {:shape/id "O5UB9IG9UB8KDVA2", :db/id "22NF08ZVGH9N7QGG_1"} [:db/add 17592186047451 :drawing/shapes "22NF08ZVGH9N7QGG_0"] [:db/add 17592186047451 :drawing/shapes "22NF08ZVGH9N7QGG_1"]]

Gives this error message:
Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
Two datoms in the same transaction conflict
{:d1
[17592186047451 :drawing/shapes 17592186049126 13194139538021 true],
:d2
[17592186047451 :drawing/shapes 17592186049127 13194139538021 true]}

This gives the same error message as the first attempt:
[{:db/id 17592186047451, :drawing/shapes ["22NF08ZVGH9N7QGG_0" "22NF08ZVGH9N7QGG_1"]} {:shape/id "3DZYWHEPQIAELF25", :db/id "22NF08ZVGH9N7QGG_0"} {:shape/id "GJ804SOOU36YQX6Y", :db/id "22NF08ZVGH9N7QGG_1"}]

Getting rid of the tempids by doing it in a nested way:
[{:db/id 17592186047451, :drawing/shapes [{:shape/id "GEMRMRFG0E6N262M"} {:shape/id "HTH2C7R90BQFFNXT"}]}]

Just yields this error message:
Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
Unable to resolve entity: {:shape/id "GEMRMRFG0E6N262M"}

java.lang.IllegalArgumentException: Cannot resolve key: 24a96e20-f526-4f7f-ba38-4f684caa5607

Mon, 06 Jun 2022 19:37:30 +0000

I am trying to implement resursive rule (loglcal OR) by following this video - Lucas Cavalcanti & Edward Wible - Exploring four hidden superpowers of Datomic

Here the code

(defn owns? [cid pid db]
(d/q '{:find [?pur]

     :in [$ ?cus-id ?pur %]
     :where
     [(owns? ?cus-id ?pur)]}
db cid [:purchase/id pid] owner-rules))

(comment
(owns?

#uuid "0fb7ea94-44af-46fa-98ca-0ddb5eb23123"
#uuid "24a96e20-f526-4f7f-ba38-4f684caa5607"
(d/db bank-conn)))

While running the above example I am getting this error:
java.lang.IllegalArgumentException Cannot resolve key: 24a96e20-f526-4f7f-ba38-4f684caa5607

Here is the full code:

(def purchase-schema
[{:db/ident :purchase/id

:db/doc "The purchase ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :purchase/account

:db/doc "Purchase Account"
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}])

(def account-schema
[{:db/ident :account/id

:db/doc "The Account ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :account/customer

:db/doc "Customer who own's this account"
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}])

(def customer-schema
[{:db/ident :customer/id

:db/doc "Customer ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :customer/name

:db/doc "Customer name"
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one}])

;;
;; Create client
(def client (d/client {:server-type :dev-local

                   :system "dev"}))

;; Create connections
(def bank-conn (d/connect client {:db-name "bank"}))

;; Transact schemas
(comment
(d/transact bank-conn {:tx-data purchase-schema})
(d/transact bank-conn {:tx-data account-schema})
(d/transact bank-conn {:tx-data customer-schema})
(d/tx-range bank-conn {:start 6}))

(comment
(java.util.UUID/randomUUID))

;; Transact fake data
(def cid #uuid "0fb7ea94-44af-46fa-98ca-0ddb5eb23123")
(def sample-customer {:customer/id cid

                  :customer/name "Jon Snow"})

(def aid #uuid "c579a9f6-f6c3-4718-8ecf-c3003021ee9a")
(def sample-account {:account/id aid

                 :account/customer [:customer/id cid]})

(def pid #uuid "24a96e20-f526-4f7f-ba38-4f684caa5607")
(def sample-purchase {:purchase/id pid

                  :purchase/account [:account/id aid]})

(comment
(d/transact bank-conn {:tx-data [sample-customer]})
(d/transact bank-conn {:tx-data [sample-account]})
(d/transact bank-conn {:tx-data [sample-purchase]}))

(comment
(d/q '[:find ?cus

     :in $ ?cid ?pid
     :where
     [?pur :purchase/id ?pid]
     [?pur :purchase/account ?acc]
     [?acc :account/customer ?cus]
     [?cus :customer/id ?cid]]
  (d/db bank-conn)
  (:customer/id sample-customer)
  (:purchase/id sample-purchase)))

;; owns v1
(defn ownsv1? [cid pid db]
(d/q '[:find ?cus

     :in $ ?customer-id ?purchase-id
     :where
     [?pur :purchase/id ?purchase-id]
     [?pur :purchase/account ?acc]
     [?acc :account/customer ?cus]
     [?cus :customer/id ?customer-id]]
db cid pid))

(comment
(ownsv1?

(:customer/id sample-customer)
(:purchase/id sample-purchase)
(d/db bank-conn)))

;; recursive rule (logical OR)
(def owner-rules
'[[(owns? ?cus-id ?e)

 [?e :customer/id ?cus-id]]
[(owns? ?cus-id ?e)
 [?e ?ref-attr ?r]
 (owns? ?cus-id ?r)]])

(defn owns? [cid pid db]
(d/q '{:find [?pur]

     :in [$ ?cus-id ?pur %]
     :where
     [(owns? ?cus-id ?pur)]}
db cid [:purchase/id pid] owner-rules))

;;
(comment
(owns?

(:customer/id sample-customer)
(:purchase/id sample-purchase)
(d/db bank-conn)))

Transactor timeout on transaction function throw

Fri, 27 May 2022 17:54:06 +0000

Version

com.datomic/datomic-pro "1.0.6242"
transactor process 0.9.6045

Setup

Given the following transaction function definition:

(d/transact
 *conn*
 [{:db/ident :demo-throw-fn
   :db/fn (d/function
           {:lang "clojure"
            :params '[db]
            :code '(throw (ex-info "demo" {:type (type #{:one})}))})}])

Execute the transaction function in two environments:

A local environment
An environment where the transactor runs in a separate process

Expectation

Regardless of the environment, transaction function runtime errors are thrown.

Actual

In a local environment, execution of the demo function indeed throws an error - in line with expectations.
In an environment where the transactor runs in a separate process, the transactor times out.

Additional Notes

The hypothesis for the timeout is a second error occurs due to the
(type #{:one}) expression's result not being serializable. Unfortunately, the timeout error does not reveal what was likely known at some point during execution.

Inconsistent return value using set?

Fri, 27 May 2022 17:51:40 +0000

Version

com.datomic/datomic-pro "1.0.6242"
transactor process 0.9.6045

Setup

Given the following transaction function definition.

(d/transact
 *conn*
 [{:db/ident :demo-fn
   :db/fn (d/function
           {:lang "clojure"
            :params '[db]
            :code '(set? #{:one})})}])

Execute the transaction function in two environments:

A local environment
An environment where the transactor runs in a separate process

Expectation

Regardless of the environment, the return value of set? is consistent.

Actual

In a local environment, the function returns true - as expected.
In an environment where the transactor runs in a separate process, the function returns false.

Additional Notes

The data type of a set is different in the two environments,
clojure.lang.PersistentHashSet in local and java.util.HashSet in the other.

Remove com.h2database/h2 from classpath

Tue, 01 Feb 2022 15:36:23 +0000

When running Datomic on the Postgres storage, we wanted to exclude H2 from the Peer classpath, but this turns out not to be possible.

With the following deps.edn

{:deps      {com.datomic/datomic-pro   {:mvn/version "1.0.6362"
                                        :exclusions  [com.h2database/h2]}
             org.postgresql/postgresql {:mvn/version "42.3.1"}}
 :mvn/repos {"my.datomic.com" {:url "https://my.datomic.com/repo"}}}

trying to create the database leads to

(d/create-database "datomic:sql://test?jdbc:postgresql://localhost:5432/datomic?user=datomic")
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:387).
org.h2.tools.Server

Is there another way to accomplish this?

The reason we want to exclude h2 from the classpath is because the maven artifact for h2 has several HIGH CVEs reported on it (https://mvnrepository.com/artifact/com.h2database/h2/1.3.172). I understand that Datomic does not run the H2 console and is also does not involve SQLXML, so it is probably not vulnerable to the specific CVEs, however, it would be an easy way to remove a dev-only dependency in a production deployment.

Retrieving tx from a entity - attribute pair without doing a query on history db

Fri, 07 Jan 2022 10:09:22 +0000

Given an entity and one of its attributes, is there a way to see the tx that added it without doing a query on history db?

i.e. something other than

(d/q
       '[:find ?tx ?attr ?val ?added
         :in $ ?e
         :where
         [?e ?attr ?val ?tx ?added]]
       (d/history my-db)
       my-eid)

How to retract entities that reference a retracted entity

Thu, 06 Jan 2022 09:52:00 +0000

In my database model, I have several entities referencing the user entity.
When I retract the user entity using :db/retractEntity, the user is correctly retracted, as well as its refs in the other entities, for example :message/user.
For some entities, when the user is retracted the entity itself should also be retracted, but since the entity is not a component of the user, as the relation is the other way around, I did not find a way to retract the entire entity.
Is there a way to make this happen for some entities?

Can I run recent versions of Datomic in Windows?

Tue, 31 Aug 2021 23:31:45 +0000

I'm not sure if Windows is (or ever was?) a supported platform for Datomic On-Prem, but at the moment when I try to start a REPL, transactor, or peer server in any recent version I get:

"The input line is too long. The syntax of the command is incorrect."

My Datomic path is short (C:\Datomic). I'm guessing the error results from classpath.cmd trying to build a CP_LIST that comprises all 400+ files in lib, though this behaviour is consistent as far back as 0.9.5927, which only has 240 jars in lib; things work as expected in 0.9.5786 (which has 232 jars) and earlier.

I'd rather not use WSL/Docker etc. at this time - can anyone advise if there's a native solution that will allow me to use a version more recent than 2018?

Retracting entities, including tuples with refs

Tue, 31 Aug 2021 13:36:44 +0000

In my schema I have attributes of type tuple that has :db.type/ref as one of the tuple's types.
My usecase is maintaining a set of ordered to-many relations.

When I retract an entity using :db/retractEntity, the tuple attrs that refer to the retracted entity still remain, continuing to refer to a now non-existent entity.
I'm not sure whether this is the intended behaviour, and if so, what would be a standard approach to clean up these tuples? My intention would be for these tuples to be completely redacted. As I'm using this in a cardinality-many setup, I'd expect the attribute to retain all tuple values, except the one that does include the ref to the retracted entity in any position.
Should I be writing a custom retractEntity-like transaction function?
Thanks!

`Unable to read keyfile` error when trying to initialize Datomic Cloud with Client API

Mon, 23 Aug 2021 13:25:08 +0000

Howdy all.

I'm testing setting up Datomic Cloud using an IntelliJ IDE. I'm following the Client API tut from Datomic but am stuck initializing the client.

The spec from an API client is here, and the tut is here, under the step Using Datomic Cloud.

So the tut says to init a client like so:

(require '[datomic.client.api :as d])
(def cfg {:server-type :ion
      :region "<your AWS Region>" ;; e.g. us-east-1
      :system "<system name>"
      :creds-profile "<your_aws_profile_if_not_using_the_default>"
      :endpoint "<your endpoint>"})

They say to include an AWS profile if not using the default. I am using the default as far as I know--I'm not part of an org in AWS.

This is the (partially redacted) code from my tutorial.core namespace, where I'm trying to init Datomic:

(ns tutorial.core   
(:gen-class))
(require '[datomic.client.api :as d])
(def cfg {:server-type :cloud
      :region "us-east-2"
      :system "roam"
      :endpoint "https://API_ID.execute-api.us-east-2.amazonaws.com"
      })
(def client (d/client cfg))
(d/create-database client {:db-name "blocks"})
(d/connect client {:db-name "blocks"})

However, I'm getting an error from Datomic like: Forbidden to read keyfile at s3://URL/roam/datomic/access/admin/.keys. Make sure that your endpoint is correct, and that your ambient AWS credentials allow you to GetObject on the keyfile.

Do I need some sort of credential? Could anything else be causing this error? I got the endpoint URL from the ClientApiGatewayEndpoint in my CloudFormation Datomic stack.

Please let me know if I should provide more info! Thanks.

Query values for more than one entity in a single datomic query (Roam Alpha Api)

Sat, 26 Jun 2021 19:47:35 +0000

I'd like to in a single datomic/datalog query get output relating to multiple entities

E.g.

In my db (querying Roam graph via RoamAlphaApi) for one value ("YjpbFUsTx") I can do this:

[:find ?e ?stri :where [?e :block/uid "YjpbFUsTx"][?e :block/string ?stri]]

gives me

But I want to pass on two values "YjpbFUsTx" and "TgpgOssBM"

to give me

How do I do that?

I've tried several variations but none give any output.

Useful reference: https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html .

ClassCastException, d/with inside transact function

Thu, 24 Jun 2021 14:57:51 +0000

Calling a transaction function that uses d/with in a dev-local, filesystem (i.e., not :storage-dir :mem) throws a ClassCastException on internal Datomic code.

Environment

com.datomic/dev-local {:mvn/version "0.9.232"}
client-cloud 0.8.105

History

Example with exception using file system dev-local.

(def client (d/client {:server-type :dev-local
                         :storage-dir "/tmp/cce-bug3"
                         :system      "test"}))
=> #'user/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'user/conn
(d/transact conn {:tx-data [{:db/ident       :user/name
                               :db/valueType   :db.type/string
                               :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "test",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "test",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T00:49:46.400-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(defn my-tx-fn
    [db argm]
    (let [tx-report (d/with db {:tx-data [{:user/name "asd"}]})]
      []))
=> #'user/my-tx-fn
(d/transact conn {:tx-data [(list 'user/my-tx-fn {})]})
Execution error (ClassCastException) at datomic.dev-local.tx/datom-lookup-valfn (tx.clj:397).
class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')

Working memdb example.

(def client-memdb (d/client {:server-type :dev-local
                               :storage-dir :mem
                               :system      "test"}))
=> #'user/client-memdb
(d/create-database client-memdb {:db-name "test"})
=> true
(def conn-memdb (d/connect client-memdb {:db-name "test"}))
=> #'user/conn-memdb
(d/transact conn-memdb {:tx-data [{:db/ident       :user/name
                                     :db/valueType   :db.type/string
                                     :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:02:57.965-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(d/transact conn-memdb {:tx-data [(list 'user/my-tx-fn {})]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 6,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 7,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533319 50 #inst"2021-05-09T01:03:08.042-00:00" 13194139533319 true]],
 :tempids {}}

Expectation

Transaction functions that use d/with do not throw a ClassCastException. Additionally, the behavior of d/with in all Datomic environments (Cloud, File system, Memdb) is consistent. Given a previous Datomic Cloud release fixed a problem with using with-db from an Ion, it seems using d/with in a transaction function is a supported behavior.

Actual

The final d/transact transaction function throws a ClassCastException when using d/with on some internal Datomic code. This only occurs when using a dev-local, file system environment.

Full stacktrace is pasted below.

#error{:cause "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
       :via [{:type java.lang.ClassCastException,
              :message "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
              :at [datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]}],
       :trace [[datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]
               [datomic.dev_local.tx$datom_lookup_valfn invoke "tx.clj" 397]
               [datomic.dev_local.local_log.LocalLog valAt "local_log.clj" 56]
               [clojure.lang.RT get "RT.java" 760]
               [datomic.dev_local.btindex.BTIndex cons "btindex.clj" 281]
               [clojure.lang.RT conj "RT.java" 677]
               [clojure.core$conj__5390 invokeStatic "core.clj" 85]
               [clojure.core$conj__5390 invoke "core.clj" 82]
               [datomic.core.db.Db addData "db.clj" 2322]
               [datomic.core.db$add_ensured_data invokeStatic "db.clj" 3353]
               [datomic.core.db$add_ensured_data invoke "db.clj" 3351]
               [datomic.core.db$with_tx invokeStatic "db.clj" 3370]
               [datomic.core.db$with_tx invoke "db.clj" 3357]
               [datomic.core.db.Db with "db.clj" 2164]
               [datomic.core.local_db$fn__25633 invokeStatic "local_db.clj" 67]
               [datomic.core.local_db$fn__25633 invoke "local_db.clj" 24]
               [datomic.client.api.protocols$fn__11959$G__11877__11966 invoke "protocols.clj" 126]
               [datomic.client.api$with invokeStatic "api.clj" 363]
               [datomic.client.api$with invoke "api.clj" 353]
               [user$my_tx_fn invokeStatic "user.clj" 3]
               [user$my_tx_fn invoke "user.clj" 20]
               [clojure.lang.AFn applyToHelper "AFn.java" 156]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.lang.Var applyTo "Var.java" 705]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$apply invoke "core.clj" 660]
               [datomic.core.db.ProcessExpander inject "db.clj" 3229]
               [datomic.core.db.ProcessInpoint inject "db.clj" 2950]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586$fn__18587 invoke "btindex_db.clj" 440]
               [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
               [clojure.core$reduce invokeStatic "core.clj" 6827]
               [clojure.core$reduce invoke "core.clj" 6810]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586 invoke "btindex_db.clj" 440]
               [datomic.dev_local.btindex_db$expand_tx invokeStatic "btindex_db.clj" 444]
               [datomic.dev_local.btindex_db$expand_tx invoke "btindex_db.clj" 429]
               [datomic.dev_local.btindex_db$with_tx invokeStatic "btindex_db.clj" 468]
               [datomic.dev_local.btindex_db$with_tx invoke "btindex_db.clj" 461]
               [datomic.dev_local.impl.DurableConnection transact "impl.clj" 161]
               [datomic.client.api$transact invokeStatic "api.clj" 200]
               [datomic.client.api$transact invoke "api.clj" 183]
               [user$eval124260 invokeStatic "user.clj" 25]
               [user$eval124260 invoke "user.clj" 25]
               [clojure.lang.Compiler eval "Compiler.java" 7177]
               [clojure.lang.Compiler eval "Compiler.java" 7132]
               [clojure.core$eval invokeStatic "core.clj" 3214]
               [clojure.core$eval invoke "core.clj" 3210]
               [nrepl.middleware.interruptible_eval$evaluate$fn__959 invoke "interruptible_eval.clj" 91]
               [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
               [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
               [clojure.main$repl$fn__9095 invoke "main.clj" 458]
               [clojure.main$repl invokeStatic "main.clj" 458]
               [clojure.main$repl doInvoke "main.clj" 368]
               [clojure.lang.RestFn invoke "RestFn.java" 1523]
               [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
               [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
               [nrepl.middleware.interruptible_eval$interruptible_eval$fn__985$fn__989
                invoke
                "interruptible_eval.clj"
                155]
               [clojure.lang.AFn run "AFn.java" 22]
               [nrepl.middleware.session$session_exec$main_loop__1086$fn__1090 invoke "session.clj" 190]
               [nrepl.middleware.session$session_exec$main_loop__1086 invoke "session.clj" 189]
               [clojure.lang.AFn run "AFn.java" 22]
               [java.lang.Thread run "Thread.java" 829]]}

Evidence

The previously pasted example is reproducible and always throws a ClassCastException.

Impact

We are unable to test functionality in a consistent manner. Local setups using dev-local file systems will not function as expected. Developers are unable to test features in the same way they will work in production.

d/client external server communication

Thu, 24 Jun 2021 14:55:59 +0000

Environment

com.datomic/client-cloud 0.8.105

History

Get the docstring for d/client.

(clojure.repl/doc d/client)
-------------------------
datomic.client.api/client
([arg-map])
  Create a client for a Datomic system. This function does not
communicate with a server and returns immediately.

For a cloud system, arg-map requires:

  :server-type   - :cloud
  :region        - AWS region, e.g. "us-east-1"
  :system        - your system name
  :endpoint      - IP address of your system or query group

Optionally, a cloud system arg-map accepts:

  :creds-provider  - instance of com.amazonaws.auth.AWSCredentialsProvider. Defaults to DefaultAWSCredentialsProviderChain
  :creds-profile   - name of an AWS Named Profile. See http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
  :proxy-port      - local port for SSH tunnel to bastion 

  Note: :creds-provider and :creds-profile are mutually exclusive, providing both will result in an error.

For a dev-local system, arg-map comprises:

  :server-type   - :dev-local (required)
  :system        - a system name (required)
  :storage-dir   - optional, overrides :storage-dir in ~/.datomic/dev-local.edn

dev-local stores databases under ${storage-dir}/${system}/${db-name}.

For a peer-server system, arg-map requires:

  :server-type   - :peer-server
  :access-key    - access-key from peer server launch
  :secret        - secret from peer server launch
  :endpoint      - peer server host:port
  :validate-hostnames  - false

Returns a client object.

Note the second sentence in the docstring: "This function does not communicate with a server and returns immediately." Next call d/client with an endpoint that does not exist.

(d/client {:server-type :ion
             :system      "my-system"
             :endpoint    "http://i-dont-exist.datomic.net:8182"
             :region      "us-west-2"})
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to http://i-dont-exist.datomic.net:8182

Expectation

Calling d/client does not communicate with a server.

Actual

Calling d/client does communicate with a server.

Evidence

The above reproducible example indicates the d/client function call is attempted to communicate the the passing in :endpoint string given the exception thrown.

Impact

The d/client docstring misleads the user by saying no server communication will occur when it does occur. The result is calls to d/client can fail in unexpected ways when an external server communication is not accounted for. For example, we spin up Datomic Cloud query groups dynamically and attempt to communicate with the newly spun up cluster. Until the cluster is up and healthy, we expect calls to fail at d/connect with an unavailable anomaly, and instead the calls would fail at d/client with an unavailable anomaly.

DDB size reduction after restore

Tue, 25 May 2021 09:19:36 +0000

Hello. We have a large datomic database (15B datoms, 20TB DDB size). We have done a backup and a restore as a test and the resulting DDB is 500GB in size. Is this expected? A couple of datom count queries match on both databases, but the reduction from 20TB to 500GB still look a bit scary

Should you cache d/connect calls?

Wed, 06 Jan 2021 03:21:16 +0000

Is it a best practice to cache the result of calls to d/connect, as seen in the Ion event-example ns? What are the trade-offs with not doing so?

Add default :timeout to client api ns docstring

Thu, 24 Dec 2020 00:12:44 +0000

It'd be great to have the default value for :timeout (60000) included in the datomic.client.api ns docstring. Currently this value can only be found in the docs on the website. As of client-cloud 0.8.102, the ns docstring specifies the default value for :limit.

Can you use d/with inside a Cloud transaction function?

Tue, 15 Dec 2020 16:58:24 +0000

Can you use d/with inside a Cloud transaction function? It's not clear from the the docs if the db passed to the tx fn is one created by d/with-db.

Does deleting a database permanently remove all data stored in said database?

Fri, 04 Dec 2020 17:43:04 +0000

Does calling d/delete-database permanently remove all data stored in that database (historical or otherwise)?

How do I remove a unique tuple constraint?

Fri, 20 Nov 2020 21:12:31 +0000

I have a tuple constraint as follows:

#:db{:id 182,
 :ident :ps2st/product-set+style-tier,
 :valueType :db.type/tuple,
 :cardinality :db.cardinality/one,
 :unique :db.unique/identity,
 :tupleAttrs [:ps2st/product-set :ps2st/style-tier]}

This constraint is overly restrictive. I read somewhere that tupleAttrs can not be altered, so I'd like to remove the constraint instead and then create a new one with more tuple attrs.

How can I delete a constraint?

Datomic cloud client get-s3-auth-path: Unable to connect to localhost:8182

Thu, 19 Nov 2020 18:55:38 +0000

I could use some help debugging recent connection issues we're having with
a Datomic cloud solo topology. We were working to add analytics support so enabled it in Cloud
Formation, added and synced a config directory and adjusted our internal AWS
subaccounts to access via IAM roles.

After this we're unable to establish a connection through the client even though
it seems like all of the parts are working and wired up. I tried some
debugging by Google
and tried the suggestions of restarting the worker and gateway nodes but
unfortunately that didn't help.

If anyone could provide pointers to what to poke or try next to re-establish
authentication. it would be much appreciated.

Now all the details, first we successfully connect to the access Gateway:
`

$ datomic client access -r us-east-2 datomic-solo
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
[...]
debug1: Authentication succeeded (publickey).
Authenticated to 18.216.250.130 ([18.216.250.130]:22).
debug1: Local connections to LOCALHOST:8182 forwarded to remote address socks:0
debug1: Local forwarding listening on ::1 port 8182.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8182.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

`
The gateway connection seems to work:

$ curl -x socks5h://localhost:8182
http://entry.datomic-solo.us-east-2.datomic.net:8182/
{:s3-auth-path "datomic-solo-storagef7f305e7-wutene4nee-s3datomic-ci7fce3hk2gj"}

But when trying to connect in Clojure ((d/client cfg)), it times out:

Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to localhost:8182

ginkgo.metadata.core=> *e
#error {
 :cause "Unable to connect to localhost:8182"
 :data {:cognitect.anomalies/category :cognitect.anomalies/unavailable, :cognitect.anomalies/message "Total timeout 60000 ms elapsed", :config {:server-type :cloud, :region "us-east-2", :system "datomic-solo", :endpoint "http://entry.datomic-solo.us-east-2.datomic.net:8182", :proxy-port 8182, :endpoint-map {:headers {"host" "entry.datomic-solo.us-east-2.datomic.net:8182"}, :scheme "http", :server-name "entry.datomic-solo.us-east-2.datomic.net", :server-port 8182}}}
 :via
 [{:type java.lang.RuntimeException
   :message "could not start [#'ginkgo.metadata.ferment/conn] due to"
   :at [mount.core$up$fn__247 invoke "core.cljc" 94]}
  {:type clojure.lang.ExceptionInfo
   :message "Unable to connect to localhost:8182"
   :data {:cognitect.anomalies/category :cognitect.anomalies/unavailable, :cognitect.anomalies/message "Total timeout 60000 ms elapsed", :config {:server-type :cloud, :region "us-east-2", :system "datomic-solo", :endpoint "http://entry.datomic-solo.us-east-2.datomic.net:8182", :proxy-port 8182, :endpoint-map {:headers {"host" "entry.datomic-solo.us-east-2.datomic.net:8182"}, :scheme "http", :server-name "entry.datomic-solo.us-east-2.datomic.net", :server-port 8182}}}
   :at [datomic.client.impl.cloud$get_s3_auth_path invokeStatic "cloud.clj" 179]}]
 :trace
 [[datomic.client.impl.cloud$get_s3_auth_path invokeStatic "cloud.clj" 179]
  [datomic.client.impl.cloud$get_s3_auth_path invoke "cloud.clj" 170]
  [datomic.client.impl.cloud$create_client invokeStatic "cloud.clj" 211]
  [datomic.client.impl.cloud$create_client invoke "cloud.clj" 194]
```
On the access client side it seems to connect and forward okay:
```
debug1: Connection to port 8182 forwarding to socks port 0 requested.
debug1: channel 2: new [dynamic-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 8182 for
entry.datomic-solo.us-east-2.datomic.net port 8182, connect from 127.0.0.1 port
59870 to 127.0.0.1 port 8182, nchannels 3

This is where I'm a bit stuck and not sure where to debug next. Does anyone have
suggestions for what to poke or try next? Thanks much.

What to do about "syncContext completed and\/or Request lifecycle recycled"?

Mon, 16 Nov 2020 22:03:14 +0000

We get into a state where we see a lot of syncContext completed and\/or Request lifecycle recycled

What does it mean?
What can be done about it?

The stack trace is:

"Trace": [
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "state",
                "AsyncContextState.java",
                52
            ],
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "getResponse",
                "AsyncContextState.java",
                118
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invokeStatic",
                "jetty.clj",
                326
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invoke",
                "jetty.clj",
                323
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invokeStatic",
                "jetty.clj",
                351
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invoke",
                "jetty.clj",
                348
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "fail_request",
                "http_endpoint.clj",
                126
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "respond",
                "http_endpoint.clj",
                147
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127$fn__13129",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invokeStatic",
                "ioc_macros.clj",
                978
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invoke",
                "ioc_macros.clj",
                977
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invokeStatic",
                "ioc_macros.clj",
                982
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invoke",
                "ioc_macros.clj",
                980
            ],
            [
                "clojure.core.async.impl.ioc_macros$take_BANG_$fn__5769",
                "invoke",
                "ioc_macros.clj",
                991
            ],
            [
                "clojure.core.async.impl.channels.ManyToManyChannel$fn__589$fn__590",
                "invoke",
                "channels.clj",
                95
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor",
                "runWorker",
                "ThreadPoolExecutor.java",
                1149
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor$Worker",
                "run",
                "ThreadPoolExecutor.java",
                624
            ],
            [
                "clojure.core.async.impl.concurrent$counted_thread_factory$reify__492$fn__493",
                "invoke",
                "concurrent.clj",
                29
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.lang.Thread",
                "run",
                "Thread.java",
                748
            ]
        ]

What approach do you use to detect a good connection to avoid :cognitect.anomalies/unavailable exceptions?

Tue, 27 Oct 2020 20:44:31 +0000

I'm working on a retry+backoff solution for the :cognitect.anomalies/unavailable "Loading Database" exception that we get after our ions restart. If this is a common issue like the troubleshooting docs suggest, I'm wondering how others are handling this.

My current approach when getting the connection with d/connect is to try/catch when performing a simple check if the connection is active (something like (d/db-stats (d/db conn))). I've also considered doing a d/q query for some specific datom that I know is present.

I've also wondered if the ion has some other mechanism of denying incoming http or lambda requests after restart until the Datomic connection is viable.

Any thoughts or other ideas? Thanks for your input!

Expose an alt-port transactor configuration property

Sun, 25 Oct 2020 19:45:14 +0000

The alt-host configuration property is great and helps in many scenarios, but not having an equivalent for the port (alt-port?) is a blocker in some other scenarios. It would be great to have such property, which would default to the same port as the transactor is listening.

`gcDeletedDBs` available in peer API

Sun, 25 Oct 2020 19:36:18 +0000

It would be very helpful to be able to call gcDeletedDBs programmatically from the API. Providing the felxibility of passing in a Datasource. Allows users to get GC capabilities in the app.

Store values encrypted, allowing key rotation.

Sun, 25 Oct 2020 19:14:02 +0000

In our application we have a requirement to store certain fields encrypted. This encryption is done in the application. However, we would like to be able to rotate the encryption key should it become compromised, and retain the historical query features that you would get with unencrypted storage. In order to rotate the key, we have to excise the data and re-transact it with the new key, or store the encrypted values outside of Datomic. It would be nice to have an encryption API that enabled this use case.

Create DB from sequence of transactions

Sun, 25 Oct 2020 18:35:40 +0000

Given an input sequence of transactions expressed as EDN or some other format, provide a process to create a new Datomic database "offline" as quickly as possible, either in a supported storage backend or as a backup suitable for restore-db.

The input transactions will use real Entity IDs, not tempids. Provide public APIs for constructing Entity IDs correctly.

No other transactions or queries can be performed while the new database is being built. The input transactions cannot call any transaction functions.

Similarly, provide an "export" process to write out the transaction log of an existing database to a file in the same format.

This is essentially an optimization of the current process for bulk imports with a Transactor, with the added benefit of being able to control the creation of Entity IDs. This might be used, for example, to make a copy of a Datomic database with a filter or transformation applied to the transactions.

Allow for setting transactor properties using environment variables

Sun, 25 Oct 2020 18:08:09 +0000

Datomic should be able to read it's settings from environment variables, following a naming standard like what Spring Boot does. E.g.:

DATOMIC_LICENSE_KEY
DATOMIC_PROTOCOL
DATOMIC_PORT
…

Support AWS dynamo point in time backup in on-prem.

Sun, 25 Oct 2020 16:57:02 +0000

https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/

Currently Datomic supplies backup/restore capability here:

https://docs.datomic.com/on-prem/backup.html

Dynamo's PITR remains untested for use with Datomic DBs. Implementation of this feature would ensure that using Dynamo's PITR causes no issues on a Datomic DB.

Pass along DDB request IDs with transactions

Sun, 25 Oct 2020 16:41:05 +0000

Include the Dynamo DB request IDs with transactions.

In order to provide more information for troubleshooting, the DDB team has indicated that passing along the DDB request ID with transactions would enable further troubleshooting, down to the specific problematic transactions.