Datomic Knowledgebase - Recent questions in On-Prem

Datomic Pro's executable shell scripts (e.g. transactor.sh) fail on NixOS

Thu, 18 Dec 2025 13:40:50 +0000

This is because the shebang is #!/bin/bash which is not standards compliant and NixOS does not expose this. The compliant shebang that works on all systems is #!/usr/bin/env bash (according to the NixOS user). The impact of this is that when I do user interview calls for Datomic Browser and ask users to install Datomic Pro with mbrainz, NixOS users get errors on restore-db etc. The resolution is I have to instruct them to edit in place the Datomic shell scripts in bin/ to rewrite their shebang to the correct value.

datomic pro 1.0.7469

Why does changing the #inst reader make my query fail?

Mon, 20 Oct 2025 17:21:00 +0000

does datomic pro print and read queries during datomic.api/q, maybe as part of its query cache? we’ve changed the #inst reader in our project to return java.time.Instant and now a java.util.Date we pass to a query comes out as a java.time.Instant. Is that expected & documented somewhere?

(require '[datomic.api])

;; this is demonstrating an issue we ran into where overwriting the
;; `#inst` reader broke `datomic.api/q` in surprising ways.

(def schema
  [{:db/ident :exit/datetime
    :db/valueType :db.type/instant
    :db/cardinality :db.cardinality/one}])

(def conn
  (datomic.api/connect (doto "datomic:mem://inst-java-time-readers" datomic.api/create-database)))

@(datomic.api/transact conn schema)

(def db+data
  (:db-after
   @(datomic.api/transact conn [{:exit/datetime #inst "2024"}
                                {:exit/datetime #inst "2025"}
                                {:exit/datetime #inst "2026"}])))

;; this works
(count (datomic.api/q [:find '?f
                       :where
                       '[?f :exit/datetime ?exit]
                       [(list '< '?exit (java.util.Date.))]]
                      db+data))


;; fails silently (returns incorrect number of results)
(binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
  (count (datomic.api/q [:find '?f
                         :where
                         '[?f :exit/datetime ?exit]
                         [(list '< '?exit (java.util.Date.))]]
                        db+data)))

;; fails loudly with processing rule: (q__124130 ?f), message:
;; processing clause: [?f :exit/datetime ?exit], message:
;; java.lang.ClassCastException: class java.time.Instant cannot be
;; cast to class java.util.Date (java.time.Instant and java.util.Date
;; are in module java.base of loader 'bootstrap')
(binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
  (count (datomic.api/q [:find '?f
                         :where
                         '[?f :exit/datetime ?exit]
                         [(list java.util.Date/.before '?exit (java.util.Date.))]]
                        db+data)))

;; My guess is that this is related to datomic's [query
;; caching](https://docs.datomic.com/query/query-executing.html#query-caching)
;; because once a query has been cached, a sucessive call with the
;; same query value and a bound #inst reader succeeds:
(let [query [:find '?f
             :where
             '[?f :exit/datetime ?exit]
             [(list java.util.Date/.before '?exit (java.util.Date.))]]]
  (= (count (datomic.api/q query db+data))
     (binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
       (count (datomic.api/q query db+data)))))

Using a collection of datoms as input to a query

Thu, 24 Apr 2025 12:51:36 +0000

I was wondering if there is a way to use a collection of datoms obtained from d/tx-range + get a single tx or received from a listener that subscribed to a tx queue as input to d/q? The nature of this collection looks very similar to history db which can be used with d/q, however passing this collection as input does not produce the expected result. Neither there is any error.

Why does Datomic store all databases in a single datomic_kvs table instead of having a separate table per database?

Mon, 03 Mar 2025 17:06:33 +0000

I am curious about the reasoning behind keeping all Datomic databases in a single datomic_kvs table rather than creating a separate table for each database.

Some disadvantages of the current approach that I see:

More complex maintenance: For example, if VACUUM FULL is needed because one database is highly active and generates a lot of garbage, it will block all Datomic databases, not just the affected one.
Difficult to determine per-database segment size: Since the table contains entries for all databases, simply counting elements in datomic_kvs does not provide a per-database segment count.
Database deletion requires a special utility: Instead of simply dropping a table, removing a database requires a dedicated tool.
Disk space is not immediately freed: Because of the previous point, space is not reclaimed until VACUUM FULL is run, which again locks all Datomic databases.

Given these limitations, what are the key advantages that justify using a single datomic_kvs table instead of separate tables per database?

Also, is there any plan to introduce a mode where each Datomic database has its own separate table?

Regression in latest datomic when transacting data & schema in one tx

Wed, 19 Feb 2025 20:44:39 +0000

Let's say you have a transaction like:

{:db/id :admin/id,
 :db/unique :db.unique/identity
 :db.install/_attribute :db.part/db}
{:db/id "admin"
  :db/ident :admin/bug
  :admin/id admin-id}

(So both schema & data in one tx)

Normally you would expect idents to work properly - e.g. after that transaction (d/entity db [:admin/id admin-id]) would return the :admin/bug entity. Unfortunately that's not the case (though it used to work in previous versions of datomic) - it returns nil. Splitting the transaction into two makes everything work.

Here's a full repro:

(require '[datomic.api :as d])
(def conn (d/connect (doto (str "datomic:mem://" (d/squuid)) (d/create-database))))
@(d/transact conn [{:db/id "new"
                    :db/ident :admin/id,
                    :db/valueType :db.type/uuid,
                    :db/cardinality :db.cardinality/one,
                    :db/doc "Id of the admin"
                    :db.install/_attribute :db.part/db}])

(def admin-id (d/squuid))

@(d/transact conn [{:db/id :admin/id,
                    :db/unique :db.unique/identity
                    :db.install/_attribute :db.part/db}
                   {:db/id "admin"
                    :db/ident :admin/bug
                    :admin/id admin-id}])

(def db (d/db conn))

(if (some? (d/entity db [:admin/id admin-id]))
  (println "All OK - admin is found by identity")
  (println "Admin identity not there? Let's see the entity: " (d/touch (d/entity db :admin/bug))))

"No suitable driver" when starting peer

Fri, 24 Jan 2025 01:10:09 +0000

In the peer container:

root@3249b8857f6f:/opt/datomic# bin/repl
Clojure 1.11.4
user=> (Class/forName "org.postgresql.Driver")
org.postgresql.Driver
user=> (require 'datomic.peer-server)
nil
user=> (datomic.peer-server/-main
"-h" "0.0.0.0"
"-p" "8998"
"-a" "none,none"
"-d" "datomic,datomic:sql://mydb?jdbc=postgresql://catalyst-postgres-1:5432/datomic?user=datomic&password=datomic&jdbc-class=org.postgresql.Driver")
Execution error (SQLException) at java.sql.DriverManager/getDriver (DriverManager.java:299).
No suitable driver
user=>

transactor on-prem logging

Mon, 23 Dec 2024 12:18:20 +0000

Transactor emits too much log data with INFO level which makes it difficult to separate important realtime metrics/events from low level stuff useful only for troubleshooting. E.g.:
`
2024-12-23 12:14:42.147 INFO default datomic.kv-cluster - {:event :kv-cluster/create-val, :val-key "67695432-37bb-4155-9731-0ad26be7e1de", :bufsize 882, :msec 5.65, :phase :end, :pid 9, :tid 692}
2024-12-23 12:14:42.151 INFO default datomic.kv-cluster - {:event :kv-cluster/create-val, :val-key "67695432-11c9-4b95-85dc-1d081032a132", :bufsize 1170, :msec 6.64, :phase :end, :pid 9, :tid 694}

...

2024-12-23 12:32:35.033 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 70, :pid 9, :tid 5}
2024-12-23 12:32:35.396 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 63, :pid 9, :tid 5}
2024-12-23 12:32:35.749 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 49, :pid 9, :tid 5}
2024-12-23 12:32:36.135 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 68, :pid 9, :tid 5}
2024-12-23 12:32:36.487 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 65, :pid 9, :tid 5}
2024-12-23 12:32:36.853 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 55, :pid 9, :tid 5}
2024-12-23 12:32:37.239 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 73, :pid 9, :tid 5}
2024-12-23 12:32:37.615 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 65, :pid 9, :tid 5}
2024-12-23 12:32:37.961 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 69, :pid 9, :tid 5}
2024-12-23 12:32:38.285 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 58, :pid 9, :tid 5}
2024-12-23 12:32:38.713 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 68, :pid 9, :tid 5}
2024-12-23 12:32:39.039 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 62, :pid 9, :tid 5}
2024-12-23 12:32:39.364 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 66, :pid 9, :tid 5}
2024-12-23 12:32:39.721 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 78, :pid 9, :tid 5}
2024-12-23 12:32:39.923 INFO default datomic.lifecycle - {:tid 19, :username "krVN7t+5clwdg5/HazGJsrE0ckBlt50uk+HlYn9vbwY=", :port 4334, :rev 4243344, :host "0.0.0.0", :pid 9, :event :transactor/heartbeat, :version "1.0.7187", :timestamp 1734957159894, :encrypt-channel true}
`

It would be nice to review log levels and have a paper on messages generated with INFO level. Also the log line seems to be EDN for which there is no parser on ELK/Grafana side. Can this be made JSON to enable structural logging?

How to deprecate a composite tuple?

Sun, 24 Nov 2024 09:06:05 +0000

As Datomic does not allow deleting attributes from schema we use the policy of adding a prefix to the attributes that are no longer used, e.g. :foo/bar becomes :foo/#bar. We then stop inserting deprecated attributes in the app code. This way old entities get to keep deprecated attributes, new ones do not have them. This works with all kinds of attributes, however this is not sufficient for composite tuples as Datomic calculates/inserts their values automatically.

Is there a way to instruct Datomic to stop populating a composite tuple?

Using Valcache via Docker bind mounts

Thu, 11 Apr 2024 09:50:54 +0000

I'm evaluating caching options for Datomic Pro. Valcache sounds like a great fit, but there is one thing that might be a showstopper.

The Prerequisites docs says that "Valcache relies on an SSD with the strictatime and lazytime flags set". This requirement I can meet pretty easily by remounting the disk on the hosts running the peers.

Now, the applications that are Datomic peers, who would be the users of these Valcaches, run as Docker containers in a Docker Swarm. The idea is to mount the Valcache as a Docker bind mount.

Would this work? The underlaying SSD disk would be mounted with strictatime and lazytime, but would whatever mechanism that requires these flags to be set work through a Docker bind mount?

How do I restart a gc-deleted-dbs after it got canceled in "Deleting garbage segments" phase?

Tue, 19 Mar 2024 04:10:45 +0000

I recently (backed up and) deleted a fairly large database in production, and had gotten partway through reclaiming the space in Postgres with bin/datomic -Ddatomic.gcStoragePaceMsec=5 gc-deleted-dbs [...] when I wanted to speed it up by removing the pacing, so I stopped it with Ctrl-C. The docs explicitly say that "You can kill a gc-deleted-dbs process and restart it later with no adverse affects" and I had previously done exactly that to speed up the pacing once before.

Unfortunately, it no longer recognizes any need to clean up garbage segments, the third stage (after log and index segments). Adding a new temporary database, filling it with a bunch of junk transactions, and deleting it, successfully goes through the whole process for that DB only, but does not trigger any additional garbage segment cleanup. (If I don't add the junk transactions it skips that stage entirely.)

Possibly security issue with the dev H2 storage?

Wed, 13 Mar 2024 21:58:26 +0000

While trying to understand more about the behaviour of Datomic dev (mainly from a security/encryption point of view), I attempted to connect to datomic.mv.db from an instance of the H2 console. I accidentally tried to log in as "admin" with a blank password and was surprised when I connected successfully. Connecting as "admin" with any password fails, so it seems explicitly to be blank. As far as I can tell I only have public schema access and can't see any transacted data, but I am able to run SQL queries in the console.

Clearly a dev transactor isn't intended to be high security, but admin with a blank password seems risky even for a dev environment. In most SQL environments I work with, the sa/admin account is just disabled if not used. Am I missing something?

As an aside, does the dev transactor use H2's AES encryption or is the resulting data file unencrypted?

Why is the pull pattern for wild card giving me an error?

Wed, 31 Jan 2024 16:27:11 +0000

I am having an issue with a query and wild card patterns:
(d/q {:query '[:find (pull ?page pattern)

                         :in $ ?id pattern
                         :where
                         [?page :page/id ?id]]
                  :args [(d/db conn)
                         id
                         '[*]]})

It gives me an error:
Attribute identifier clojure.core/* of class: class java.lang.String does not start with a colon
Any idea why it doesn't work? I also tried adding the pattern directly to the pull expression, but I get the exact same issue

java.lang.IllegalStateException: Can't set!: io-index from non-binding thread

Sat, 13 Jan 2024 22:49:31 +0000

I'm getting the following error in the transactor when trying to transact some data using a transaction function. It only happens for a specific query and set of data.

I'm not sure about the correct place to report errors like this, so feel free to direct me to another place if this is not the correct place.

Also let me know if you need any other information.

I am using datomic version 1.0.7075, peer and transactor have the same version.

I'm using JDK 21, Temurin (datomic transactor is running inside clojure:temurin-21-tools-deps-alpine docker container)

taskfeed-datomic  | java.lang.IllegalStateException: Can't set!: *io-index* from non-binding thread
taskfeed-datomic  |     at clojure.lang.Var.set(Var.java:223)
taskfeed-datomic  |     at datomic.measure.io_stats$using_index_BANG_.invokeStatic(io_stats.clj:57)
taskfeed-datomic  |     at datomic.measure.io_stats$using_index_BANG_.invoke(io_stats.clj:54)
taskfeed-datomic  |     at datomic.index.Index.seek(index.clj:553)
taskfeed-datomic  |     at datomic.btset$seek.invokeStatic(btset.clj:399)
taskfeed-datomic  |     at datomic.btset$seek.invoke(btset.clj:394)
taskfeed-datomic  |     at datomic.db.Db.seekEAVT(db.clj:2410)
taskfeed-datomic  |     at datomic.query$eav.invokeStatic(query.clj:78)
taskfeed-datomic  |     at datomic.query$eav.invoke(query.clj:69)
taskfeed-datomic  |     at datomic.query.EntityMap.valAt(query.clj:213)                                                                                                                               taskfeed-datomic  |     at datomic.query.EntityMap.entryAt(query.clj:199)                                                                                                                             taskfeed-datomic  |     at clojure.lang.RT.find(RT.java:860)
taskfeed-datomic  |     at clojure.core$select_keys.invokeStatic(core.clj:1562)
taskfeed-datomic  |     at clojure.core$select_keys.invoke(core.clj:1555)
taskfeed-datomic  |     at ns_272$eval273$fn__274$fn__275$fn__276.invoke(NO_SOURCE_FILE:143)
taskfeed-datomic  |     at clojure.walk$prewalk.invokeStatic(walk.clj:65)
taskfeed-datomic  |     at clojure.walk$prewalk.invoke(walk.clj:61)
taskfeed-datomic  |     at clojure.core$partial$fn__5908.invoke(core.clj:2641)
taskfeed-datomic  |     at clojure.core$map$fn__5935.invoke(core.clj:2770)
taskfeed-datomic  |     at clojure.lang.LazySeq.sval(LazySeq.java:42)
taskfeed-datomic  |     at clojure.lang.LazySeq.seq(LazySeq.java:51)
taskfeed-datomic  |     at clojure.lang.RT.seq(RT.java:535)
taskfeed-datomic  |     at clojure.core$seq__5467.invokeStatic(core.clj:139)
taskfeed-datomic  |     at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
taskfeed-datomic  |     at clojure.core$reduce.invokeStatic(core.clj:6886)
taskfeed-datomic  |     at clojure.core$into.invokeStatic(core.clj:6958)
taskfeed-datomic  |     at clojure.walk$walk.invokeStatic(walk.clj:50)
taskfeed-datomic  |     at clojure.walk$prewalk.invokeStatic(walk.clj:65)
taskfeed-datomic  |     at clojure.walk$prewalk.invoke(walk.clj:61)
taskfeed-datomic  |     at clojure.core$partial$fn__5908.invoke(core.clj:2641)
taskfeed-datomic  |     at clojure.core$map$fn__5935.invoke(core.clj:2772)
taskfeed-datomic  |     at clojure.lang.LazySeq.sval(LazySeq.java:42)
taskfeed-datomic  |     at clojure.lang.LazySeq.seq(LazySeq.java:51)
taskfeed-datomic  |     at clojure.lang.Cons.next(Cons.java:39)
taskfeed-datomic  |     at clojure.lang.RT.next(RT.java:713)                                                                                                                                          taskfeed-datomic  |     at clojure.core$next__5451.invokeStatic(core.clj:64)
taskfeed-datomic  |     at clojure.core.protocols$fn__8249.invokeStatic(protocols.clj:169)
taskfeed-datomic  |     at clojure.core.protocols$fn__8249.invoke(protocols.clj:124)
taskfeed-datomic  |     at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
taskfeed-datomic  |     at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)                                                                                                        [3165/9184]
taskfeed-datomic  |     at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
taskfeed-datomic  |     at clojure.core$reduce.invokeStatic(core.clj:6886)
taskfeed-datomic  |     at clojure.core$into.invokeStatic(core.clj:6958)
taskfeed-datomic  |     at clojure.walk$walk.invokeStatic(walk.clj:50)
taskfeed-datomic  |     at clojure.walk$prewalk.invokeStatic(walk.clj:65)
taskfeed-datomic  |     at clojure.walk$prewalk.invoke(walk.clj:61)
taskfeed-datomic  |     at ns_272$eval273$fn__274$fn__275.invoke(NO_SOURCE_FILE:93)
taskfeed-datomic  |     at clojure.core$pmap$fn__8552$fn__8553.invoke(core.clj:7089)
taskfeed-datomic  |     at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
taskfeed-datomic  |     at clojure.lang.AFn.call(AFn.java:18)
taskfeed-datomic  |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
taskfeed-datomic  |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
taskfeed-datomic  |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
taskfeed-datomic  |     at java.base/java.lang.Thread.run(Thread.java:1583)

On the client side, I get the following error:

taskfeed-1  | 01:09:47 [clojure-agent-send-off-pool-21] WARN  datomic.connector - {:message "error executing future", :pid 1, :tid 97} {  }
taskfeed-1  | java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Throwable (java.lang.String and java.lang.Throwable are in module java.base of loader 'bootstrap'
)
taskfeed-1  |   at user$eval172500$fn__172501.invoke(NO_SOURCE_FILE)
taskfeed-1  |   at datomic.error$deserialize_exception.invokeStatic(error.clj:178)
taskfeed-1  |   at datomic.error$deserialize_exception.invoke(error.clj:160)
taskfeed-1  |   at datomic.peer.Connection.notify_error(peer.clj:388)
taskfeed-1  |   at datomic.connector$fn__16648.invokeStatic(connector.clj:154)
taskfeed-1  |   at datomic.connector$fn__16648.invoke(connector.clj:152)
taskfeed-1  |   at clojure.lang.MultiFn.invoke(MultiFn.java:234)
taskfeed-1  |   at datomic.connector$create_hornet_notifier$fn__16654$fn__16655$fn__16659$fn__16660.invoke(connector.clj:178)
taskfeed-1  |   at datomic.connector$create_hornet_notifier$fn__16654$fn__16655$fn__16659.invoke(connector.clj:173)
taskfeed-1  |   at datomic.connector$create_hornet_notifier$fn__16654$fn__16655.invoke(connector.clj:171)
taskfeed-1  |   at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
taskfeed-1  |   at clojure.lang.AFn.call(AFn.java:18)
taskfeed-1  |   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
taskfeed-1  |   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
taskfeed-1  |   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
taskfeed-1  |   at java.base/java.lang.Thread.run(Thread.java:1583)

And then a few seconds later, it throws the following error:

taskfeed-1  | Caused by:  :db.error/transaction-timeout Transaction timed out.
taskfeed-1  | clojure.lang.ExceptionInfo: :db.error/transaction-timeout Transaction timed out.                                                                                                        taskfeed-1  | #:db{:error :db.error/transaction-timeout}
taskfeed-1  |  at datomic.error$raise.invokeStatic (error.clj:70)
taskfeed-1  |     datomic.error$raise.invoke (error.clj:58)
taskfeed-1  |     datomic.error$raise.invokeStatic (error.clj:68)
taskfeed-1  |     datomic.error$raise.invoke (error.clj:58)
taskfeed-1  |     datomic.peer$await_tx_result.invokeStatic (peer.clj:79)
taskfeed-1  |     datomic.peer$await_tx_result.invoke (peer.clj:70)
taskfeed-1  |     datomic.peer.Connection.transact (peer.clj:315)
taskfeed-1  |     datomic.api$transact.invokeStatic (api.clj:107)
taskfeed-1  |     datomic.api$transact.doInvoke (api.clj:105)

how to run a dev transactor from the same process as a peer?

Wed, 10 Jan 2024 22:30:06 +0000

At scale you definitely want a separate transactor, but for development or for deploying an early prototype , it would be very handy to be able to have the transactor running in the same process as the peer. This would trade reliability and predictability for resource efficiency, and allow deploying on an inexpensive instance somewhere in a public cloud. This would probably lower the bar for trying out datomic in many settings.

In fact, it seems that datomic local allows having an embedded H2 database and a transactor into the same JVM process as your client.

You mentioned that it is possible to do the same for datomic on-prem, could you detail how one might do it?

Is it possible to run a transactor within the scope of my application?

Sun, 03 Dec 2023 11:52:25 +0000

Hi,

Is it possible to run a transactor from within a Clojure application? I'm trying to gain more control over/feedback about the state of the transactor, and simply shelling out to bin\transactor doesn't seem like a neat solution. I'd also prefer that the end user doesn't have to provide and maintain their own transactor.

PS, I understand concerns about contention for resources, but what I envision is a simple wrapper application that runs in its own JVM process separate from the rest of my application logic.

Is it possible to separate Datomic On-Prem from Cloud?

Sun, 03 Dec 2023 11:48:45 +0000

I see that AWS-specific jars comprise more than half the size of Datomic install; adding these 300+ jars to the classpath also makes Datomic unusable in Windows due to "input line is too long".

As an exclusively On-Prem user, is there any way to omit these unnecessary components and still have a viable Datomic On-Prem transactor? Or have the two capabilities been too-deeply intermingled for this to be possible?

Database functions, Classpath functions, and Transaction functions

Wed, 01 Nov 2023 08:56:45 +0000

Datomic Pro documentation, reference/database functions states:

You can deploy code to a Datomic system by installing database functions into a Datomic database, or by adding classpath functions to the Java classpath. Both database functions and classpath functions can be used as transaction functions that run inside a transaction to enforce arbitrary data invariants.

I understand that either kind "can be used as" a transaction function.
But in reality is this the only way either is used in practice?

I've been trying to unfangle what an implied other kind (that isn't used as a transaction function) could be and haven't found any examples.

So the conditions regarding taking the db value as first argument, and being pure function, actually apply to every function, whether database or classpath, that we may write?

Is there way to use s3 compatible backup storage

Mon, 16 Oct 2023 18:17:34 +0000

We use google cloud and managing the backup persistent disks is not a huge burden but still less than ideal. Google cloud storage offers s3 compatible storage, but we would need to know how to provide their endpoint in the URI.

Support for DynamoDB compatible services

Thu, 11 May 2023 12:18:56 +0000

Hi!
There is a DynamoDB compatible service called YDB.
YDB in the Yandex Cloud has a serverless mode with DynamoDB support..

So I have:

AWS_ACCESS_KEY_ID
AWS_SECRET_KEY
region: ru-central1
endpoint like https://docapi.serverless.yandexcloud.net/ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1

I have tried using the ddb and ddb-local protocols with different settings.

Variants of aws-dynamodb-override-endpoint
1. docapi.serverless.yandexcloud.net/ru-central1/.../...
2. docapi.serverless.yandexcloud.net:443/ru-central1/.../...
3. https://docapi.serverless.yandexcloud.net/ru-central1/.../...

$ bin/datomic ensure-transactor config/ddb.properties config/ddb-gen.properties
java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in authority at index 7: http://"https://docapi.serverless.yandexcloud.net/ru-central1/b1g31q09ln3ca15cg5mf/etnckvjnbskiko1s639l"
	at com.amazonaws.util.RuntimeHttpUtils.toUri(RuntimeHttpUtils.java:216)
	at com.amazonaws.util.RuntimeHttpUtils.toUri(RuntimeHttpUtils.java:190)
	at com.amazonaws.AmazonWebServiceClient.toURI(AmazonWebServiceClient.java:329)
	at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:317)
	at datomic.ddb$client.invokeStatic(ddb.clj:14)

I think that ddb-local always uses http protocol and there is no way to configure it with https.

There are some examples of using AwsSDK with YDB in Yandex cloud

AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
          .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("<Document API endpoint>", "ru-central1"))
          .build();

Could you add support for DynamoDB compatible services?
Could it be the new `aws-dynamodb-override-protocol' setting, which is by default 'http' ?

Why is this error message saying the entity can't be resolved?

Thu, 04 May 2023 22:53:42 +0000

Here is the error message:

cognitect.anomalies/message: "Unable to resolve entity: 1.7592186046217E13 in datom [\"shape_VVVVVVVVVVVVVVVV_0_1\" :shape/source-shape 1.7592186046217E13]"
                           datom: ["shape_VVVVVVVVVVVVVVVV_0_1" :shape/source-shape 1.7592186046217E13]
                        db/error: :db.error/not-an-entity
                             dbs: [{:database-id
                                    "datomic:dev://datomic-transactor:4334/blah",
                                    :t 1818,
                                    :next-t 1819,
                                    :history false}]
                          entity: 1.7592186046217E13

I can query for the :db/id 17592186046217, so I know it exists. My question is why can't Datomic find it? There's only one transaction and it failed, so it is guaranteed that entity 17592186046217 existed before the transaction as well.

This is the schema for :shape/source-shape:

{:db/ident :shape/source-shape
 :db/valueType :db.type/ref  
 :db/cardinality :db.cardinality/one}

The version of Datomic is datomic-pro-1.0.6269.

What defines a datomic on-prem system?

Wed, 29 Mar 2023 20:55:33 +0000

The price for an on-prem pro license is for one system. What defines a system? Does a single system include a transactor, a stand-by transactor and multiple peers?

Is it possible for Datomic On-prem to get the storage credentials from Secrets Manager or similar software?

Thu, 02 Mar 2023 10:50:19 +0000

Is there a way to run the transactor service without storing the credentials to disk?

equality on d/entity ignores db

Sun, 20 Nov 2022 20:49:46 +0000

Expected: entity object equality semantics to consider their database value in determining equality, such that entities derived from differing database values are not equal

Observed: entities derived from inconsistent database values are considered equal.

(def edward 694891374557546)
(def db2 (:db-after (d/with test/datomic-db [[:db/add edward :google/display-name "a"]])))
(= (d/entity test/datomic-db edward) (d/entity db2 edward)) ; true

Why we care: when using Datomic entity API with a reactive dataflow engine, the reactive engine uses = to detect dirty expressions to schedule reevaluation, and here, two very different entity values are considered equal causing the reactive engine to incorrectly cache the expression without reevaluating (i.e. reuse a buffered prior instance of the entity that is inconsistent with the latest db).

also see https://github.com/tonsky/datascript/issues/433 – Datascript had the same bug.

Impl note: false negatives are not an issue (i.e. in the case of two equal database values that are not identical references, returning a false negative here may be acceptable, resulting in imperfect caching but no visibly broken behavior in user apps)

Hi guys, I need to retrieve id's from all users in database that are older then 30 days, every user has creation date.

Mon, 01 Aug 2022 22:13:08 +0000

Hi guys,
I need to retrieve id's from all users in database that are older then 30 days, every user has creation date.,
(let [conn (client/get-conn)

  db (client/db)
  last-month (t/minus (t/now) (t/month 1))
  q '{:find  [(pull ?eid [*])]
      :where [[?eid :user/created]]}]

;;.............
)
I have last-month #inst date, I just dont know how to get all users "older" then 30 days.
I am new in datomic so,
if someone can help, thank you in advance...

How to add new entities (as references) to the many attribute of an existing entity?

Wed, 27 Jul 2022 19:01:04 +0000

By way of example I'm going to use a drawing as the existing entity which is to have two new shapes added to it. This is the tx that is failing:
[{:shape/id "BKF806TXXTAFWII0", :db/id "22NF08ZVGH9N7QGG_0"} {:shape/id "YKIZU4CJC0JXJKVQ", :db/id "22NF08ZVGH9N7QGG_1"} [:db/add 17592186047451 :drawing/shapes ["22NF08ZVGH9N7QGG_0" "22NF08ZVGH9N7QGG_1"]]]

As you can see the existing drawing has :db/id of 17592186047451. I'm trying to create two new shapes and add them to the many cardinality attribute :drawing/shapes. "22NF08ZVGH9N7QGG_0" and "22NF08ZVGH9N7QGG_1" are tempids that are supposed to be translated to the same new :db/ids wherever they appear.

I should say that the drawing doesn't have any existing shapes. If it did I would expect them to be orphaned.

This is the error message I'm getting:

Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
Cannot interpret as a keyword: 22NF08ZVGH9N7QGG_0, no leading :

Something similar I've also tried:
[{:shape/id "9HTL5BMMHT6QUJM6", :db/id "22NF08ZVGH9N7QGG_0"} {:shape/id "O5UB9IG9UB8KDVA2", :db/id "22NF08ZVGH9N7QGG_1"} [:db/add 17592186047451 :drawing/shapes "22NF08ZVGH9N7QGG_0"] [:db/add 17592186047451 :drawing/shapes "22NF08ZVGH9N7QGG_1"]]

Gives this error message:
Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
Two datoms in the same transaction conflict
{:d1
[17592186047451 :drawing/shapes 17592186049126 13194139538021 true],
:d2
[17592186047451 :drawing/shapes 17592186049127 13194139538021 true]}

This gives the same error message as the first attempt:
[{:db/id 17592186047451, :drawing/shapes ["22NF08ZVGH9N7QGG_0" "22NF08ZVGH9N7QGG_1"]} {:shape/id "3DZYWHEPQIAELF25", :db/id "22NF08ZVGH9N7QGG_0"} {:shape/id "GJ804SOOU36YQX6Y", :db/id "22NF08ZVGH9N7QGG_1"}]

Getting rid of the tempids by doing it in a nested way:
[{:db/id 17592186047451, :drawing/shapes [{:shape/id "GEMRMRFG0E6N262M"} {:shape/id "HTH2C7R90BQFFNXT"}]}]

Just yields this error message:
Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
Unable to resolve entity: {:shape/id "GEMRMRFG0E6N262M"}

How do you visualize your schema ?

Fri, 08 Jul 2022 15:58:59 +0000

Hi I made a quick search in this channel to find tools for visualizing Datomic schema. The most promising tool I found was Hodur Datomic Schema. This is nice because you can create entity types and connect them through attributes. I would like to share with you an example I made. Here is a simple Datomic Schema in Hodur Descriptive Domain Modeling Engine EDN format and here is my clojure gist to run and output the result. There is a better Hodur schema visualization using GoJS library but I have not figured out yet how to make the http web server to work with the Hodur visualizer clojurescript it seems users have issues with it.

OK that said, I am aware that in Clojure you are trying to break away from entity types, object types in general and work with attribute oriented modeling. But the way you are modeling schema you cannot escape from namespace prefixes and these imply entity types. In any case it is standard practice in any significant large IT project that users want to see and work easily on a white-board with some kind of graph data model that represents an abstract data model. That was the scope of Hodur I guess. In the RDBMs world and more recently in Graph DBMS there are great tools to visualize both the schema and data. So how do you visualize the data modeling process and it's evolution in large projects using Datomic ?

I had a quick look at schema-cartographer.

It creates schema-cartographer dependent edn files when you download schema transactions and schema files from their Web APP UI (see schema.edn) and (schema-txs.edn)
You have to work everything on the UI. But I think you cannot download and run this independently on your premises.
There is this Create a schema file for an existing annotated ON PREM database from a REPL but again you have to load the resulting output file in the UI that resides at https://schema-cartographer.com
It is not flexible enough and it will result in poor visualization when you have a big schema and/or you want to view part of the schema
The schema-cartographer UI says GoJS 2.0 evaluation (c) 1998-2019 Northwoods Software Not for distribution or production use gojs.net

So what tool Cognitect people use to visualize Datomic schema. Is that the tool ?

Is using entid function to resolve lookup refs a supported behaviour?

Wed, 08 Jun 2022 13:30:29 +0000

I'm using datomic.api/entid to resolve a lookup ref to an entity ID, like
$ (entid db [:user/id "test@test.com"]) 17592186045432
This works fine, but a colleague pointed out this is not a documented behaviour.
>Usage: (entid db ident)
>Returns the entity id associated with a symbolic keyword, or the id
itself if passed.

Is this a gap in documentation, or is this behaviour accidental and not recommended to rely on?

Transactor timeout on transaction function throw

Fri, 27 May 2022 17:54:06 +0000

Version

com.datomic/datomic-pro "1.0.6242"
transactor process 0.9.6045

Setup

Given the following transaction function definition:

(d/transact
 *conn*
 [{:db/ident :demo-throw-fn
   :db/fn (d/function
           {:lang "clojure"
            :params '[db]
            :code '(throw (ex-info "demo" {:type (type #{:one})}))})}])

Execute the transaction function in two environments:

A local environment
An environment where the transactor runs in a separate process

Expectation

Regardless of the environment, transaction function runtime errors are thrown.

Actual

In a local environment, execution of the demo function indeed throws an error - in line with expectations.
In an environment where the transactor runs in a separate process, the transactor times out.

Additional Notes

The hypothesis for the timeout is a second error occurs due to the
(type #{:one}) expression's result not being serializable. Unfortunately, the timeout error does not reveal what was likely known at some point during execution.

Inconsistent return value using set?

Fri, 27 May 2022 17:51:40 +0000

Version

com.datomic/datomic-pro "1.0.6242"
transactor process 0.9.6045

Setup

Given the following transaction function definition.

(d/transact
 *conn*
 [{:db/ident :demo-fn
   :db/fn (d/function
           {:lang "clojure"
            :params '[db]
            :code '(set? #{:one})})}])

Execute the transaction function in two environments:

A local environment
An environment where the transactor runs in a separate process

Expectation

Regardless of the environment, the return value of set? is consistent.

Actual

In a local environment, the function returns true - as expected.
In an environment where the transactor runs in a separate process, the function returns false.

Additional Notes

The data type of a set is different in the two environments,
clojure.lang.PersistentHashSet in local and java.util.HashSet in the other.

Advice on CVEs of h2 version 1.3.171

Tue, 05 Apr 2022 16:21:32 +0000

Using com.datomic/datomic-pro version "1.0.6362" brings in a transitive dependency on com.h2database/h2 version "1.3.171" (on-prem).

The latter suffers from various CVEs:

(c.f. mvnrepository.com/artifact/com.h2database/h2/1.3.171

Is a Datomic upgrade planned to resolve this?
If not and in the meantime, can you give any advice on how to deal with this? In particular:
Can you give detailed advice on whether these vulnerabilities actually do or do not constitute a security risk?
3.1 Some of the CVEs mention h2 console as part of an attack vector, and I have heard this would not apply to Datomic. However, I have not found official information on this. Moreover, if I understood correctly, there are other attack vectors. Please advise.
3.2 The first linked CVE is a SQLXML XXE. It seems like this would not apply as long as getSQLXML is not called, but this is just my impression and I do not know about usage in Datomic. Please advise.
Do you have any advice on overriding the h2 dependency with an h2 version not suffering from these CVEs, say version "2.1.210"? As this a major upgrade, I expect problems. In particular, I expect migrating existing h2 databases to be problematic.

Thank you very much for your assistance.

Retrieving tx from a entity - attribute pair without doing a query on history db

Fri, 07 Jan 2022 10:09:22 +0000

Given an entity and one of its attributes, is there a way to see the tx that added it without doing a query on history db?

i.e. something other than

(d/q
       '[:find ?tx ?attr ?val ?added
         :in $ ?e
         :where
         [?e ?attr ?val ?tx ?added]]
       (d/history my-db)
       my-eid)

Why is :db.unique/identity only allowed on :db.cardinality/one?

Tue, 21 Dec 2021 18:21:49 +0000

I'm curious about the rationale for only allowing :db.cardinality/one attributes to be :db.unique/identity? An immediate use case in the wild is email addresses, which uniquely identify an individual but an individual can have many email addresses.

Can I run recent versions of Datomic in Windows?

Tue, 31 Aug 2021 23:31:45 +0000

I'm not sure if Windows is (or ever was?) a supported platform for Datomic On-Prem, but at the moment when I try to start a REPL, transactor, or peer server in any recent version I get:

"The input line is too long. The syntax of the command is incorrect."

My Datomic path is short (C:\Datomic). I'm guessing the error results from classpath.cmd trying to build a CP_LIST that comprises all 400+ files in lib, though this behaviour is consistent as far back as 0.9.5927, which only has 240 jars in lib; things work as expected in 0.9.5786 (which has 232 jars) and earlier.

I'd rather not use WSL/Docker etc. at this time - can anyone advise if there's a native solution that will allow me to use a version more recent than 2018?

Retracting entities, including tuples with refs

Tue, 31 Aug 2021 13:36:44 +0000

In my schema I have attributes of type tuple that has :db.type/ref as one of the tuple's types.
My usecase is maintaining a set of ordered to-many relations.

When I retract an entity using :db/retractEntity, the tuple attrs that refer to the retracted entity still remain, continuing to refer to a now non-existent entity.
I'm not sure whether this is the intended behaviour, and if so, what would be a standard approach to clean up these tuples? My intention would be for these tuples to be completely redacted. As I'm using this in a cardinality-many setup, I'd expect the attribute to retain all tuple values, except the one that does include the ref to the retracted entity in any position.
Should I be writing a custom retractEntity-like transaction function?
Thanks!

Renaming attribute best practice in development

Thu, 24 Jun 2021 08:34:10 +0000

Assume I have a schema.clj which contains my datomic schema.

Every time I start the application (in development and in production) this schema is transacted so whatever has been added gets correctly transacted too. Now, some time ago I installed a new attribute

{:db/ident :foo/bar ...}

now after a few weeks turns out that I want to rename this attribute :alice/bob.

Following the documentation of datomic i'm supposed to transact

{:db/id :foo/bar :db/ident :alice/bob}

but that clearly doesn't work in development as :foo/bar isn't yet defined when i start my system (the whole schema is transacted at once in the same transaction) but would work on a running system with the attribute already installed.

How do you handle these cases?

Long blocking on event `:kv-cluster/get-val` without timeout/retry/failover semantics?

Wed, 23 Jun 2021 09:19:28 +0000

On a test environment, we're seeing this kind of log from the Datomic Peer:

;; ~5 hours CI inactive (if this is relevant)

DEBUG 2021-06-18T13:32:37.702 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :phase :begin, :pid 7, :tid 225}

;; 15 minutes pass....

DEBUG 2021-06-18T13:48:25.091 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :msec 947000.0, :phase :end, :pid 7, :tid 225}

;; This next one finishes in 21ms:

DEBUG 2021-06-18T13:48:25.094 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :phase :begin, :pid 7, :tid 225}
DEBUG 2021-06-18T13:48:25.115 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :msec 21.0, :phase :end, :pid 7, :tid 225}

;; and now we start processing the transacts that queued up when the tests started...

INFO  2021-06-18T13:48:25.117 [manifold-execute-43]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :start, :pid 7, :tid 282}
DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :begin, :pid 7, :tid 226}
DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :msec 0.462, :phase :end, :pid 7, :tid 226}
INFO  2021-06-18T13:48:25.235 [clojure-agent-send-off-pool-24]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :end, :pid 7, :tid 226}

After those 15 minutes of waiting, the peer moved on and transacted all the transactions that happened to queue in the meantime, as if nothing happened. There do not seem to be any interesting or anomalous logs in the Transactor for this time period.

Datomic Peer (1.0.6269) with PostgreSQL storage. The peer and transactor are both running on Kubernetes, but Postgres is hosted outside of the k8s cluster.

I think my followup questions are:

Any idea what could be going on with the :kv-cluster/get-val and how to go about debugging this further?
Is there some timeout we can configure to avoid this kind of situation in a production environment?
Is the probable culprit Kubernetes? If so, has someone seen or fixed similar behavior before?
These logs are happening at the DEBUG level, otherwise we would not even know that the Peer was blocking/hanging/waiting. Are there other metrics or logs we should be monitoring to identify this kind of problem?

Do Peers work with Java 11?

Wed, 09 Jun 2021 08:12:11 +0000

I have read that Transactors require Java 8 – c.f. https://forum.datomic.com/t/java-11-0-1-ssl-exception/734.

In the Changelog (Datomic Free 0.9.5703) I have read that

Transactors [...] require Java 7 or
later. (Peers continue to work with Java 6 or later.)

and, as of 1.0.6165,

The transactor and peer now require Java 1.8 or greater,

while the Docs state that

Datomic requires the Java 8 JDK,

see https://docs.datomic.com/on-prem/getting-started/get-datomic.html.

Is Java 11 supported on Peers?

Does the transactor eagerly fill memcached with indexed segments?

Wed, 07 Apr 2021 15:21:18 +0000

The datomic transactor memcached documentation says this:

When configured to used memcached, a Datomic process will automatically write into memcached any segment it needs that is not already present in memcached. In addition, the transactor (only) will write index and log segments to memcached as the segments are produced.

(Emphasis mine.)

Based on this sentence and assuming a memcached memory pool larger than the storage size that is shared by transactor and peer, my expectation is the following:

As the transactor is building an index, it writes newly-created index segments to storage and to memcached at roughly the same time.
When the index is finished, it commits the new index root and informs all peers of the new index.
Any peers using the same memcached cluster should, at the moment the index is released:
- have a decreasing ObjectCache hitrate because newly-indexed segments are not cached
- but an increasing memcached GET count
- and a flat StorageGet and MemcachedPut count because the segments are already in memcached
- For an overall flat or increasing memcached hitrate.

Instead, what I actually see is that when peers accept a new index,

there's a dramatic decrease in Memcached hitrate
and increase in MemcachedPut counts and Storage Get counts.
The IndexWrite count reported by the transactor is very similar to the MemcachedPut count reported by the peer.

This behavior is consistent with the transactor not writing newly-indexed segments into memcached eagerly while indexing, which is contrary to the documentation.

So the question: Does the transactor really write index segments to memcached as it is indexing?

“Error communicating with HOST localhost on PORT 4334” on fresh install of 1.0.6165

Tue, 09 Feb 2021 19:10:32 +0000

Works on other laptops... just not this one. Brought our coding session to grinding halt.

Error message when querying dev-local

Sat, 05 Dec 2020 18:10:18 +0000

Why do I get this message:

:db.error/not-an-entity Unable to resolve entity: :release/name

when I run the query below in the "Movies" db in dev-local?

Here is the code:

(defonce client (d/client {:server-type :dev-local
                           :system "datomic-samples"}))

(d/create-database client {:db-name "movies"})

(defonce conn (d/connect client {:db-name "movies"}))

(defonce db (d/db conn))

(d/q '[:find ?release-name
       :in $
       :where [$ _ :release/name ?release-name]]
     db)

Many thanks for any solution!

How do I remove a unique tuple constraint?

Fri, 20 Nov 2020 21:12:31 +0000

I have a tuple constraint as follows:

#:db{:id 182,
 :ident :ps2st/product-set+style-tier,
 :valueType :db.type/tuple,
 :cardinality :db.cardinality/one,
 :unique :db.unique/identity,
 :tupleAttrs [:ps2st/product-set :ps2st/style-tier]}

This constraint is overly restrictive. I read somewhere that tupleAttrs can not be altered, so I'd like to remove the constraint instead and then create a new one with more tuple attrs.

How can I delete a constraint?

What does the memcached/item-too-large log warning mean?

Tue, 27 Oct 2020 17:54:45 +0000

I'm seeing a fair frequency of :memcached/item-too-large statements in my Datomic logs.

Is this a size limitation internal to Datomic? If so, could it mean our system is missing out on some cached segments?

A recent datomic.process-monitor log statement is showing:

{:tid 23,
 :CacheRepair {:lo 1, :count 637, :sum 637, :hi 1},
 :ObjectCacheCount 16634,
 :PeerAcceptNewMsec {:lo 0, :count 348, :sum 1, :hi 1},
 :MemcachedPutSucceededMsec {:lo 0, :count 602, :sum 1276, :hi 177},
 :AvailableMB 5550.0,
 :Memcache {:lo 0, :count 12175, :sum 11537, :hi 1},
 :StorageGetMsec {:lo 2, :count 637, :sum 10089, :hi 831},
 :MemcacheItemTooLarge {:lo 1, :count 4, :sum 4, :hi 1},
 :pid 18,
 :event :metrics,
 :ObjectCache {:lo 0, :count 702668, :sum 690336, :hi 1},
 :MetricsReport {:lo 1, :count 1, :sum 1, :hi 1},
 :PeerFulltextBatch {:lo 1, :count 332, :sum 348, :hi 6},
 :DbAddFulltextMsec {:lo 0, :count 21, :sum 30, :hi 12},
 :MemcachedPutFailedMsec {:lo 0, :count 31, :sum 541, :hi 52},
 :StorageGetBytes {:lo 122, :count 637, :sum 92206399, :hi 31294091}}

Which would seem to suggest a high hit rate judging from the :Memcache :sum / :count

Is this something I can fix or would cause issues?

Thanks!!

Expose an alt-port transactor configuration property

Sun, 25 Oct 2020 19:45:14 +0000

The alt-host configuration property is great and helps in many scenarios, but not having an equivalent for the port (alt-port?) is a blocker in some other scenarios. It would be great to have such property, which would default to the same port as the transactor is listening.

Allow for setting transactor properties using environment variables

Sun, 25 Oct 2020 18:08:09 +0000

Datomic should be able to read it's settings from environment variables, following a naming standard like what Spring Boot does. E.g.:

DATOMIC_LICENSE_KEY
DATOMIC_PROTOCOL
DATOMIC_PORT
…

Support AWS dynamo point in time backup in on-prem.

Sun, 25 Oct 2020 16:57:02 +0000

https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/

Currently Datomic supplies backup/restore capability here:

https://docs.datomic.com/on-prem/backup.html

Dynamo's PITR remains untested for use with Datomic DBs. Implementation of this feature would ensure that using Dynamo's PITR causes no issues on a Datomic DB.

Relation between T and :db/txInstant

Sat, 24 Oct 2020 14:57:57 +0000

Hello folks,

Is there a precise relation between db's T value and the txInstant of an entity?

I have a database and if I do (:t db) I get 7 as value. Fair.

Meanwhile, if I look for a :db/txInstant for an entity via (def query '[:find ?tx :where [?e :person/id _ ?tx]]) I get very long number.

What is the relationship between the two? To give some context, what I am trying to do is:

"Given a certain entity ID, what was the latest t that has introduced/updated such entity?

New engine for executing java scripts on the transactor

Fri, 23 Oct 2020 13:52:54 +0000

Is it planned to consider another script executing engines than janino? There is script evaluator provided by groovy.lang.GroovyShell which can execute java scripts too. I have compared the time to evaluate the same java script by both engines and groovy engine executes significantly faster. Another positive side effect: support of groovy scripts. I believe using of new script evaluator will make improvements to the time of executing transactor functions written in java

https://docs.groovy-lang.org/latest/html/api/groovy/lang/GroovyShell.html

UPD. How much are java transactor scripts executted longer than clojure transactor scripts?

Can datomic.api/q be used inside or/or-join to query data from other DB?

Fri, 23 Oct 2020 10:53:08 +0000

Hi, when I'm executing query in repl in such way:

(d/q "[:find
 [?Post ...]
 :in
 $
 $comment
 [?Post_value_0 ...]
 :where

[?Post :IModel/typeName ?Post_value_0]
(or-join 
 [?Post ?Comment_text_value_3]

 [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

  [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

)
 [?Post :Post/comments ?Comment_uid_2]
]
" (d/db conn) (d/db connComment) #{"Post"})

The next error is returned:
:db.error/not-an-entity Unable to resolve entity: :Comment/text

My supposing is that query inside datomic.api/q is executed for database returned by (d/db conn) but not (d/db connComment) because if I create :Comment/text for the first database from input list then query is executed without errors.

If to avoid or-join and execute the query in the next way:

(d/q "[:find
 [?Post ...]
 :in
 $
 $comment
 [?Post_value_0 ...]
 :where

[?Post :IModel/typeName ?Post_value_0]
 [?Post ?Comment_text_value_3]

 [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

  [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]


 [?Post :Post/comments ?Comment_uid_2]
]
" (d/db conn) (d/db connComment) #{"Post"})

then query is executed without errors.

Datamic on prem 0.9.6.* version

Delete all except the latest point in time within a backup URI

Thu, 22 Oct 2020 21:46:45 +0000

Also deleting a single point in time might be useful

Provide integrity check on Datomic Backups

Thu, 22 Oct 2020 18:03:10 +0000

A tool to confirm the validity and completeness of a Datomic backup.