Datomic Knowledgebase - Recent questions

Ion push failing with error message about (missing?, unreadable?) maven artifact ion-1.0.47.jar

Tue, 19 May 2026 19:40:21 +0000

Attempting to push an ion and get an error about not finding (or unable to read) ion-1.0.47.jar. I tried some troubleshooting and I don't think it is a permissions issue, because I am able to use the aws cli and copy ion-0.9.7.jar. So why is 1.0.47 not working?

deps.edn:

{:mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}

 :aliases {
           :ion-dev {:deps {com.datomic/ion-dev {:mvn/version "1.0.352"}
                            com.datomic/client-cloud {:mvn/version "1.0.131"}
                            com.datomic/ion {:mvn/version "1.0.71"}}
                     :main-opts ["-m" "datomic.ion.dev"]}}}

command I'm trying to run:

clojure -M:ion-dev '{:op :push :creds-profile "byt-wrought-test" :region "us-west-2"}'

error:

{:command-failed
 "{:op :push :creds-profile \"byt-wrought-test\" :region \"us-west-2\"}",
 :causes
 ({:message
   "Failed to read artifact descriptor for com.datomic:ion-http-direct:jar:1.0.47",
   :class ArtifactDescriptorException}
  {:message
   "Could not transfer artifact com.datomic:ion-http-direct:pom:1.0.47 from/to datomic-cloud (s3://datomic-releases-1fc2183a/maven/releases): Unexpected error downloading artifact from datomic-releases-1fc2183a",
   :class ArtifactResolutionException}
  {:message
   "Could not transfer artifact com.datomic:ion-http-direct:pom:1.0.47 from/to datomic-cloud (s3://datomic-releases-1fc2183a/maven/releases): Unexpected error downloading artifact from datomic-releases-1fc2183a",
   :class ArtifactTransferException}
  {:message
   "Unexpected error downloading artifact from datomic-releases-1fc2183a",
   :class ExceptionInfo,
   :data
   {:bucket "datomic-releases-1fc2183a",
    :path
    "maven/releases/com/datomic/ion-http-direct/1.0.47/ion-http-direct-1.0.47.pom",
    :reason :cognitect.anomalies/fault}})}

Based on some conversations I found on slack and datomic dev forum, I tried this as a troubleshooting step:

aws --profile byt-wrangle-test s3 cp s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/0.9.7/ion-0.9.7.jar .

download: s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/0.9.7/ion-0.9.7.jar to ./ion-0.9.7.jar

and that works. But the same attempt with 1.0.47 does not:

aws --profile byt-wrangle-test s3 cp s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/1.0.47/ion-1.0.47.jar .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Datomic peer - Support SSO

Fri, 13 Feb 2026 17:36:46 +0000

Datomic currently does not support AWS SSO out-of-the-box

This issue is probably because it uses an old version of dynamodb SDK

I believe that this issue can be fixed by upgrading the AWS SDK to v2

Here a workaround:

https://gist.github.com/souenzzo/42060567479ab2dd916769f1d73042fa

Datomic peer on dynamodb - misleading error message

Fri, 13 Feb 2026 16:19:56 +0000

Starting from a simple setup:

{:deps {com.datomic/peer       {:mvn/version "1.0.7491"}
        org.slf4j/slf4j-simple {:mvn/version "2.0.17"}}}

And trying to connect into a dynamodb connection, it fails with a "s3" error:

user=> (require '[datomic.api :as d])
nil
user=> (d/connect "datomic:ddb://us-east-1/prd/db")
[main] INFO datomic.process-monitor - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :phase :begin, :pid 311, :tid 3}
[main] INFO datomic.process-monitor - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :msec 0.696, :phase :end, :pid 311, :tid 3}
[main] INFO datomic.process-monitor - {:metrics/started clojure.core/identity, :pid 311, :tid 3}
[clojure-agent-send-off-pool-0] INFO datomic.domain - {:event :cache/create, :cache-bytes 502267904, :pid 311, :tid 25}
[clojure-agent-send-off-pool-0] INFO datomic.process-monitor - {:AvailableMB 895.0, :ObjectCacheCount 0, :event :metrics, :pid 311, :tid 25}
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:580).
com.amazonaws.services.s3.model.MultiObjectDeleteException

From this error message, it is pretty hard to find that datomic actually requires an extra dynamodb dependency to work.

https://docs.datomic.com/operation/storage.html#dynamodb-aws-peer-dependency

Possible fixes:

A better error message
Adding this error message to datomic docs, as a "common errors" - do google will be able to find.

Datomic on JDK25 - WARNING: A restricted method in java.lang.System has been called

Fri, 13 Feb 2026 16:19:55 +0000

Using this dependencies

{:deps {com.datomic/peer                    {:mvn/version "1.0.7491"}
        org.slf4j/slf4j-nop                 {:mvn/version "2.0.17"}
        com.amazonaws/aws-java-sdk-dynamodb {:mvn/version "1.12.564"}}}

Note: aws-java-sdk-dynamodb has an updated version, but 1.12.564 is the recommended

Generate these warnings:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by io.netty.util.internal.PlatformDependent0$4 (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$4
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by io.netty.util.internal.NativeLibraryUtil in an unnamed module (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

This issue report focus in the first part:

WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by io.netty.util.internal.NativeLibraryUtil in an unnamed module (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)

Trace:
- io.netty/netty-common 4.1.100
- required by org.apache.activemq/artemis-commons 2.31.2
- required by org.apache.activemq/artemis-core-client 2.31.2
- required by com.datomic/peer.

Datomic on JDK25 - WARNING: A terminally deprecated method in sun.misc.Unsafe has been called

Fri, 13 Feb 2026 16:19:51 +0000

Using this dependencies

{:deps {com.datomic/peer                    {:mvn/version "1.0.7491"}
        org.slf4j/slf4j-nop                 {:mvn/version "2.0.17"}
        com.amazonaws/aws-java-sdk-dynamodb {:mvn/version "1.12.564"}}}

Note: aws-java-sdk-dynamodb has an updated version, but 1.12.564 is the recommended

Generate these warnings:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by io.netty.util.internal.PlatformDependent0$4 (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$4
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by io.netty.util.internal.NativeLibraryUtil in an unnamed module (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

This issue report focus in the first part:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by io.netty.util.internal.PlatformDependent0$4 (file:/root/.m2/repository/io/netty/netty-common/4.1.100.Final/netty-common-4.1.100.Final.jar)

Trace:
- io.netty/netty-common 4.1.100
- required by org.apache.activemq/artemis-commons 2.31.2
- required by org.apache.activemq/artemis-core-client 2.31.2
- required by com.datomic/peer.

Datomic Pro's executable shell scripts (e.g. transactor.sh) fail on NixOS

Thu, 18 Dec 2025 13:40:50 +0000

This is because the shebang is #!/bin/bash which is not standards compliant and NixOS does not expose this. The compliant shebang that works on all systems is #!/usr/bin/env bash (according to the NixOS user). The impact of this is that when I do user interview calls for Datomic Browser and ask users to install Datomic Pro with mbrainz, NixOS users get errors on restore-db etc. The resolution is I have to instruct them to edit in place the Datomic shell scripts in bin/ to rewrite their shebang to the correct value.

datomic pro 1.0.7469

Why does changing the #inst reader make my query fail?

Mon, 20 Oct 2025 17:21:00 +0000

does datomic pro print and read queries during datomic.api/q, maybe as part of its query cache? we’ve changed the #inst reader in our project to return java.time.Instant and now a java.util.Date we pass to a query comes out as a java.time.Instant. Is that expected & documented somewhere?

(require '[datomic.api])

;; this is demonstrating an issue we ran into where overwriting the
;; `#inst` reader broke `datomic.api/q` in surprising ways.

(def schema
  [{:db/ident :exit/datetime
    :db/valueType :db.type/instant
    :db/cardinality :db.cardinality/one}])

(def conn
  (datomic.api/connect (doto "datomic:mem://inst-java-time-readers" datomic.api/create-database)))

@(datomic.api/transact conn schema)

(def db+data
  (:db-after
   @(datomic.api/transact conn [{:exit/datetime #inst "2024"}
                                {:exit/datetime #inst "2025"}
                                {:exit/datetime #inst "2026"}])))

;; this works
(count (datomic.api/q [:find '?f
                       :where
                       '[?f :exit/datetime ?exit]
                       [(list '< '?exit (java.util.Date.))]]
                      db+data))


;; fails silently (returns incorrect number of results)
(binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
  (count (datomic.api/q [:find '?f
                         :where
                         '[?f :exit/datetime ?exit]
                         [(list '< '?exit (java.util.Date.))]]
                        db+data)))

;; fails loudly with processing rule: (q__124130 ?f), message:
;; processing clause: [?f :exit/datetime ?exit], message:
;; java.lang.ClassCastException: class java.time.Instant cannot be
;; cast to class java.util.Date (java.time.Instant and java.util.Date
;; are in module java.base of loader 'bootstrap')
(binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
  (count (datomic.api/q [:find '?f
                         :where
                         '[?f :exit/datetime ?exit]
                         [(list java.util.Date/.before '?exit (java.util.Date.))]]
                        db+data)))

;; My guess is that this is related to datomic's [query
;; caching](https://docs.datomic.com/query/query-executing.html#query-caching)
;; because once a query has been cached, a sucessive call with the
;; same query value and a bound #inst reader succeeds:
(let [query [:find '?f
             :where
             '[?f :exit/datetime ?exit]
             [(list java.util.Date/.before '?exit (java.util.Date.))]]]
  (= (count (datomic.api/q query db+data))
     (binding [*data-readers* (assoc *data-readers* 'inst (fn [cs] (java.util.Date/.toInstant (clojure.instant/read-instant-date cs))))]
       (count (datomic.api/q query db+data)))))

500 errors from api gateway for large Ion responses

Wed, 07 May 2025 01:49:57 +0000

I'm having trouble with 500 errors that I'm pretty sure are coming from the Jetty container.

My internal logging is showing that I am responding with a status 200 and a valid response.

Smaller responses work fine but larger responses are failing with a 500 at the client and I can see that 500 in the API Gateway logs.

We are using HTTP direct in this case so it's an HTTP API gateway.

All these attempts have failed and don't have any telemetry or logs indicating the true source of the failure.

To learn more I suspect we need to enable jetty logs to stop being swallowed and start appearing in the CloudWatch logs but I haven't have any luck in finding how to do this.

Are there instructions somewhere on how to enable or increase the level of logging from the Jetty container?

Using a collection of datoms as input to a query

Thu, 24 Apr 2025 12:51:36 +0000

I was wondering if there is a way to use a collection of datoms obtained from d/tx-range + get a single tx or received from a listener that subscribed to a tx queue as input to d/q? The nature of this collection looks very similar to history db which can be used with d/q, however passing this collection as input does not produce the expected result. Neither there is any error.

Why does Datomic store all databases in a single datomic_kvs table instead of having a separate table per database?

Mon, 03 Mar 2025 17:06:33 +0000

I am curious about the reasoning behind keeping all Datomic databases in a single datomic_kvs table rather than creating a separate table for each database.

Some disadvantages of the current approach that I see:

More complex maintenance: For example, if VACUUM FULL is needed because one database is highly active and generates a lot of garbage, it will block all Datomic databases, not just the affected one.
Difficult to determine per-database segment size: Since the table contains entries for all databases, simply counting elements in datomic_kvs does not provide a per-database segment count.
Database deletion requires a special utility: Instead of simply dropping a table, removing a database requires a dedicated tool.
Disk space is not immediately freed: Because of the previous point, space is not reclaimed until VACUUM FULL is run, which again locks all Datomic databases.

Given these limitations, what are the key advantages that justify using a single datomic_kvs table instead of separate tables per database?

Also, is there any plan to introduce a mode where each Datomic database has its own separate table?

datomic's spymemcached fork collides with original

Mon, 24 Feb 2025 19:02:52 +0000

Datomic seems to be shading spymemcached. The logger implementation of spymemcached uses the property net.spy.log.LoggerImpl.

Unfortunately that property wasn't shaded properly in datomic - the fork uses the naked property without a prefix. This means setting that property crashes either datomic startup or spymemcached startup, because the same class cannot be used in both places.

Regression in latest datomic when transacting data & schema in one tx

Wed, 19 Feb 2025 20:44:39 +0000

Let's say you have a transaction like:

{:db/id :admin/id,
 :db/unique :db.unique/identity
 :db.install/_attribute :db.part/db}
{:db/id "admin"
  :db/ident :admin/bug
  :admin/id admin-id}

(So both schema & data in one tx)

Normally you would expect idents to work properly - e.g. after that transaction (d/entity db [:admin/id admin-id]) would return the :admin/bug entity. Unfortunately that's not the case (though it used to work in previous versions of datomic) - it returns nil. Splitting the transaction into two makes everything work.

Here's a full repro:

(require '[datomic.api :as d])
(def conn (d/connect (doto (str "datomic:mem://" (d/squuid)) (d/create-database))))
@(d/transact conn [{:db/id "new"
                    :db/ident :admin/id,
                    :db/valueType :db.type/uuid,
                    :db/cardinality :db.cardinality/one,
                    :db/doc "Id of the admin"
                    :db.install/_attribute :db.part/db}])

(def admin-id (d/squuid))

@(d/transact conn [{:db/id :admin/id,
                    :db/unique :db.unique/identity
                    :db.install/_attribute :db.part/db}
                   {:db/id "admin"
                    :db/ident :admin/bug
                    :admin/id admin-id}])

(def db (d/db conn))

(if (some? (d/entity db [:admin/id admin-id]))
  (println "All OK - admin is found by identity")
  (println "Admin identity not there? Let's see the entity: " (d/touch (d/entity db :admin/bug))))

"No suitable driver" when starting peer

Fri, 24 Jan 2025 01:10:09 +0000

In the peer container:

root@3249b8857f6f:/opt/datomic# bin/repl
Clojure 1.11.4
user=> (Class/forName "org.postgresql.Driver")
org.postgresql.Driver
user=> (require 'datomic.peer-server)
nil
user=> (datomic.peer-server/-main
"-h" "0.0.0.0"
"-p" "8998"
"-a" "none,none"
"-d" "datomic,datomic:sql://mydb?jdbc=postgresql://catalyst-postgres-1:5432/datomic?user=datomic&password=datomic&jdbc-class=org.postgresql.Driver")
Execution error (SQLException) at java.sql.DriverManager/getDriver (DriverManager.java:299).
No suitable driver
user=>

transactor on-prem logging

Mon, 23 Dec 2024 12:18:20 +0000

Transactor emits too much log data with INFO level which makes it difficult to separate important realtime metrics/events from low level stuff useful only for troubleshooting. E.g.:
`
2024-12-23 12:14:42.147 INFO default datomic.kv-cluster - {:event :kv-cluster/create-val, :val-key "67695432-37bb-4155-9731-0ad26be7e1de", :bufsize 882, :msec 5.65, :phase :end, :pid 9, :tid 692}
2024-12-23 12:14:42.151 INFO default datomic.kv-cluster - {:event :kv-cluster/create-val, :val-key "67695432-11c9-4b95-85dc-1d081032a132", :bufsize 1170, :msec 6.64, :phase :end, :pid 9, :tid 694}

...

2024-12-23 12:32:35.033 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 70, :pid 9, :tid 5}
2024-12-23 12:32:35.396 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 63, :pid 9, :tid 5}
2024-12-23 12:32:35.749 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 49, :pid 9, :tid 5}
2024-12-23 12:32:36.135 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 68, :pid 9, :tid 5}
2024-12-23 12:32:36.487 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 65, :pid 9, :tid 5}
2024-12-23 12:32:36.853 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "GCLocker Initiated GC", :event :gc, :duration 55, :pid 9, :tid 5}
2024-12-23 12:32:37.239 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 73, :pid 9, :tid 5}
2024-12-23 12:32:37.615 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 65, :pid 9, :tid 5}
2024-12-23 12:32:37.961 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 69, :pid 9, :tid 5}
2024-12-23 12:32:38.285 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 58, :pid 9, :tid 5}
2024-12-23 12:32:38.713 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 68, :pid 9, :tid 5}
2024-12-23 12:32:39.039 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 62, :pid 9, :tid 5}
2024-12-23 12:32:39.364 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 66, :pid 9, :tid 5}
2024-12-23 12:32:39.721 INFO default datomic.log-gc - {:gcName "G1 Young Generation", :gcAction "end of minor GC", :gcCause "G1 Evacuation Pause", :event :gc, :duration 78, :pid 9, :tid 5}
2024-12-23 12:32:39.923 INFO default datomic.lifecycle - {:tid 19, :username "krVN7t+5clwdg5/HazGJsrE0ckBlt50uk+HlYn9vbwY=", :port 4334, :rev 4243344, :host "0.0.0.0", :pid 9, :event :transactor/heartbeat, :version "1.0.7187", :timestamp 1734957159894, :encrypt-channel true}
`

It would be nice to review log levels and have a paper on messages generated with INFO level. Also the log line seems to be EDN for which there is no parser on ELK/Grafana side. Can this be made JSON to enable structural logging?

How to deprecate a composite tuple?

Sun, 24 Nov 2024 09:06:05 +0000

As Datomic does not allow deleting attributes from schema we use the policy of adding a prefix to the attributes that are no longer used, e.g. :foo/bar becomes :foo/#bar. We then stop inserting deprecated attributes in the app code. This way old entities get to keep deprecated attributes, new ones do not have them. This works with all kinds of attributes, however this is not sufficient for composite tuples as Datomic calculates/inserts their values automatically.

Is there a way to instruct Datomic to stop populating a composite tuple?

how do I connect to a datomic transactor from a tomcat servlet?

Thu, 01 Aug 2024 21:39:06 +0000

i have a transactor running that I can connect to via the repl and make transactions, but whenever I go to launch tomcat, i get a FileNotFoundException because it can't find datomic.clj

Why does Datomic block shutting down when using :timeout in d/query?

Tue, 16 Jul 2024 12:19:45 +0000

We started adding a :timeout to our d/query calls, and noticed that this made our app and test suite not shutdown properly anymore. I have pinpointed this to an thread pool-2-thread-1 that is keeping the VM alive, because it does not have the daemon flag set. If we don't use :timeout, this thread is not being created.

It seems that the ThreadFactory used by datomic.datalog/cancel-service does not set this daemon flag. Is this intentional?

As a workaround we now use this, which seems to work and doesn't block shutdown:

(.setThreadFactory datomic.datalog/cancel-service
                   (reify java.util.concurrent.ThreadFactory
                     (newThread [_this runnable]
                       (doto (Thread. runnable)
                         (.setName "cancel-service")
                         (.setDaemon true))

I think the current behavior is a bug/oversight, and it would be great if these threads were made daemons by default in the future (just like every other thread Datomic seems to create).

is there a "un-divert" to undo divert-system

Wed, 15 May 2024 18:23:15 +0000

after using datomic.dev-local/divert, is there a way to undo this in a repl to switch back to the original endpoint?

http-direct configuation not working as per tutorials

Tue, 07 May 2024 19:17:52 +0000

I cannot get http-direct working. The AWS configuration consoles have changed and the tutorial does not appear to work.

Specifically: in the API Gateway->APIs->Create API creation I appear to have to choose between HTTPapi or RESTapi. Between the two, the tutorial appears to follow the REST Api route (before HTTP was an option apparently). If I do that, the load balancer does not show up as an option. The vpc links are created for either HTTP or REST. There is not a LoadBalancer for REST.

I have been able to work through getting a request routed to the LoadBalancer with the preconfigured 8184 listener but I get a Ion excution failed and I'm too far off the tutorial to understand what I need.

How to write the Metaschema file?

Mon, 22 Apr 2024 09:54:04 +0000

I'm learning the Analytics Support of Datomic Pro. I have configured the Transactor, Peer Server, and Presto Server and ensured that they can run properly.
I am trying to use PrestoJDBC to query the content in the Datomic database through SQL statements. By reading the Datomic Pro documentation, I realized that I need to write a Metaschema file to implement this operation. But I didn't find any examples of Metasahema files on the internet.
So I would like to seek help information related to Metaschema file examples.If I misunderstood, please point out my mistake!
I really need your help in the short term!
Thank you!

Using Valcache via Docker bind mounts

Thu, 11 Apr 2024 09:50:54 +0000

I'm evaluating caching options for Datomic Pro. Valcache sounds like a great fit, but there is one thing that might be a showstopper.

The Prerequisites docs says that "Valcache relies on an SSD with the strictatime and lazytime flags set". This requirement I can meet pretty easily by remounting the disk on the hosts running the peers.

Now, the applications that are Datomic peers, who would be the users of these Valcaches, run as Docker containers in a Docker Swarm. The idea is to mount the Valcache as a Docker bind mount.

Would this work? The underlaying SSD disk would be mounted with strictatime and lazytime, but would whatever mechanism that requires these flags to be set work through a Docker bind mount?

How do I restart a gc-deleted-dbs after it got canceled in "Deleting garbage segments" phase?

Tue, 19 Mar 2024 04:10:45 +0000

I recently (backed up and) deleted a fairly large database in production, and had gotten partway through reclaiming the space in Postgres with bin/datomic -Ddatomic.gcStoragePaceMsec=5 gc-deleted-dbs [...] when I wanted to speed it up by removing the pacing, so I stopped it with Ctrl-C. The docs explicitly say that "You can kill a gc-deleted-dbs process and restart it later with no adverse affects" and I had previously done exactly that to speed up the pacing once before.

Unfortunately, it no longer recognizes any need to clean up garbage segments, the third stage (after log and index segments). Adding a new temporary database, filling it with a bunch of junk transactions, and deleting it, successfully goes through the whole process for that DB only, but does not trigger any additional garbage segment cleanup. (If I don't add the junk transactions it skips that stage entirely.)

Possibly security issue with the dev H2 storage?

Wed, 13 Mar 2024 21:58:26 +0000

While trying to understand more about the behaviour of Datomic dev (mainly from a security/encryption point of view), I attempted to connect to datomic.mv.db from an instance of the H2 console. I accidentally tried to log in as "admin" with a blank password and was surprised when I connected successfully. Connecting as "admin" with any password fails, so it seems explicitly to be blank. As far as I can tell I only have public schema access and can't see any transacted data, but I am able to run SQL queries in the console.

Clearly a dev transactor isn't intended to be high security, but admin with a blank password seems risky even for a dev environment. In most SQL environments I work with, the sa/admin account is just disabled if not used. Am I missing something?

As an aside, does the dev transactor use H2's AES encryption or is the resulting data file unencrypted?

import-cloud fails with "No reader function for tag object"

Wed, 14 Feb 2024 18:08:30 +0000

The recent release of version 1.0.277 of datomic-local allowed me to run import-cloud to completion successfully. Unfortunately, the resulting database is unusable. When I try to connect, I get this error message:

{:type java.lang.RuntimeException
:message "No reader function for tag object"
:at [clojure.lang.EdnReader$TaggedReader readTagged "EdnReader.java" 801]}]
:trace
[[clojure.lang.EdnReader$TaggedReader readTagged "EdnReader.java" 801]
[clojure.lang.EdnReader$TaggedReader invoke "EdnReader.java" 783]
[clojure.lang.EdnReader$DispatchReader invoke "EdnReader.java" 549]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$VectorReader invoke "EdnReader.java" 672]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader read "EdnReader.java" 145]
[clojure.lang.EdnReader read "EdnReader.java" 111]
[clojure.lang.EdnReader readString "EdnReader.java" 67]
[clojure.edn$read_string invokeStatic "edn.clj" 46]
[clojure.edn$read_string invokeStatic "edn.clj" 37]
[clojure.edn$read_string invoke "edn.clj" 37]
[datomic.dev_local.btindex_set$read_latest_root invokeStatic "btindex_set.clj" 99]
[datomic.dev_local.btindex_set$read_latest_root invoke "btindex_set.clj" 95]
[datomic.dev_local.btindex_set$current_index_files invokeStatic "btindex_set.clj" 149]
[datomic.dev_local.btindex_set$current_index_files invoke "btindex_set.clj" 146]
[datomic.dev_local.btindex_set$old_index_files invokeStatic "btindex_set.clj" 159]
[datomic.dev_local.btindex_set$old_index_files invoke "btindex_set.clj" 156]
[datomic.dev_local.btindex_set$delete_old_index_files invokeStatic "btindex_set.clj" 176]
[datomic.dev_local.btindex_set$delete_old_index_files invoke "btindex_set.clj" 169]
[datomic.dev_local.btindex_db$delete_unused_files invokeStatic "btindex_db.clj" 604]
[datomic.dev_local.btindex_db$delete_unused_files invoke "btindex_db.clj" 602]
[datomic.dev_local.impl$create_durable_connection invokeStatic "impl.clj" 189]
[datomic.dev_local.impl$create_durable_connection invoke "impl.clj" 187]
[datomic.dev_local.impl$create_connection invokeStatic "impl.clj" 205]
[datomic.dev_local.impl$create_connection invoke "impl.clj" 201]
[datomic.dev_local.impl.DurableClient connect "impl.clj" 271]
[datomic.client.api$connect invokeStatic "api.clj" 151]
[datomic.client.api$connect invoke "api.clj" 138]
[st.system$ensure_db_BANG_ invokeStatic "system.clj" 117]
[st.system$ensure_db_BANG_ invoke "system.clj" 114]
[st.system$eval57307$fn__57309$fn__57311 invoke "system.clj" 122]
[clojure.core$mapv$fn__8535 invoke "core.clj" 6979]
...

Note that I use a custom credentials provider and it appears as an opaque java object in the resulting db-root-00000000 file.

Why is the pull pattern for wild card giving me an error?

Wed, 31 Jan 2024 16:27:11 +0000

I am having an issue with a query and wild card patterns:
(d/q {:query '[:find (pull ?page pattern)

                         :in $ ?id pattern
                         :where
                         [?page :page/id ?id]]
                  :args [(d/db conn)
                         id
                         '[*]]})

It gives me an error:
Attribute identifier clojure.core/* of class: class java.lang.String does not start with a colon
Any idea why it doesn't work? I also tried adding the pattern directly to the pull expression, but I get the exact same issue

Are DB values obtained from a single conn guaranteed to include transactions submitted against the same connection?

Wed, 31 Jan 2024 05:03:55 +0000

It stands to reason that all transactions submitted (via datomic.api/transact) and transacted (demonstrated via deref) on a single peer would be included in all subsequent DB values obtained (via datomic.api/db) on the Peer, but I'd love to know if that is guaranteed.

For example, if a test transacts data resulting in some T, and the rest of the test only cares that it runs against a DB value with T' >= T, is the following sufficient?:

@(datomic.api/transact conn tx) ; T (let [db' (datomic.api/db conn)] ; T' >= T? ,,(do-something-with db')

Or is it strictly necessary to do:

(let [{:keys [db-after]} @(datomic.api/transact conn tx)] ,,(do-something-with db-after))

I know that the former tends to work, but I'm particularly interested if it's guaranteed to work.

I do understand that additional transactions from other threads / clients / etc. may have been included in db' in the first scenario, so I realize the code is not equivalent in the general case. I'm only concerned about tx having been transacted and included in the fetched db' value.

Thank you!

(Apologies for formatting—I can't seem to convince this thing to care about my whitespace.)

Multiple CVEs are reported on Datomic Peer 1.0.7075

Mon, 15 Jan 2024 07:50:02 +0000

Starting about two weeks ago, OWASP Dependency Check (via nvd-clojure) is reporting multiple CVEs on Datomic. It's in the package memcache-asg-java-client-1.1.0.36.jar (file path $HOME/.m2/repository/com/datomic/memcache-asg-java-client/1.1.0.36/memcache-asg-java-client-1.1.0.36.jar). They are reported as level "CRITICAL". The reported CVEs are:

Partial dependency tree (clojure -Stree):

[own code]
  . com.datomic/peer 1.0.7075
    . com.datomic/memcache-asg-java-client 1.1.0.36

At the very least I want to make sure that you're aware of this, and I'd also appreciate your advice on whether this is ACTUALLY an issue in Datomic context.

Thank you!

java.lang.IllegalStateException: Can't set!: io-index from non-binding thread

Sat, 13 Jan 2024 22:49:31 +0000

I'm getting the following error in the transactor when trying to transact some data using a transaction function. It only happens for a specific query and set of data.

I'm not sure about the correct place to report errors like this, so feel free to direct me to another place if this is not the correct place.

Also let me know if you need any other information.

I am using datomic version 1.0.7075, peer and transactor have the same version.

I'm using JDK 21, Temurin (datomic transactor is running inside clojure:temurin-21-tools-deps-alpine docker container)

taskfeed-datomic  | java.lang.IllegalStateException: Can't set!: *io-index* from non-binding thread
taskfeed-datomic  |     at clojure.lang.Var.set(Var.java:223)
taskfeed-datomic  |     at datomic.measure.io_stats$using_index_BANG_.invokeStatic(io_stats.clj:57)
taskfeed-datomic  |     at datomic.measure.io_stats$using_index_BANG_.invoke(io_stats.clj:54)
taskfeed-datomic  |     at datomic.index.Index.seek(index.clj:553)
taskfeed-datomic  |     at datomic.btset$seek.invokeStatic(btset.clj:399)
taskfeed-datomic  |     at datomic.btset$seek.invoke(btset.clj:394)
taskfeed-datomic  |     at datomic.db.Db.seekEAVT(db.clj:2410)
taskfeed-datomic  |     at datomic.query$eav.invokeStatic(query.clj:78)
taskfeed-datomic  |     at datomic.query$eav.invoke(query.clj:69)
taskfeed-datomic  |     at datomic.query.EntityMap.valAt(query.clj:213)                                                                                                                               taskfeed-datomic  |     at datomic.query.EntityMap.entryAt(query.clj:199)                                                                                                                             taskfeed-datomic  |     at clojure.lang.RT.find(RT.java:860)
taskfeed-datomic  |     at clojure.core$select_keys.invokeStatic(core.clj:1562)
taskfeed-datomic  |     at clojure.core$select_keys.invoke(core.clj:1555)
taskfeed-datomic  |     at ns_272$eval273$fn__274$fn__275$fn__276.invoke(NO_SOURCE_FILE:143)
taskfeed-datomic  |     at clojure.walk$prewalk.invokeStatic(walk.clj:65)
taskfeed-datomic  |     at clojure.walk$prewalk.invoke(walk.clj:61)
taskfeed-datomic  |     at clojure.core$partial$fn__5908.invoke(core.clj:2641)
taskfeed-datomic  |     at clojure.core$map$fn__5935.invoke(core.clj:2770)
taskfeed-datomic  |     at clojure.lang.LazySeq.sval(LazySeq.java:42)
taskfeed-datomic  |     at clojure.lang.LazySeq.seq(LazySeq.java:51)
taskfeed-datomic  |     at clojure.lang.RT.seq(RT.java:535)
taskfeed-datomic  |     at clojure.core$seq__5467.invokeStatic(core.clj:139)
taskfeed-datomic  |     at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
taskfeed-datomic  |     at clojure.core$reduce.invokeStatic(core.clj:6886)
taskfeed-datomic  |     at clojure.core$into.invokeStatic(core.clj:6958)
taskfeed-datomic  |     at clojure.walk$walk.invokeStatic(walk.clj:50)
taskfeed-datomic  |     at clojure.walk$prewalk.invokeStatic(walk.clj:65)
taskfeed-datomic  |     at clojure.walk$prewalk.invoke(walk.clj:61)
taskfeed-datomic  |     at clojure.core$partial$fn__5908.invoke(core.clj:2641)
taskfeed-datomic  |     at clojure.core$map$fn__5935.invoke(core.clj:2772)
taskfeed-datomic  |     at clojure.lang.LazySeq.sval(LazySeq.java:42)
taskfeed-datomic  |     at clojure.lang.LazySeq.seq(LazySeq.java:51)
taskfeed-datomic  |     at clojure.lang.Cons.next(Cons.java:39)
taskfeed-datomic  |     at clojure.lang.RT.next(RT.java:713)                                                                                                                                          taskfeed-datomic  |     at clojure.core$next__5451.invokeStatic(core.clj:64)
taskfeed-datomic  |     at clojure.core.protocols$fn__8249.invokeStatic(protocols.clj:169)
taskfeed-datomic  |     at clojure.core.protocols$fn__8249.invoke(protocols.clj:124)
taskfeed-datomic  |     at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
taskfeed-datomic  |     at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
taskfeed-datomic  |     at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
taskfeed-datomic  |     at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)                                                                                                        [3165/9184]
taskfeed-datomic  |     at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
taskfeed-datomic  |     at clojure.core$reduce.invokeStatic(core.clj:6886)
taskfeed-datomic  |     at clojure.core$into.invokeStatic(core.clj:6958)
taskfeed-datomic  |     at clojure.walk$walk.invokeStatic(walk.clj:50)
taskfeed-datomic  |     at clojure.walk$prewalk.invokeStatic(walk.clj:65)
taskfeed-datomic  |     at clojure.walk$prewalk.invoke(walk.clj:61)
taskfeed-datomic  |     at ns_272$eval273$fn__274$fn__275.invoke(NO_SOURCE_FILE:93)
taskfeed-datomic  |     at clojure.core$pmap$fn__8552$fn__8553.invoke(core.clj:7089)
taskfeed-datomic  |     at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
taskfeed-datomic  |     at clojure.lang.AFn.call(AFn.java:18)
taskfeed-datomic  |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
taskfeed-datomic  |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
taskfeed-datomic  |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
taskfeed-datomic  |     at java.base/java.lang.Thread.run(Thread.java:1583)

On the client side, I get the following error:

taskfeed-1  | 01:09:47 [clojure-agent-send-off-pool-21] WARN  datomic.connector - {:message "error executing future", :pid 1, :tid 97} {  }
taskfeed-1  | java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Throwable (java.lang.String and java.lang.Throwable are in module java.base of loader 'bootstrap'
)
taskfeed-1  |   at user$eval172500$fn__172501.invoke(NO_SOURCE_FILE)
taskfeed-1  |   at datomic.error$deserialize_exception.invokeStatic(error.clj:178)
taskfeed-1  |   at datomic.error$deserialize_exception.invoke(error.clj:160)
taskfeed-1  |   at datomic.peer.Connection.notify_error(peer.clj:388)
taskfeed-1  |   at datomic.connector$fn__16648.invokeStatic(connector.clj:154)
taskfeed-1  |   at datomic.connector$fn__16648.invoke(connector.clj:152)
taskfeed-1  |   at clojure.lang.MultiFn.invoke(MultiFn.java:234)
taskfeed-1  |   at datomic.connector$create_hornet_notifier$fn__16654$fn__16655$fn__16659$fn__16660.invoke(connector.clj:178)
taskfeed-1  |   at datomic.connector$create_hornet_notifier$fn__16654$fn__16655$fn__16659.invoke(connector.clj:173)
taskfeed-1  |   at datomic.connector$create_hornet_notifier$fn__16654$fn__16655.invoke(connector.clj:171)
taskfeed-1  |   at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
taskfeed-1  |   at clojure.lang.AFn.call(AFn.java:18)
taskfeed-1  |   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
taskfeed-1  |   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
taskfeed-1  |   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
taskfeed-1  |   at java.base/java.lang.Thread.run(Thread.java:1583)

And then a few seconds later, it throws the following error:

taskfeed-1  | Caused by:  :db.error/transaction-timeout Transaction timed out.
taskfeed-1  | clojure.lang.ExceptionInfo: :db.error/transaction-timeout Transaction timed out.                                                                                                        taskfeed-1  | #:db{:error :db.error/transaction-timeout}
taskfeed-1  |  at datomic.error$raise.invokeStatic (error.clj:70)
taskfeed-1  |     datomic.error$raise.invoke (error.clj:58)
taskfeed-1  |     datomic.error$raise.invokeStatic (error.clj:68)
taskfeed-1  |     datomic.error$raise.invoke (error.clj:58)
taskfeed-1  |     datomic.peer$await_tx_result.invokeStatic (peer.clj:79)
taskfeed-1  |     datomic.peer$await_tx_result.invoke (peer.clj:70)
taskfeed-1  |     datomic.peer.Connection.transact (peer.clj:315)
taskfeed-1  |     datomic.api$transact.invokeStatic (api.clj:107)
taskfeed-1  |     datomic.api$transact.doInvoke (api.clj:105)

how to run a dev transactor from the same process as a peer?

Wed, 10 Jan 2024 22:30:06 +0000

At scale you definitely want a separate transactor, but for development or for deploying an early prototype , it would be very handy to be able to have the transactor running in the same process as the peer. This would trade reliability and predictability for resource efficiency, and allow deploying on an inexpensive instance somewhere in a public cloud. This would probably lower the bar for trying out datomic in many settings.

In fact, it seems that datomic local allows having an embedded H2 database and a transactor into the same JVM process as your client.

You mentioned that it is possible to do the same for datomic on-prem, could you detail how one might do it?

Is it possible to run a transactor within the scope of my application?

Sun, 03 Dec 2023 11:52:25 +0000

Hi,

Is it possible to run a transactor from within a Clojure application? I'm trying to gain more control over/feedback about the state of the transactor, and simply shelling out to bin\transactor doesn't seem like a neat solution. I'd also prefer that the end user doesn't have to provide and maintain their own transactor.

PS, I understand concerns about contention for resources, but what I envision is a simple wrapper application that runs in its own JVM process separate from the rest of my application logic.

Is it possible to separate Datomic On-Prem from Cloud?

Sun, 03 Dec 2023 11:48:45 +0000

I see that AWS-specific jars comprise more than half the size of Datomic install; adding these 300+ jars to the classpath also makes Datomic unusable in Windows due to "input line is too long".

As an exclusively On-Prem user, is there any way to omit these unnecessary components and still have a viable Datomic On-Prem transactor? Or have the two capabilities been too-deeply intermingled for this to be possible?

Equivalents for d/with; transaction function test scaffold story when using cloud

Wed, 01 Nov 2023 11:24:08 +0000

Is there absolutely no way to access the peer api on datomic cloud?

My thinking is that, conceptually, since it uses clients, that implies that peer servers do exist.

I realise that light horizontal scaling is the point in DC, but the peer api, and in particular d/with I'm finding important to allow test scaffolds for transaction functions.

So the thought here is it would be nice to be able to access the richer peer api just for one off tasks, such as managing schema or even here making speculative databases so one can create a test scaffold to test a more complex transaction function.

I don't know what the test scaffold story is with DC or if there is one. Bit of a dark area on the map for me at the moment so any suggestions would be appreciated.

I guess one last-resort option is to develop using Pro and test transaction functions that way. Then copy them over to DC. I have found though that in a previous iteration trying to think about Pro and Cloud at the same time as designing for your domain is something I would want to avoid where possible.

Update: I've just realised that the client api does actually include d/with, but I'll leave the question up anyway, in case anyone has any related thoughts they'd like to share.

Further Update: looking back at my initial inept attempts to build a datomic system and schema which utilised transaction functions to avoid race conditions given a complex content hierarchy in the domain I am modelling, it looks like the reason I felt forced to use the Peer API and not Client, for those Database Functions, was the absence of d/function in the Client API. I wonder if Client API actually is fully capable in terms of installing and testing database functions, but that wasn't clear to me at the time. So much is required to be able to develop in Datomic. It might be that my missing a little detail of Clojure/Java design (an easy syntactic alternative to d/function?) ended up pushing my design decision in an unnecessary direction.
I suppose the answer may be, re. DB functions - "Client API yes", "Cloud no"?
(Illustrating the point that it is best not to try to think about Pro and Cloud at the same time...)

Remaining question: I'm finding it very difficult to find out what the difference is between d/function on the peer API, vs. use of the #db/fn literal. Cannot locate an explanation in the docs. GPT is telling me the latter is for an inline database function which will not be installed, whereas d/function is for installing the fn into the database, but it I know it could be hallucinating in such a case.

Database functions, Classpath functions, and Transaction functions

Wed, 01 Nov 2023 08:56:45 +0000

Datomic Pro documentation, reference/database functions states:

You can deploy code to a Datomic system by installing database functions into a Datomic database, or by adding classpath functions to the Java classpath. Both database functions and classpath functions can be used as transaction functions that run inside a transaction to enforce arbitrary data invariants.

I understand that either kind "can be used as" a transaction function.
But in reality is this the only way either is used in practice?

I've been trying to unfangle what an implied other kind (that isn't used as a transaction function) could be and haven't found any examples.

So the conditions regarding taking the db value as first argument, and being pure function, actually apply to every function, whether database or classpath, that we may write?

Is there way to use s3 compatible backup storage

Mon, 16 Oct 2023 18:17:34 +0000

We use google cloud and managing the backup persistent disks is not a huge burden but still less than ideal. Google cloud storage offers s3 compatible storage, but we would need to know how to provide their endpoint in the URI.

Why do values of :db.type/uri return as com.cognitect.transit.impl.URIImpl?

Mon, 02 Oct 2023 03:56:45 +0000

I've recently worked with dev-local try out Datomic and when I moved to Cloud I noticed the following scenario, whenever I transact a fact that is of the java.net.URI type and try to query it back I get com.cognitect.transit.impl.URIImpl back as the type, I have found no documentation about this, I was expecting that the types remain the same, is there any way to disable this behavior or at least convert back to java.net.URI?

This matters because I had some code break because it was expecting java.net.URI and because this behavior is undocumented (or if documented inaccessible by searching the Datomic documentation and search engines)

Thank you for your time

deps.edn

org.clojure/clojure {:mvn/version "1.11.1"}
com.datomic/client-cloud {:mvn/version "1.0.123"}

How to change the default 1 minute jetty async servlet timeout?

Thu, 10 Aug 2023 16:01:07 +0000

We have some ring routes in our HTTP direct ring handler, which returns some Excel files, which are constructed on the fly.

This handler sometimes runs for more than a minute, resulting in the following error:
HTTP ERROR 503 Async servlet timeout URI: /... STATUS: 503 MESSAGE: Async servlet timeout SERVLET: cognitect.http_endpoint.jetty.FnServlet-49bf6f8c Powered by Jetty:// 9.4.48.v20220622

How can we raise this timeout to ~5minutes, which is the maximum timeout of AWS NLBs?

(We are exposing the HTTP direct ion endpoint to the public internet via an AWS NLB, to avoid the 30s timeout and 10MB response payload limit of the AWS API Gateway)

Multiple sources on the internet - eg. https://github.com/ring-clojure/ring/issues/299 - mention a org.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT Java property which is supposed to affect Jetty's async context timeout.

Since the change of this property at runtime is not picked up by a running jetty instance, we have added provided it as a startup option, via the OverrideSettings CloudFormation template parameter:

`
{:OverrideSettings
(str "export JVM_FLAGS=\"$JVM_FLAGS"

  " -Dclojure.server.repl="
  "{:address,\\\"0.0.0.0\\\",:port,50505,:accept,clojure.core.server/repl}"
  " -Dorg.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT=300000"
  "\"")}

It is indeed observable within the ion JVM process:
(Long/getLong "org.eclipse.jetty.server.HttpChannelState.DEFAULT_TIMEOUT") => 300000

Yet we are getting a 503 timeout response from the cognitect.http_endpoint.jetty.FnServlet-49bf6f8c servlet after 1 minute still.

Request, which take less then a minute work as expected.

Is `(pull $db2 ?e [*])` supported officially?

Sat, 15 Jul 2023 04:42:45 +0000

The documentation only mentions the syntax without a src-var:

pull-expr = ['pull' variable pattern]

but found that this syntax also works:

pull-expr = ['pull' src-var variable pattern]

I have a self-contained code example here, which demonstrates a use-case, where an entity is split between 2 Datomic databases, for privacy reasons:

https://forum.datomic.com/t/pull-db-entity-pattern-is-undocumented/2299

Is this variant of the pull-expr in Datalog queries are not supported officially?

Should I use it in production or does it have some undefined behaviour, compared to its binary arity variant?

Or the documentation is just incomplete?

Datomic Cloud IAM role drift on setup - is this normal?

Wed, 12 Jul 2023 09:34:33 +0000

hi, just set up DC for the first time (ran the two AWS cloudformation templates). The storage stack is showing drift on the IAM managed policy datomic-admin-..., shows 'MODIFIED', in red. Is this normal upon setup? (checking the changes in the 'versions' tab, the difference is addition of autoscaling and s3 actions to the policy; the second version, presumably causing the 'drift', was added during the provisioning). Is this just a setup artifact nothing to worry about?

How create more one production compute?

Wed, 05 Jul 2023 13:54:33 +0000

Hello,
I need to use my existing storage with additional compute and tried create new production compute with this system name(storage stack name). And I get creation error:

Export with name myname-VpcLinkId is already exported by stack myname-compute. Rollback requested by user.

I understand that this is due to a name conflict, but I don’t understand how then to correctly create a compute.

I want to run my application in a separate server(сompute) but use the same storage.

Support for DynamoDB compatible services

Thu, 11 May 2023 12:18:56 +0000

Hi!
There is a DynamoDB compatible service called YDB.
YDB in the Yandex Cloud has a serverless mode with DynamoDB support..

So I have:

AWS_ACCESS_KEY_ID
AWS_SECRET_KEY
region: ru-central1
endpoint like https://docapi.serverless.yandexcloud.net/ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1

I have tried using the ddb and ddb-local protocols with different settings.

Variants of aws-dynamodb-override-endpoint
1. docapi.serverless.yandexcloud.net/ru-central1/.../...
2. docapi.serverless.yandexcloud.net:443/ru-central1/.../...
3. https://docapi.serverless.yandexcloud.net/ru-central1/.../...

$ bin/datomic ensure-transactor config/ddb.properties config/ddb-gen.properties
java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in authority at index 7: http://"https://docapi.serverless.yandexcloud.net/ru-central1/b1g31q09ln3ca15cg5mf/etnckvjnbskiko1s639l"
	at com.amazonaws.util.RuntimeHttpUtils.toUri(RuntimeHttpUtils.java:216)
	at com.amazonaws.util.RuntimeHttpUtils.toUri(RuntimeHttpUtils.java:190)
	at com.amazonaws.AmazonWebServiceClient.toURI(AmazonWebServiceClient.java:329)
	at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:317)
	at datomic.ddb$client.invokeStatic(ddb.clj:14)

I think that ddb-local always uses http protocol and there is no way to configure it with https.

There are some examples of using AwsSDK with YDB in Yandex cloud

AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
          .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("<Document API endpoint>", "ru-central1"))
          .build();

Could you add support for DynamoDB compatible services?
Could it be the new `aws-dynamodb-override-protocol' setting, which is by default 'http' ?

Why is this error message saying the entity can't be resolved?

Thu, 04 May 2023 22:53:42 +0000

Here is the error message:

cognitect.anomalies/message: "Unable to resolve entity: 1.7592186046217E13 in datom [\"shape_VVVVVVVVVVVVVVVV_0_1\" :shape/source-shape 1.7592186046217E13]"
                           datom: ["shape_VVVVVVVVVVVVVVVV_0_1" :shape/source-shape 1.7592186046217E13]
                        db/error: :db.error/not-an-entity
                             dbs: [{:database-id
                                    "datomic:dev://datomic-transactor:4334/blah",
                                    :t 1818,
                                    :next-t 1819,
                                    :history false}]
                          entity: 1.7592186046217E13

I can query for the :db/id 17592186046217, so I know it exists. My question is why can't Datomic find it? There's only one transaction and it failed, so it is guaranteed that entity 17592186046217 existed before the transaction as well.

This is the schema for :shape/source-shape:

{:db/ident :shape/source-shape
 :db/valueType :db.type/ref  
 :db/cardinality :db.cardinality/one}

The version of Datomic is datomic-pro-1.0.6269.

Will dev-local be available in Maven Central?

Sun, 30 Apr 2023 21:30:38 +0000

With Datomic going free (thank you!), that's one less special configuration required for a password protected Maven repository.

Any chance dev-local will end up in Maven Central as well like datomic-free used to be? This would make initial dev setup much easier as developers would not need to configure any password protected Maven repositories to get started.

Having dev-local in Maven central would make it possible to jump directly into development on something like lambdaisland/plenish without any additional setup.

Can a tuple value be added when one of the parts of the tuple is an lookup-ref?

Thu, 20 Apr 2023 20:09:32 +0000

I'm wanting to use a lookup ref instead of an entity id (:db-id) when specifying the value for a tuple of attributes where one of the attributes is of type :db.type/ref. Is this possible? I've looked over the Datomic Cloud grammar on transactions and I don't see how to specify a tuple value at all (maybe I'm missing something). I've tried what I thought would be the obvious way to do it and this doesn't work. Here's what I have...

Truncated schema:

{:db/ident       :molecule/legacy-id
 :db/unique      :db.unique/identity
 :db/valueType   :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc         "The Molecule/Complex id from the ProcessDB Oracle database."}
{:db/ident       :molecule-part/number
 :db/valueType   :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc         "The number of molecules for this molecule. The entity with this attribute may be reused and is
 not unique for a particular complex so do not change this number. Instead, check for another existing entity
 or create one."}
{:db/ident       :molecule-part/molecule
 :db/valueType   :db.type/ref
 :db/cardinality :db.cardinality/one
 :db/doc         "The molecule this part refers to. The entity with this attribute may be reused and is
 not unique for a particular complex."}
{:db/ident       :molecule-part/number+molecule
 :db/unique      :db.unique/identity
 :db/valueType   :db.type/tuple
 :db/tupleAttrs  [:molecule-part/number :molecule-part/molecule]
 :db/cardinality :db.cardinality/one
 :db/doc         "The number and molecule that collectively form the unique identity of this entity."}

Then I try to add a molecule part using

(d/transact (db/get-connection) {:tx-data [{:molecule-part/molecule [:molecule/legacy-id 717] :molecule-part/number 1 :molecule-part/number+molecule [1 [:molecule/legacy-id 717]]}]})

And get the following error

:db.error/invalid-tuple-value Invalid tuple value

If I replace "[:molecule/legacy-id 717]" with the entity id (:db/id) for that molecule then it works. But I don't want to use the entity id... I want to use the lookup ref, because what I have is the :molecule/legacy-id. I don't want to have to do a query to resolve this before doing a transaction to add the :molecule-part.

Is there only one deployment target for ions?

Sun, 16 Apr 2023 19:26:10 +0000

I have deployed and used the ion-starter project. I see that I can have multiple databases and multiple connections all of which work.

But I fail when I try to create an ion from another project folder. the {:ion-dev "push"} appears to work but deployment fails.

Should it be a "new" stack? a different deploy group? Or for ions is there only one project folder that must be used for all ions for a datomic instance?

Backup Datomic Cloud to another account

Fri, 31 Mar 2023 13:55:41 +0000

Is there a way to backup a datomic cloud instance to another aws account? I know that cloud is based on HA technology inside the AWS, (dynamo, s3, etc) but backups are important for business continuity. In the event of an attack, everything can be deleted within our aws account with one breach. I can make the tools to copy s3 items, it does not seem possible to import an existing s3 db into datomic.

What defines a datomic on-prem system?

Wed, 29 Mar 2023 20:55:33 +0000

The price for an on-prem pro license is for one system. What defines a system? Does a single system include a transactor, a stand-by transactor and multiple peers?

Request Entity Too Large - clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

Tue, 21 Mar 2023 12:26:20 +0000

There seems to be a bug in Datomic 990-9202 that breaks the d/tx-range function.

Earlier I could use the tx-range function the fetch data from my database.

But after upgrading to Datomic 990-9202 I get and exception when a transaction has too much data.

`
clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

cognitect.anomalies/category: :cognitect.anomalies/incorrect
 cognitect.anomalies/message: "Response body did not conform to Datomic client protocol"
                 http-result: {:status 413,
                               :headers
                               {"apigw-requestid" "Vg4UxisCDoEEJsQ=",
                                "server" "Jetty(9.4.44.v20210927)",
                                "connection" "keep-alive",
                                "content-length" "38",
                                "date" "Tue, 19 Jul 2022 12:17:47 GMT",
                                "content-type" "application/json"},
                               :body "{\"message\":\"Request Entity Too Large\"}"}

How can I connect to Datomic after upgrade to 990-9202, the bastion disappeared?

Fri, 17 Mar 2023 09:42:45 +0000

Hi!

After I upgraded Datomic cloud to version 2023/02/28 990-9202 the bastion disappeared in AWS. So now I can't connect to Datomic anymore.

When I try to connect to it with datomic client access <SYSTEM-NAME>

I get this error:

`
{:clojure.main/message
"Execution error at datomic.tools.ops.aws/get-ip (aws.clj:116).\nGateway not found, make sure gateway is running.\n",
:clojure.main/triage
{:clojure.error/class java.lang.RuntimeException,
:clojure.error/line 116,
:clojure.error/cause
"Gateway not found, make sure gateway is running.",
:clojure.error/symbol datomic.tools.ops.aws/get-ip,
:clojure.error/source "aws.clj",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type java.lang.RuntimeException,

:message "Gateway not found, make sure gateway is running.",
:at [datomic.tools.ops.aws$get_ip invokeStatic "aws.clj" 116]}],

:trace
[[datomic.tools.ops.aws$get_ip invokeStatic "aws.clj" 116]
[datomic.tools.ops.aws$get_ip invoke "aws.clj" 113]
[datomic.tools.ops.ssh$access invokeStatic "ssh.clj" 77]
[datomic.tools.ops.ssh$access invoke "ssh.clj" 56]
[datomic.tools.ops.cli.client$eval13301$fn__13302

invoke
"client.clj"
68]

[clojure.lang.MultiFn invoke "MultiFn.java" 229]
[datomic.tools.ops$_main invokeStatic "ops.clj" 83]
[datomic.tools.ops$_main doInvoke "ops.clj" 76]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.main$main_opt invokeStatic "main.clj" 514]
[clojure.main$main_opt invoke "main.clj" 510]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]],
:cause "Gateway not found, make sure gateway is running."}}
`

What should I do to be able to connect to Datomic again?

How to get io-stats from pull and pull-many?

Wed, 08 Mar 2023 15:04:12 +0000

I tried looking here but it doesn't seem to be documented https://docs.datomic.com/on-prem/clojure/index.html

I was able to get the io-stats from query, thanks to an example on this page: https://docs.datomic.com/on-prem/api/io-stats.html#query

Is it possible for Datomic On-prem to get the storage credentials from Secrets Manager or similar software?

Thu, 02 Mar 2023 10:50:19 +0000

Is there a way to run the transactor service without storing the credentials to disk?