Datomic Knowledgebase - Questions without answers in On-Prem

Using a collection of datoms as input to a query

Thu, 24 Apr 2025 12:51:36 +0000

I was wondering if there is a way to use a collection of datoms obtained from d/tx-range + get a single tx or received from a listener that subscribed to a tx queue as input to d/q? The nature of this collection looks very similar to history db which can be used with d/q, however passing this collection as input does not produce the expected result. Neither there is any error.

Using Valcache via Docker bind mounts

Thu, 11 Apr 2024 09:50:54 +0000

I'm evaluating caching options for Datomic Pro. Valcache sounds like a great fit, but there is one thing that might be a showstopper.

The Prerequisites docs says that "Valcache relies on an SSD with the strictatime and lazytime flags set". This requirement I can meet pretty easily by remounting the disk on the hosts running the peers.

Now, the applications that are Datomic peers, who would be the users of these Valcaches, run as Docker containers in a Docker Swarm. The idea is to mount the Valcache as a Docker bind mount.

Would this work? The underlaying SSD disk would be mounted with strictatime and lazytime, but would whatever mechanism that requires these flags to be set work through a Docker bind mount?

Possibly security issue with the dev H2 storage?

Wed, 13 Mar 2024 21:58:26 +0000

While trying to understand more about the behaviour of Datomic dev (mainly from a security/encryption point of view), I attempted to connect to datomic.mv.db from an instance of the H2 console. I accidentally tried to log in as "admin" with a blank password and was surprised when I connected successfully. Connecting as "admin" with any password fails, so it seems explicitly to be blank. As far as I can tell I only have public schema access and can't see any transacted data, but I am able to run SQL queries in the console.

Clearly a dev transactor isn't intended to be high security, but admin with a blank password seems risky even for a dev environment. In most SQL environments I work with, the sa/admin account is just disabled if not used. Am I missing something?

As an aside, does the dev transactor use H2's AES encryption or is the resulting data file unencrypted?

Is it possible for Datomic On-prem to get the storage credentials from Secrets Manager or similar software?

Thu, 02 Mar 2023 10:50:19 +0000

Is there a way to run the transactor service without storing the credentials to disk?

Is using entid function to resolve lookup refs a supported behaviour?

Wed, 08 Jun 2022 13:30:29 +0000

I'm using datomic.api/entid to resolve a lookup ref to an entity ID, like
$ (entid db [:user/id "test@test.com"]) 17592186045432
This works fine, but a colleague pointed out this is not a documented behaviour.
>Usage: (entid db ident)
>Returns the entity id associated with a symbolic keyword, or the id
itself if passed.

Is this a gap in documentation, or is this behaviour accidental and not recommended to rely on?

Retracting entities, including tuples with refs

Tue, 31 Aug 2021 13:36:44 +0000

In my schema I have attributes of type tuple that has :db.type/ref as one of the tuple's types.
My usecase is maintaining a set of ordered to-many relations.

When I retract an entity using :db/retractEntity, the tuple attrs that refer to the retracted entity still remain, continuing to refer to a now non-existent entity.
I'm not sure whether this is the intended behaviour, and if so, what would be a standard approach to clean up these tuples? My intention would be for these tuples to be completely redacted. As I'm using this in a cardinality-many setup, I'd expect the attribute to retain all tuple values, except the one that does include the ref to the retracted entity in any position.
Should I be writing a custom retractEntity-like transaction function?
Thanks!

Does the transactor eagerly fill memcached with indexed segments?

Wed, 07 Apr 2021 15:21:18 +0000

The datomic transactor memcached documentation says this:

When configured to used memcached, a Datomic process will automatically write into memcached any segment it needs that is not already present in memcached. In addition, the transactor (only) will write index and log segments to memcached as the segments are produced.

(Emphasis mine.)

Based on this sentence and assuming a memcached memory pool larger than the storage size that is shared by transactor and peer, my expectation is the following:

As the transactor is building an index, it writes newly-created index segments to storage and to memcached at roughly the same time.
When the index is finished, it commits the new index root and informs all peers of the new index.
Any peers using the same memcached cluster should, at the moment the index is released:
- have a decreasing ObjectCache hitrate because newly-indexed segments are not cached
- but an increasing memcached GET count
- and a flat StorageGet and MemcachedPut count because the segments are already in memcached
- For an overall flat or increasing memcached hitrate.

Instead, what I actually see is that when peers accept a new index,

there's a dramatic decrease in Memcached hitrate
and increase in MemcachedPut counts and Storage Get counts.
The IndexWrite count reported by the transactor is very similar to the MemcachedPut count reported by the peer.

This behavior is consistent with the transactor not writing newly-indexed segments into memcached eagerly while indexing, which is contrary to the documentation.

So the question: Does the transactor really write index segments to memcached as it is indexing?

Expose an alt-port transactor configuration property

Sun, 25 Oct 2020 19:45:14 +0000

The alt-host configuration property is great and helps in many scenarios, but not having an equivalent for the port (alt-port?) is a blocker in some other scenarios. It would be great to have such property, which would default to the same port as the transactor is listening.

Allow for setting transactor properties using environment variables

Sun, 25 Oct 2020 18:08:09 +0000

Datomic should be able to read it's settings from environment variables, following a naming standard like what Spring Boot does. E.g.:

DATOMIC_LICENSE_KEY
DATOMIC_PROTOCOL
DATOMIC_PORT
…

Support AWS dynamo point in time backup in on-prem.

Sun, 25 Oct 2020 16:57:02 +0000

https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/

Currently Datomic supplies backup/restore capability here:

https://docs.datomic.com/on-prem/backup.html

Dynamo's PITR remains untested for use with Datomic DBs. Implementation of this feature would ensure that using Dynamo's PITR causes no issues on a Datomic DB.

New engine for executing java scripts on the transactor

Fri, 23 Oct 2020 13:52:54 +0000

Is it planned to consider another script executing engines than janino? There is script evaluator provided by groovy.lang.GroovyShell which can execute java scripts too. I have compared the time to evaluate the same java script by both engines and groovy engine executes significantly faster. Another positive side effect: support of groovy scripts. I believe using of new script evaluator will make improvements to the time of executing transactor functions written in java

https://docs.groovy-lang.org/latest/html/api/groovy/lang/GroovyShell.html

UPD. How much are java transactor scripts executted longer than clojure transactor scripts?

Can datomic.api/q be used inside or/or-join to query data from other DB?

Fri, 23 Oct 2020 10:53:08 +0000

Hi, when I'm executing query in repl in such way:

(d/q "[:find
 [?Post ...]
 :in
 $
 $comment
 [?Post_value_0 ...]
 :where

[?Post :IModel/typeName ?Post_value_0]
(or-join 
 [?Post ?Comment_text_value_3]

 [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

  [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

)
 [?Post :Post/comments ?Comment_uid_2]
]
" (d/db conn) (d/db connComment) #{"Post"})

The next error is returned:
:db.error/not-an-entity Unable to resolve entity: :Comment/text

My supposing is that query inside datomic.api/q is executed for database returned by (d/db conn) but not (d/db connComment) because if I create :Comment/text for the first database from input list then query is executed without errors.

If to avoid or-join and execute the query in the next way:

(d/q "[:find
 [?Post ...]
 :in
 $
 $comment
 [?Post_value_0 ...]
 :where

[?Post :IModel/typeName ?Post_value_0]
 [?Post ?Comment_text_value_3]

 [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]

  [(datomic.api/q
   \"[:find [?Comment_uid_2 ...] :in $ :where 
   [?Comment_by_text_1 :Comment/text _]
   [?Comment_by_text_1 :IModel/uid ?Comment_uid_2]
   ]\"
   $comment)
  [?Comment_uid_2 ...]]


 [?Post :Post/comments ?Comment_uid_2]
]
" (d/db conn) (d/db connComment) #{"Post"})

then query is executed without errors.

Datamic on prem 0.9.6.* version

Delete all except the latest point in time within a backup URI

Thu, 22 Oct 2020 21:46:45 +0000

Also deleting a single point in time might be useful

Provide integrity check on Datomic Backups

Thu, 22 Oct 2020 18:03:10 +0000

A tool to confirm the validity and completeness of a Datomic backup.

Add Cassandra Cluster callback to the Peer

Thu, 22 Oct 2020 17:52:42 +0000

The transactor provides a configurable cassandra cluster callback, but the peers do not. The Cassandra driver depends on old versions of metrics libraries that are incompatible with newer versions used in (for example) clojure metrics. Datastax solution for this requires use of the cluster callback https://docs.datastax.com/en/developer/java-driver/3.5/manual/metrics/#metrics-4-compatibility, but that isn't available in Peers.

Primary-Replica Database Replication

Thu, 22 Oct 2020 17:44:38 +0000

Provide a turnkey Primary-Replica feature for disaster recover. Alternatively provide API such that a "Replication Peer Instance" can: connect to a "primary" database simultaneously connect to a "replica" database (in a different data center) * read transactions from primary sequentially and write those transactions to the replica.

Provide a progress bar/percentage on backup/restore process.

Thu, 22 Oct 2020 17:10:52 +0000

It would be nice to be able to see the percentage complete when the restoring progress is not clear.

Ability to start peer without transactor.

Thu, 22 Oct 2020 16:48:41 +0000

If all transactors are down, no peers can start. Add the ability to start a peer in a read-only mode while continuing to try to connect to the transactor.

Migration tool to migrate a DB from on-prem to Cloud

Thu, 22 Oct 2020 16:34:33 +0000

Provide a path for moving an on-prem DB to the cloud

S3 Storage Option

Thu, 22 Oct 2020 16:05:07 +0000

Useful for storing large amounts of data, which don't necessarily need low latency guarantees, but would want lower storage costs.

Export from console

Thu, 22 Oct 2020 16:01:20 +0000

It would be great if Datomic Console provided a way to export to CSV files

Expose standby/active transactor status in on-prem metrics.

Thu, 22 Oct 2020 16:00:30 +0000

Provide a method through an API to allow monitoring of standby/active status of transactors in on-prem.

Support backup to a database URI

Thu, 22 Oct 2020 15:40:41 +0000

Currently backup is only possible from a database URI to a file URI. Support backup from a database URI to another database URI. Maintain the incremental feature present in current backup/restore for this feature.

Readonly peer access

Thu, 22 Oct 2020 15:30:43 +0000

A peer that can read from the backing store, but not transact