Datomic Knowledgebase - Recent questions in Cloud

is there a "un-divert" to undo divert-system

Wed, 15 May 2024 18:23:15 +0000

after using datomic.dev-local/divert, is there a way to undo this in a repl to switch back to the original endpoint?

import-cloud fails with "No reader function for tag object"

Wed, 14 Feb 2024 18:08:30 +0000

The recent release of version 1.0.277 of datomic-local allowed me to run import-cloud to completion successfully. Unfortunately, the resulting database is unusable. When I try to connect, I get this error message:

{:type java.lang.RuntimeException
:message "No reader function for tag object"
:at [clojure.lang.EdnReader$TaggedReader readTagged "EdnReader.java" 801]}]
:trace
[[clojure.lang.EdnReader$TaggedReader readTagged "EdnReader.java" 801]
[clojure.lang.EdnReader$TaggedReader invoke "EdnReader.java" 783]
[clojure.lang.EdnReader$DispatchReader invoke "EdnReader.java" 549]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$VectorReader invoke "EdnReader.java" 672]
[clojure.lang.EdnReader readDelimitedList "EdnReader.java" 757]
[clojure.lang.EdnReader$MapReader invoke "EdnReader.java" 680]
[clojure.lang.EdnReader read "EdnReader.java" 145]
[clojure.lang.EdnReader read "EdnReader.java" 111]
[clojure.lang.EdnReader readString "EdnReader.java" 67]
[clojure.edn$read_string invokeStatic "edn.clj" 46]
[clojure.edn$read_string invokeStatic "edn.clj" 37]
[clojure.edn$read_string invoke "edn.clj" 37]
[datomic.dev_local.btindex_set$read_latest_root invokeStatic "btindex_set.clj" 99]
[datomic.dev_local.btindex_set$read_latest_root invoke "btindex_set.clj" 95]
[datomic.dev_local.btindex_set$current_index_files invokeStatic "btindex_set.clj" 149]
[datomic.dev_local.btindex_set$current_index_files invoke "btindex_set.clj" 146]
[datomic.dev_local.btindex_set$old_index_files invokeStatic "btindex_set.clj" 159]
[datomic.dev_local.btindex_set$old_index_files invoke "btindex_set.clj" 156]
[datomic.dev_local.btindex_set$delete_old_index_files invokeStatic "btindex_set.clj" 176]
[datomic.dev_local.btindex_set$delete_old_index_files invoke "btindex_set.clj" 169]
[datomic.dev_local.btindex_db$delete_unused_files invokeStatic "btindex_db.clj" 604]
[datomic.dev_local.btindex_db$delete_unused_files invoke "btindex_db.clj" 602]
[datomic.dev_local.impl$create_durable_connection invokeStatic "impl.clj" 189]
[datomic.dev_local.impl$create_durable_connection invoke "impl.clj" 187]
[datomic.dev_local.impl$create_connection invokeStatic "impl.clj" 205]
[datomic.dev_local.impl$create_connection invoke "impl.clj" 201]
[datomic.dev_local.impl.DurableClient connect "impl.clj" 271]
[datomic.client.api$connect invokeStatic "api.clj" 151]
[datomic.client.api$connect invoke "api.clj" 138]
[st.system$ensure_db_BANG_ invokeStatic "system.clj" 117]
[st.system$ensure_db_BANG_ invoke "system.clj" 114]
[st.system$eval57307$fn__57309$fn__57311 invoke "system.clj" 122]
[clojure.core$mapv$fn__8535 invoke "core.clj" 6979]
...

Note that I use a custom credentials provider and it appears as an opaque java object in the resulting db-root-00000000 file.

Equivalents for d/with; transaction function test scaffold story when using cloud

Wed, 01 Nov 2023 11:24:08 +0000

Is there absolutely no way to access the peer api on datomic cloud?

My thinking is that, conceptually, since it uses clients, that implies that peer servers do exist.

I realise that light horizontal scaling is the point in DC, but the peer api, and in particular d/with I'm finding important to allow test scaffolds for transaction functions.

So the thought here is it would be nice to be able to access the richer peer api just for one off tasks, such as managing schema or even here making speculative databases so one can create a test scaffold to test a more complex transaction function.

I don't know what the test scaffold story is with DC or if there is one. Bit of a dark area on the map for me at the moment so any suggestions would be appreciated.

I guess one last-resort option is to develop using Pro and test transaction functions that way. Then copy them over to DC. I have found though that in a previous iteration trying to think about Pro and Cloud at the same time as designing for your domain is something I would want to avoid where possible.

Update: I've just realised that the client api does actually include d/with, but I'll leave the question up anyway, in case anyone has any related thoughts they'd like to share.

Further Update: looking back at my initial inept attempts to build a datomic system and schema which utilised transaction functions to avoid race conditions given a complex content hierarchy in the domain I am modelling, it looks like the reason I felt forced to use the Peer API and not Client, for those Database Functions, was the absence of d/function in the Client API. I wonder if Client API actually is fully capable in terms of installing and testing database functions, but that wasn't clear to me at the time. So much is required to be able to develop in Datomic. It might be that my missing a little detail of Clojure/Java design (an easy syntactic alternative to d/function?) ended up pushing my design decision in an unnecessary direction.
I suppose the answer may be, re. DB functions - "Client API yes", "Cloud no"?
(Illustrating the point that it is best not to try to think about Pro and Cloud at the same time...)

Remaining question: I'm finding it very difficult to find out what the difference is between d/function on the peer API, vs. use of the #db/fn literal. Cannot locate an explanation in the docs. GPT is telling me the latter is for an inline database function which will not be installed, whereas d/function is for installing the fn into the database, but it I know it could be hallucinating in such a case.

Why do values of :db.type/uri return as com.cognitect.transit.impl.URIImpl?

Mon, 02 Oct 2023 03:56:45 +0000

I've recently worked with dev-local try out Datomic and when I moved to Cloud I noticed the following scenario, whenever I transact a fact that is of the java.net.URI type and try to query it back I get com.cognitect.transit.impl.URIImpl back as the type, I have found no documentation about this, I was expecting that the types remain the same, is there any way to disable this behavior or at least convert back to java.net.URI?

This matters because I had some code break because it was expecting java.net.URI and because this behavior is undocumented (or if documented inaccessible by searching the Datomic documentation and search engines)

Thank you for your time

deps.edn

org.clojure/clojure {:mvn/version "1.11.1"}
com.datomic/client-cloud {:mvn/version "1.0.123"}

Is `(pull $db2 ?e [*])` supported officially?

Sat, 15 Jul 2023 04:42:45 +0000

The documentation only mentions the syntax without a src-var:

pull-expr = ['pull' variable pattern]

but found that this syntax also works:

pull-expr = ['pull' src-var variable pattern]

I have a self-contained code example here, which demonstrates a use-case, where an entity is split between 2 Datomic databases, for privacy reasons:

https://forum.datomic.com/t/pull-db-entity-pattern-is-undocumented/2299

Is this variant of the pull-expr in Datalog queries are not supported officially?

Should I use it in production or does it have some undefined behaviour, compared to its binary arity variant?

Or the documentation is just incomplete?

Datomic Cloud IAM role drift on setup - is this normal?

Wed, 12 Jul 2023 09:34:33 +0000

hi, just set up DC for the first time (ran the two AWS cloudformation templates). The storage stack is showing drift on the IAM managed policy datomic-admin-..., shows 'MODIFIED', in red. Is this normal upon setup? (checking the changes in the 'versions' tab, the difference is addition of autoscaling and s3 actions to the policy; the second version, presumably causing the 'drift', was added during the provisioning). Is this just a setup artifact nothing to worry about?

How create more one production compute?

Wed, 05 Jul 2023 13:54:33 +0000

Hello,
I need to use my existing storage with additional compute and tried create new production compute with this system name(storage stack name). And I get creation error:

Export with name myname-VpcLinkId is already exported by stack myname-compute. Rollback requested by user.

I understand that this is due to a name conflict, but I don’t understand how then to correctly create a compute.

I want to run my application in a separate server(сompute) but use the same storage.

Can a tuple value be added when one of the parts of the tuple is an lookup-ref?

Thu, 20 Apr 2023 20:09:32 +0000

I'm wanting to use a lookup ref instead of an entity id (:db-id) when specifying the value for a tuple of attributes where one of the attributes is of type :db.type/ref. Is this possible? I've looked over the Datomic Cloud grammar on transactions and I don't see how to specify a tuple value at all (maybe I'm missing something). I've tried what I thought would be the obvious way to do it and this doesn't work. Here's what I have...

Truncated schema:

{:db/ident       :molecule/legacy-id
 :db/unique      :db.unique/identity
 :db/valueType   :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc         "The Molecule/Complex id from the ProcessDB Oracle database."}
{:db/ident       :molecule-part/number
 :db/valueType   :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc         "The number of molecules for this molecule. The entity with this attribute may be reused and is
 not unique for a particular complex so do not change this number. Instead, check for another existing entity
 or create one."}
{:db/ident       :molecule-part/molecule
 :db/valueType   :db.type/ref
 :db/cardinality :db.cardinality/one
 :db/doc         "The molecule this part refers to. The entity with this attribute may be reused and is
 not unique for a particular complex."}
{:db/ident       :molecule-part/number+molecule
 :db/unique      :db.unique/identity
 :db/valueType   :db.type/tuple
 :db/tupleAttrs  [:molecule-part/number :molecule-part/molecule]
 :db/cardinality :db.cardinality/one
 :db/doc         "The number and molecule that collectively form the unique identity of this entity."}

Then I try to add a molecule part using

(d/transact (db/get-connection) {:tx-data [{:molecule-part/molecule [:molecule/legacy-id 717] :molecule-part/number 1 :molecule-part/number+molecule [1 [:molecule/legacy-id 717]]}]})

And get the following error

:db.error/invalid-tuple-value Invalid tuple value

If I replace "[:molecule/legacy-id 717]" with the entity id (:db/id) for that molecule then it works. But I don't want to use the entity id... I want to use the lookup ref, because what I have is the :molecule/legacy-id. I don't want to have to do a query to resolve this before doing a transaction to add the :molecule-part.

Backup Datomic Cloud to another account

Fri, 31 Mar 2023 13:55:41 +0000

Is there a way to backup a datomic cloud instance to another aws account? I know that cloud is based on HA technology inside the AWS, (dynamo, s3, etc) but backups are important for business continuity. In the event of an attack, everything can be deleted within our aws account with one breach. I can make the tools to copy s3 items, it does not seem possible to import an existing s3 db into datomic.

Request Entity Too Large - clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

Tue, 21 Mar 2023 12:26:20 +0000

There seems to be a bug in Datomic 990-9202 that breaks the d/tx-range function.

Earlier I could use the tx-range function the fetch data from my database.

But after upgrading to Datomic 990-9202 I get and exception when a transaction has too much data.

`
clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

cognitect.anomalies/category: :cognitect.anomalies/incorrect
 cognitect.anomalies/message: "Response body did not conform to Datomic client protocol"
                 http-result: {:status 413,
                               :headers
                               {"apigw-requestid" "Vg4UxisCDoEEJsQ=",
                                "server" "Jetty(9.4.44.v20210927)",
                                "connection" "keep-alive",
                                "content-length" "38",
                                "date" "Tue, 19 Jul 2022 12:17:47 GMT",
                                "content-type" "application/json"},
                               :body "{\"message\":\"Request Entity Too Large\"}"}

How can I connect to Datomic after upgrade to 990-9202, the bastion disappeared?

Fri, 17 Mar 2023 09:42:45 +0000

Hi!

After I upgraded Datomic cloud to version 2023/02/28 990-9202 the bastion disappeared in AWS. So now I can't connect to Datomic anymore.

When I try to connect to it with datomic client access <SYSTEM-NAME>

I get this error:

`
{:clojure.main/message
"Execution error at datomic.tools.ops.aws/get-ip (aws.clj:116).\nGateway not found, make sure gateway is running.\n",
:clojure.main/triage
{:clojure.error/class java.lang.RuntimeException,
:clojure.error/line 116,
:clojure.error/cause
"Gateway not found, make sure gateway is running.",
:clojure.error/symbol datomic.tools.ops.aws/get-ip,
:clojure.error/source "aws.clj",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type java.lang.RuntimeException,

:message "Gateway not found, make sure gateway is running.",
:at [datomic.tools.ops.aws$get_ip invokeStatic "aws.clj" 116]}],

:trace
[[datomic.tools.ops.aws$get_ip invokeStatic "aws.clj" 116]
[datomic.tools.ops.aws$get_ip invoke "aws.clj" 113]
[datomic.tools.ops.ssh$access invokeStatic "ssh.clj" 77]
[datomic.tools.ops.ssh$access invoke "ssh.clj" 56]
[datomic.tools.ops.cli.client$eval13301$fn__13302

invoke
"client.clj"
68]

[clojure.lang.MultiFn invoke "MultiFn.java" 229]
[datomic.tools.ops$_main invokeStatic "ops.clj" 83]
[datomic.tools.ops$_main doInvoke "ops.clj" 76]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.main$main_opt invokeStatic "main.clj" 514]
[clojure.main$main_opt invoke "main.clj" 510]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]],
:cause "Gateway not found, make sure gateway is running."}}
`

What should I do to be able to connect to Datomic again?

Could not find artifact com.datomic:datomic-free:jar:1.0.6610

Thu, 16 Feb 2023 22:51:25 +0000

I am just exploring this for my company, and I am trying to set up free version to follow the tutorial. I downloaded datomic-1.0.6610, am on Windows with leiningen. This is in my project.clj file:

  :dependencies [[org.clojure/clojure "1.11.1"]
                 [com.datomic/datomic-free "1.0.6610"]]

When I run lein repl, I get this:

C:\Users\irinzn\personal\datomic-try>lein repl
Retrieving org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Retrieving org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Retrieving org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Retrieving nrepl/nrepl/1.0.0/nrepl-1.0.0.pom from clojars
Retrieving org/nrepl/incomplete/0.1.0/incomplete-0.1.0.pom from clojars
Retrieving org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Retrieving org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Retrieving org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Retrieving org/nrepl/incomplete/0.1.0/incomplete-0.1.0.jar from clojars
Retrieving nrepl/nrepl/1.0.0/nrepl-1.0.0.jar from clojars
Could not find artifact com.datomic:datomic-free:jar:1.0.6610 in central (https://repo1.maven.org/maven2/)
Could not find artifact com.datomic:datomic-free:jar:1.0.6610 in clojars (https://repo.clojars.org/)
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
Could not resolve dependencies

And then it hangs. I am following the instructions here: https://docs.datomic.com/on-prem/peer/integrating-peer-lib.html . Can someone tell me what I am doing wrong?

Also, peeking at those repos, this version may not be there, at least in free version?

https://repo.clojars.org/datomic-free/datomic-free/
https://repo1.maven.org/maven2/com/datomic/

Also, is this possibly the answer? https://ask.datomic.com/index.php/453/datomic-free-releases-arent-uploaded-public-clojars-anymore

d/with crashes when called rapidly in dev-local

Mon, 19 Dec 2022 18:11:18 +0000

com.datomic/dev-local {:mvn/version "1.0.243"}
datomic.client.api

One of our dataflow experiments is thrashing d/with rapidly (for each keystroke in a form connected by websocket) and it crashes. Obviously this is not a reasonable circumstance but wanted to report as a possible concurrency bug.

I'm not able to repro minimally yet. Confounding variables include concurrency model, whether the d/with calls are parallel or composed, whether the d/with calls are interrupting each other or concurrent, how Datomic responds to ThreadInterruptedException, and likely more.

d/with, tx: ([:db/add 536561674378709 :label/name "6"]) ...success

d/with, tx: ([:db/add 536561674378709 :label/name "65"]) ...failure, e:  
#error {
 :cause nil
 :via
 [{:type java.nio.channels.ClosedChannelException
   :message nil
   :at [sun.nio.ch.FileChannelImpl ensureOpen FileChannelImpl.java 159]}]
 :trace
 [[sun.nio.ch.FileChannelImpl ensureOpen FileChannelImpl.java 159]
  [sun.nio.ch.FileChannelImpl read FileChannelImpl.java 814]
  [datomic.dev_local.io$read_n_bbuf invokeStatic io.clj 66]
  [datomic.dev_local.io$read_n_bbuf invoke io.clj 63]
  [datomic.dev_local.io$read_n_buf invokeStatic io.clj 72]
  [datomic.dev_local.io$read_n_buf invoke io.clj 69]
  [datomic.dev_local.tx$keyoff_datom_bbuf invokeStatic tx.clj 374]
  [datomic.dev_local.tx$keyoff_datom_bbuf invokePrim tx.clj -1]
  [datomic.dev_local.tx$datom_lookup_valfn invokeStatic tx.clj 401]
  [datomic.dev_local.tx$datom_lookup_valfn invoke tx.clj 397]
  [datomic.dev_local.local_log.LocalLog valAt local_log.clj 56]
  [datomic.dev_local.btindex_db$double_lookup$reify__19853 valAt btindex_db.clj 404]
  [datomic.dev_local.btindex$lookup_cmp$reify__19649 valAt btindex.clj 268]
  [datomic.dev_local.btindex.BTIndexLeaf leftk btindex.clj 238]
  [datomic.dev_local.btindex$leftk invokeStatic btindex.clj 59]
  [datomic.dev_local.btindex$leftk invoke btindex.clj 58]
  [datomic.dev_local.btindex.BTIndexBranch bspos btindex.clj 165]
  [datomic.dev_local.btindex.BTIndexBranch seek btindex.clj 184]
  [datomic.dev_local.btindex.BTIndexBranch seek btindex.clj 185]
  [datomic.dev_local.btindex.BTIndex seek btindex.clj 310]
  [datomic.core.btset$seek invokeStatic btset.clj 405]
  [datomic.core.btset$seek invoke btset.clj 400]
  [datomic.core.db.Db seekEAVT db.clj 2195]
  [datomic.core.db$find_eavt invokeStatic db.clj 1135]
  [datomic.core.db$find_eavt invoke db.clj 1129]
  [datomic.core.db$find_eavt invokeStatic db.clj 1132]
  [datomic.core.db$find_eavt invoke db.clj 1129]
  [datomic.core.db$next_valid_inst invokeStatic db.clj 3041]
  [datomic.core.db$next_valid_inst invoke db.clj 3037]
  [datomic.core.db.ProcessExpander getData db.clj 3291]
  [datomic.dev_local.btindex_db$expand_tx invokeStatic btindex_db.clj 445]
  [datomic.dev_local.btindex_db$expand_tx invoke btindex_db.clj 429]
  [datomic.dev_local.with_db.WithDb with with_db.clj 73]
  [datomic.client.api$with invokeStatic api.clj 363]
  [datomic.client.api$with invoke api.clj 353]
  [clojure.core$eval39795$fn__39796$fn__39797 invoke NO_SOURCE_FILE 0]
  [clojure.core$eval39787$fn__39788$fn__39789$fn__39790 invoke NO_SOURCE_FILE 0]
  [clojure.lang.Atom swap Atom.java 37]
  [clojure.core$swap_BANG_ invokeStatic core.clj 2369]
  [clojure.core$swap_BANG_ invoke core.clj 2362]
  [clojure.core$eval39787$fn__39788$fn__39789 invoke NO_SOURCE_FILE 0]
  [hyperfiddle.photon$wrap_STAR_$cr31022_block_0__31024$cr31022_place_2__31025 invoke photon.cljc 154]
  [missionary.impl.Thunk run Thunk.java 58]
  [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1136]
  [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 635]
  [java.lang.Thread run Thread.java 833]]}

some test code that does NOT repro the bug:

  (def tx1 [[:db/add 536561674378709 :label/name "6"]])
  (def tx2 [[:db/add 536561674378709 :label/name "65"]])
  
  (def db0 (d/with-db test/datomic-conn)) ; mbrainz-samples
  
  (do
    (:db-after (d/with db0 {:tx-data tx1}))
    (:db-after (d/with db0 {:tx-data tx2})))

  (let [db1 (:db-after (d/with db0 {:tx-data tx1}))
        db2 (:db-after (d/with db1 {:tx-data tx2}))]
    db2)

What's the state of the art in distributed Datomic?

Sun, 18 Dec 2022 00:37:33 +0000

I'd like to get an idea of the state of the art in distributed Datomic. FWIW, my wish list includes the ability for selected processing nodes to have local images and/or participate in federated global images, linked by CRDTs, etc. And a pony.

-r

Querying 2000+ entities, by passing in uuids (Really slow)

Wed, 14 Dec 2022 09:24:29 +0000

Passing in over 2000 uuids to mysql for a fairly simple query takes milliseconds to retrieve relevant data. However we're running into an issue with datomic where it takes noticeable longer. Is there a specific syntax to use when processing such long lists, or is the linked example the only way?

https://docs.datomic.com/on-prem/query/query.html#collection-binding

Inconsistency between d/pull and d/q + pull for non-existent entities

Sat, 10 Dec 2022 15:55:29 +0000

Inconsistency between d/pull and d/q + pull for non-existent entities

Datomic returns a map with the :db/id of the entity being pulled, even if the entity doesn't exist in the database. For example:

(d/pull (d/db conn) '[*] 9999999999)
=> #:db{:id 9999999999}

This behavior is somewhat confusing, especially when using d/pull with time filters. I was expecting d/pull to return nil or #:db{:id nil} if the entity wasn't created at the given point in time. However, Datomic always returns a map with the entity ID:

(d/pull (as-of (d/db conn) 1) '[*] 9999999999)
=> #:db{:id 9999999999}

Surprisingly, I see the expected behavior if I pull the entity information from a query using d/q:

(d/q '{:find [(pull ?e [*])]
        :in   [?e]}
      (d/db conn) 9999999999)
=> [[#:db{:id nil}]]

Is that difference in behavior expected? How can I determine if an entity existed at any point in time using d/pull?

How do you add relationship properties, e.g. the time a person met another person?

Mon, 07 Nov 2022 08:35:28 +0000

I have some experience with Neo4j / cypher. In cypher relationships are first-class citizens, so they can have properties, which I really like.

How do I do something similar with datalog? Do I have to have a "mapping entity" in between two entities that have a relationship?

Is there a way to create permanent aliases for reverse relationships?

Wed, 31 Aug 2022 14:45:19 +0000

Datomic documentation recommends modeling relationships in one direction only. So, if I have a reference "whole" on an entity "part", I don't need to declare another reference "parts" on an entity "whole". The following attribute declaration is enough.

{:db/ident       :part/whole
 :db/valueType   :db.type/ref
 :db/cardinality :db.cardinality/one}

Then, I can use :part/_whole as a key to fetch all parts of a "whole". Even though it works, it's counterintuitive at times.

Is there a way to create a permanent alias for the reverse relationship, equivalent to the pull pattern :as option?

(d/pull db '[* {[:part/_whole :as :whole/parts] [*]}] whole-db-id)

Notifying clients of database changes in Datomic Cloud?

Sat, 04 Jun 2022 15:18:15 +0000

In your 2013 blog post The Transaction Report Queue, you write:

In Datomic, you can monitor all transactions. Any peer process in the
system can request a transaction report queue of every transaction
against a particular database.

Is there a change data capture or related mechanism in Datomic Cloud, where changes to the database can be pushed to a client?

Thank you,

Andrew

How to retract entities that reference a retracted entity

Thu, 06 Jan 2022 09:52:00 +0000

In my database model, I have several entities referencing the user entity.
When I retract the user entity using :db/retractEntity, the user is correctly retracted, as well as its refs in the other entities, for example :message/user.
For some entities, when the user is retracted the entity itself should also be retracted, but since the entity is not a component of the user, as the relation is the other way around, I did not find a way to retract the entire entity.
Is there a way to make this happen for some entities?

Why does :db/ensure appear as an attribute in my entities?

Tue, 09 Nov 2021 16:52:46 +0000

Hi there,

I wonder if you might be able to help me with something?

I am trying to use an entity spec with datomic to provide constraint checking when entities are transacted/retracted from the database using :db/ensure. This works fine - if my predicate fails the entities are not transacted into Datomic. But in the case where the predicate passes the :db/ensure key/value also appears to be stored in the database - it's visible when I pull the entity out again. This isn't what I expected to happen according to the docs:

https://docs.datomic.com/cloud/schema/schema-reference.html#entity-specs

":db/ensure is a virtual attribute. It is not added in the database; instead it triggers checks based on the named entity."

Have I done something wrong causing :db/ensure to be added to the database? Or have I misunderstood the docs?

I've got a simple test case showing this behaviour if it helps see what I mean. You can find that in the Clojurians Slack channel where I originally asked this question: https://clojurians.slack.com/archives/C03RZMDSH/p1635858493193300

If anyone can point me in the right direction that would be much appreciated, cheers!

What is the recommended way of dumping the whole datomic schema?

Mon, 01 Nov 2021 18:22:07 +0000

The recommended answer from here is to use

[:find ?attr ?type ?card
 :where
 [_ :db.install/attribute ?a]
 [?a :db/valueType ?t]
 [?a :db/cardinality ?c]
 [?a :db/ident ?attr]
 [?t :db/ident ?type]
 [?c :db/ident ?card]]

I'm curious what folks are using themselves. Also, what about db/index ?

Broken Web Link

Wed, 20 Oct 2021 07:17:30 +0000

In https://docs.datomic.com/cloud/time/filters.html - the "example inventory database" link to https://github.com/cognitect-labs/day-of-datomic-cloud/blob/master/tutorial/filters.clj is broken/404

Datom "value" - content?

Sun, 26 Sep 2021 08:29:14 +0000

What “kind” of content can the Datom value have?

Beyond a simple string like “pizza”, can the value be an entity id?

In the product docs there’s also “:green” as the value, what is that? Sorry, I’m not even sure what the colon means…

noHistory and transaction data

Wed, 25 Aug 2021 16:55:02 +0000

We've got an attribute that we write to fairly often, but don't need history for. While :db:noHistory is clearly the write solution for not logging the history of the data itself, I'm worried that we're also going to accrue a large number of transaction entities as well.

Will that happen? Are transactions still made when the the only datom is for a noHistory attribute? Will we still have database growth issues due to the transactions themselves?

How to authenticate and authorize access to API Gateway endpoints for datomic?

Thu, 19 Aug 2021 17:05:29 +0000

I'm currently evaluating Datomic and I've deployed a production topology using a split stack. DatomicCloudVersion is 9095 and DatomicCFTVersion is 884. The CloudFormation stack created two API Gateway endpoints: datomic-[system]-client-api and datomic-[system]-ions.

From my local computer I can get a client by using datomic-[system]-client-api endpoint:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

Above, https://[endpoint-id].execute-api.us-east-1.amazonaws.com/ is the client-api API Gateway endpoint. It works, but the endpoint is open to the world. I then added an IAM authorizer to datomic-[system]-client-api and used an authorized IAM user to make a request to the now authenticated endpoint:

awscurl --service execute-api -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com --profile datomic
{:s3-auth-path "long s3 path"}

awscurl is just a tool to make authenticated calls to AWS services. As you can see, I can get a response from the endpoint by using my local profile named "datomic". If I try using regular curl without the v4 AWS signature it fails:

curl -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com
{"message":"Forbidden"}

Works as expected. The problem is now I can't get a client using:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :creds-profile "datomic"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

The response I get is:

{:status 403,
:headers
{"apigw-requestid" "EUsyNi9aIAMEPlg=",
"connection" "keep-alive",
"content-length" "23",
"date" "Thu, 19 Aug 2021 17:01:40 GMT",
"content-type" "application/json"},
:body "{\"message\":\"Forbidden\"}"}

It is as if the d/client call is ignoring the profile or not generating the proper signature. I tried setting the AWS_PROFILE environment variable as well, to no avail.

Is using an IAM Authorizer not the proper way to go? How can I protect both the client-api and ions endpoints? Finally, what's the difference between the client-api and ions endpoints?

Highly available deployments when desired capacity is 1

Thu, 19 Aug 2021 12:53:18 +0000

Hi! With the new Datomic Cloud, I set a compute group's auto scaling configuration this way:

Cluster node instance type = t3.small
Desired capacity = Default (1)
Minimum instances = Default (1)
Maximum instances = Default (3)
Minimum number of instances during update = 2

Due to the last config bullet (5), should the system spin up a 2nd instance when the update starts, before deploying the new app code, in order to enable high availability during deployments, and scale back to the desired capacity after the deployment concludes?

I tested it and the system/ASG didn't scale a 2nd instance up during the deployment, so it caused downtime.

My understanding is that bullets 2-3-4 control the ASG, while bullet 5 would control something in CodeDeploy, but it's not clear to me how CodeDeploy and the ASG would coordinate to honor my setting config (5) to 2.

Are my expectations wrong about setting it up like I did?

Is there an easy way to migrate data from Datomic to DocumentDB (AWS MongDB version) ?

Mon, 26 Jul 2021 14:35:14 +0000

We use Datomic in our company, but we decided to migrate data from Datomic to DocumentDB. Is there any software tools or library to perform that in an easy way ?

Datomic returns empty results where "Insufficient bindings, will cause db scan" exception is expected

Wed, 07 Jul 2021 14:30:05 +0000

Hello

If we change #uuid"..." to "" (from uuid to string), we get the exception (desired behavior)

clj -Sdeps '{:deps {com.datomic/dev-local {:mvn/version "0.9.232"}}}' -M \
  -e "(require '[datomic.client.api :as d])" \
  -e '(def client (d/client {:server-type :dev-local :system "hello"}))' \
  -e '(d/create-database client {:db-name "hello"})' \
  -e '(def conn (d/connect client {:db-name "hello"}))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
=> #'user/client
=> true
=> #'user/conn
=> []

Occour on (old) versions of datomic-free too

clj -Sdeps '{:deps {com.datomic/datomic-free {:mvn/version "0.9.5697"}}}' \
  -M --report stderr \
  -e "(require '[datomic.api :as d])" \
  -e '(d/create-database "datomic:mem://hello")' \
  -e '(def conn (d/connect "datomic:mem://hello"))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
WARNING: requiring-resolve already refers to: #'clojure.core/requiring-resolve in namespace: datomic.common, being replaced by: #'datomic.common/requiring-resolve
=> true
=> #'user/conn
=> #{}

Data model for simple shopping app

Tue, 06 Jul 2021 06:57:58 +0000

Hi,

I've got a question about data modeling with Datomic.

I am learning Datomic by moving the storage from a small side project I built (https://github.com/rollacaster/shopping-cards) from .edn files to Datomic.

I translated the current data model 1:1 and got the following result (I used https://github.com/hodur-org/hodur-graphviz-schema to create this schema :

https://github.com/rollacaster/shopping-cards/blob/datomic/shopping-cards-schema.png

Is this is a "proper" Datomic data model or does Datomic has any capabilities to simplify this model (e.g. removing the CookedWith Entity)?

Query values for more than one entity in a single datomic query (Roam Alpha Api)

Sat, 26 Jun 2021 19:47:35 +0000

I'd like to in a single datomic/datalog query get output relating to multiple entities

E.g.

In my db (querying Roam graph via RoamAlphaApi) for one value ("YjpbFUsTx") I can do this:

[:find ?e ?stri :where [?e :block/uid "YjpbFUsTx"][?e :block/string ?stri]]

gives me

But I want to pass on two values "YjpbFUsTx" and "TgpgOssBM"

to give me

How do I do that?

I've tried several variations but none give any output.

Useful reference: https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html .

ClassCastException, d/with inside transact function

Thu, 24 Jun 2021 14:57:51 +0000

Calling a transaction function that uses d/with in a dev-local, filesystem (i.e., not :storage-dir :mem) throws a ClassCastException on internal Datomic code.

Environment

com.datomic/dev-local {:mvn/version "0.9.232"}
client-cloud 0.8.105

History

Example with exception using file system dev-local.

(def client (d/client {:server-type :dev-local
                         :storage-dir "/tmp/cce-bug3"
                         :system      "test"}))
=> #'user/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'user/conn
(d/transact conn {:tx-data [{:db/ident       :user/name
                               :db/valueType   :db.type/string
                               :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "test",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "test",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T00:49:46.400-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(defn my-tx-fn
    [db argm]
    (let [tx-report (d/with db {:tx-data [{:user/name "asd"}]})]
      []))
=> #'user/my-tx-fn
(d/transact conn {:tx-data [(list 'user/my-tx-fn {})]})
Execution error (ClassCastException) at datomic.dev-local.tx/datom-lookup-valfn (tx.clj:397).
class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')

Working memdb example.

(def client-memdb (d/client {:server-type :dev-local
                               :storage-dir :mem
                               :system      "test"}))
=> #'user/client-memdb
(d/create-database client-memdb {:db-name "test"})
=> true
(def conn-memdb (d/connect client-memdb {:db-name "test"}))
=> #'user/conn-memdb
(d/transact conn-memdb {:tx-data [{:db/ident       :user/name
                                     :db/valueType   :db.type/string
                                     :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:02:57.965-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(d/transact conn-memdb {:tx-data [(list 'user/my-tx-fn {})]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 6,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 7,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533319 50 #inst"2021-05-09T01:03:08.042-00:00" 13194139533319 true]],
 :tempids {}}

Expectation

Transaction functions that use d/with do not throw a ClassCastException. Additionally, the behavior of d/with in all Datomic environments (Cloud, File system, Memdb) is consistent. Given a previous Datomic Cloud release fixed a problem with using with-db from an Ion, it seems using d/with in a transaction function is a supported behavior.

Actual

The final d/transact transaction function throws a ClassCastException when using d/with on some internal Datomic code. This only occurs when using a dev-local, file system environment.

Full stacktrace is pasted below.

#error{:cause "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
       :via [{:type java.lang.ClassCastException,
              :message "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
              :at [datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]}],
       :trace [[datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]
               [datomic.dev_local.tx$datom_lookup_valfn invoke "tx.clj" 397]
               [datomic.dev_local.local_log.LocalLog valAt "local_log.clj" 56]
               [clojure.lang.RT get "RT.java" 760]
               [datomic.dev_local.btindex.BTIndex cons "btindex.clj" 281]
               [clojure.lang.RT conj "RT.java" 677]
               [clojure.core$conj__5390 invokeStatic "core.clj" 85]
               [clojure.core$conj__5390 invoke "core.clj" 82]
               [datomic.core.db.Db addData "db.clj" 2322]
               [datomic.core.db$add_ensured_data invokeStatic "db.clj" 3353]
               [datomic.core.db$add_ensured_data invoke "db.clj" 3351]
               [datomic.core.db$with_tx invokeStatic "db.clj" 3370]
               [datomic.core.db$with_tx invoke "db.clj" 3357]
               [datomic.core.db.Db with "db.clj" 2164]
               [datomic.core.local_db$fn__25633 invokeStatic "local_db.clj" 67]
               [datomic.core.local_db$fn__25633 invoke "local_db.clj" 24]
               [datomic.client.api.protocols$fn__11959$G__11877__11966 invoke "protocols.clj" 126]
               [datomic.client.api$with invokeStatic "api.clj" 363]
               [datomic.client.api$with invoke "api.clj" 353]
               [user$my_tx_fn invokeStatic "user.clj" 3]
               [user$my_tx_fn invoke "user.clj" 20]
               [clojure.lang.AFn applyToHelper "AFn.java" 156]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.lang.Var applyTo "Var.java" 705]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$apply invoke "core.clj" 660]
               [datomic.core.db.ProcessExpander inject "db.clj" 3229]
               [datomic.core.db.ProcessInpoint inject "db.clj" 2950]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586$fn__18587 invoke "btindex_db.clj" 440]
               [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
               [clojure.core$reduce invokeStatic "core.clj" 6827]
               [clojure.core$reduce invoke "core.clj" 6810]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586 invoke "btindex_db.clj" 440]
               [datomic.dev_local.btindex_db$expand_tx invokeStatic "btindex_db.clj" 444]
               [datomic.dev_local.btindex_db$expand_tx invoke "btindex_db.clj" 429]
               [datomic.dev_local.btindex_db$with_tx invokeStatic "btindex_db.clj" 468]
               [datomic.dev_local.btindex_db$with_tx invoke "btindex_db.clj" 461]
               [datomic.dev_local.impl.DurableConnection transact "impl.clj" 161]
               [datomic.client.api$transact invokeStatic "api.clj" 200]
               [datomic.client.api$transact invoke "api.clj" 183]
               [user$eval124260 invokeStatic "user.clj" 25]
               [user$eval124260 invoke "user.clj" 25]
               [clojure.lang.Compiler eval "Compiler.java" 7177]
               [clojure.lang.Compiler eval "Compiler.java" 7132]
               [clojure.core$eval invokeStatic "core.clj" 3214]
               [clojure.core$eval invoke "core.clj" 3210]
               [nrepl.middleware.interruptible_eval$evaluate$fn__959 invoke "interruptible_eval.clj" 91]
               [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
               [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
               [clojure.main$repl$fn__9095 invoke "main.clj" 458]
               [clojure.main$repl invokeStatic "main.clj" 458]
               [clojure.main$repl doInvoke "main.clj" 368]
               [clojure.lang.RestFn invoke "RestFn.java" 1523]
               [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
               [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
               [nrepl.middleware.interruptible_eval$interruptible_eval$fn__985$fn__989
                invoke
                "interruptible_eval.clj"
                155]
               [clojure.lang.AFn run "AFn.java" 22]
               [nrepl.middleware.session$session_exec$main_loop__1086$fn__1090 invoke "session.clj" 190]
               [nrepl.middleware.session$session_exec$main_loop__1086 invoke "session.clj" 189]
               [clojure.lang.AFn run "AFn.java" 22]
               [java.lang.Thread run "Thread.java" 829]]}

Evidence

The previously pasted example is reproducible and always throws a ClassCastException.

Impact

We are unable to test functionality in a consistent manner. Local setups using dev-local file systems will not function as expected. Developers are unable to test features in the same way they will work in production.

d/client external server communication

Thu, 24 Jun 2021 14:55:59 +0000

Environment

com.datomic/client-cloud 0.8.105

History

Get the docstring for d/client.

(clojure.repl/doc d/client)
-------------------------
datomic.client.api/client
([arg-map])
  Create a client for a Datomic system. This function does not
communicate with a server and returns immediately.

For a cloud system, arg-map requires:

  :server-type   - :cloud
  :region        - AWS region, e.g. "us-east-1"
  :system        - your system name
  :endpoint      - IP address of your system or query group

Optionally, a cloud system arg-map accepts:

  :creds-provider  - instance of com.amazonaws.auth.AWSCredentialsProvider. Defaults to DefaultAWSCredentialsProviderChain
  :creds-profile   - name of an AWS Named Profile. See http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
  :proxy-port      - local port for SSH tunnel to bastion 

  Note: :creds-provider and :creds-profile are mutually exclusive, providing both will result in an error.

For a dev-local system, arg-map comprises:

  :server-type   - :dev-local (required)
  :system        - a system name (required)
  :storage-dir   - optional, overrides :storage-dir in ~/.datomic/dev-local.edn

dev-local stores databases under ${storage-dir}/${system}/${db-name}.

For a peer-server system, arg-map requires:

  :server-type   - :peer-server
  :access-key    - access-key from peer server launch
  :secret        - secret from peer server launch
  :endpoint      - peer server host:port
  :validate-hostnames  - false

Returns a client object.

Note the second sentence in the docstring: "This function does not communicate with a server and returns immediately." Next call d/client with an endpoint that does not exist.

(d/client {:server-type :ion
             :system      "my-system"
             :endpoint    "http://i-dont-exist.datomic.net:8182"
             :region      "us-west-2"})
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to http://i-dont-exist.datomic.net:8182

Expectation

Calling d/client does not communicate with a server.

Actual

Calling d/client does communicate with a server.

Evidence

The above reproducible example indicates the d/client function call is attempted to communicate the the passing in :endpoint string given the exception thrown.

Impact

The d/client docstring misleads the user by saying no server communication will occur when it does occur. The result is calls to d/client can fail in unexpected ways when an external server communication is not accounted for. For example, we spin up Datomic Cloud query groups dynamically and attempt to communicate with the newly spun up cluster. Until the cluster is up and healthy, we expect calls to fail at d/connect with an unavailable anomaly, and instead the calls would fail at d/client with an unavailable anomaly.

Failure to transact a DB after delete and re-create

Tue, 22 Jun 2021 20:32:19 +0000

Hi, when we perform these steps:

delete-database
create-database
transact schema

...the transact fails with a cognitect.anomaly/forbidden citing a 403.

{:msg "YADA ! yada.components.user.handlers CreateUserTx:STEP MSG", :outcome " FAILURE: Datomic Client Exception", :lvl "⚠️ WARN", :loc "yada/components/user/handlers.clj:134"}
; Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
; Datomic Client Exception
clj꞉yada.core꞉> 
*e
#error {
 :cause "Datomic Client Exception"
 :data {:cognitect.anomalies/category :cognitect.anomalies/forbidden, :http-result {:status 403, :headers {"server" "Jetty(9.4.36.v20210114)", "content-length" "19", "date" "Wed, 23 Jun 2021 17:38:56 GMT", "content-type" "application/transit+msgpack"}, :body nil}}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Datomic Client Exception"
   :data {:cognitect.anomalies/category :cognitect.anomalies/forbidden, :http-result {:status 403, :headers {"server" "Jetty(9.4.36.v20210114)", "content-length" "19", "date" "Wed, 23 Jun 2021 17:38:56 GMT", "content-type" "application/transit+msgpack"}, :body nil}}
   :at [datomic.client.api.async$ares invokeStatic "async.clj" 58]}]
 :trace
 [[datomic.client.api.async$ares invokeStatic "async.clj" 58]
  [datomic.client.api.async$ares invoke "async.clj" 54]
  [datomic.client.api.sync$eval37636$fn__37641 invoke "sync.clj" 104]
  [datomic.client.api.protocols$fn__11841$G__11795__11848 invoke "protocols.clj" 72]
  [datomic.client.api$transact invokeStatic "api.clj" 200]
  [datomic.client.api$transact invoke "api.clj" 183]
  [datomic.cloud.client.local.Connection transact "local.clj" 97]
  [datomic.client.api$transact invokeStatic "api.clj" 200]
  [datomic.client.api$transact invoke "api.clj" 183]
  [yada.util.datomic$transact$fn__7133 invoke "datomic.clj" 428]
  [yada.util.anom$with_retry$fn__7039 invoke "anom.clj" 25]
  [yada.util.anom$with_retry invokeStatic "anom.clj" 25]
  [yada.util.anom$with_retry doInvoke "anom.clj" 13]
  [clojure.lang.RestFn invoke "RestFn.java" 410]
  [yada.util.datomic$transact invokeStatic "datomic.clj" 427]
  [yada.util.datomic$transact invoke "datomic.clj" 407]
  [yada.components.user.handlers$create_user_tx invokeStatic "handlers.clj" 142]
  [yada.components.user.handlers$create_user_tx invoke "handlers.clj" 131]
  [clojure.core.reducers$map$fn__26594$fn__26595 invoke "reducers.clj" 134]
  [clojure.lang.ArrayChunk reduce "ArrayChunk.java" 58]
  [clojure.core.protocols$fn__8178 invokeStatic "protocols.clj" 136]
  [clojure.core.protocols$fn__8178 invoke "protocols.clj" 124]
  [clojure.core.protocols$fn__8138$G__8133__8147 invoke "protocols.clj" 19]
  [clojure.core.protocols$seq_reduce invokeStatic "protocols.clj" 31]
  [clojure.core.protocols$fn__8174 invokeStatic "protocols.clj" 75]
  [clojure.core.protocols$fn__8174 invoke "protocols.clj" 75]
  [clojure.core.protocols$fn__8112$G__8107__8125 invoke "protocols.clj" 13]
  [clojure.core.reducers$folder$reify__26569 coll_reduce "reducers.clj" 93]
  [clojure.core$reduce invokeStatic "core.clj" 6833]
  [clojure.core$into invokeStatic "core.clj" 6900]
  [clojure.core$into invoke "core.clj" 6892]
  [yada.db.seed.app_seed.app_data.user$seed invokeStatic "user.clj" 56]
  [yada.db.seed.app_seed.app_data.user$seed invoke "user.clj" 52]
  [yada.db.seed.app_seed.core$seed invokeStatic "core.clj" 12]
  [yada.db.seed.app_seed.core$seed invoke "core.clj" 9]
  [yada.core$eval38738 invokeStatic "NO_SOURCE_FILE" 51]
  [yada.core$eval38738 invoke "NO_SOURCE_FILE" 51]
  [clojure.lang.Compiler eval "Compiler.java" 7181]
  [clojure.lang.Compiler eval "Compiler.java" 7136]
  [clojure.core$eval invokeStatic "core.clj" 3202]
  [clojure.core$eval invoke "core.clj" 3198]
  [nrepl.middleware.interruptible_eval$evaluate$fn__36286$fn__36287 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1977]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1977]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [nrepl.middleware.interruptible_eval$evaluate$fn__36286 invoke "interruptible_eval.clj" 87]
  [clojure.main$repl$read_eval_print__9112$fn__9115 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9112 invoke "main.clj" 437]
  [clojure.main$repl$fn__9121 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl doInvoke "main.clj" 368]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__36317$fn__36321 invoke "interruptible_eval.clj" 152]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__36385$fn__36389 invoke "session.clj" 202]
  [nrepl.middleware.session$session_exec$main_loop__36385 invoke "session.clj" 201]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 748]]}

If we instead:

delete-database
redeploy app using Ion's tools for this
create-database
transact schema

...then all goes fine.

So it seems that cycling the Datomic server helps somehow.

We looked to see if we were caching the client or the connection and this doesn't seem to be the case. Is there some operational characteristics we should be aware of that would be in play after deleting a DB?

Add query group ASG metrics to query group CW dashboard

Thu, 27 May 2021 17:22:27 +0000

When looking at the query group dashboard, it is helpful to know what "state" the ASG was in when referencing the various dashboard metrics. Currently, I must do one of the following: 1) reference these metrics in the ASG monitoring tab (e.g., https://us-west-2.console.aws.amazon.com/ec2autoscaling/home?region=us-west-2#/details/my-query-group?view=monitoring), 2) create a CW dashboard of my own, 3) modify the existing Datomic CW dashboard to add this chart. The ASG metrics answer questions like: 1) how many instances was my query group running? 2) was my query group trying to scale out or in? 3) why did my query group not scale out or in?

I would find the following ASG metrics (in order of importance) useful additions to the current query group dashboard.

In Service Instances (Count)
Minimum Group Size (Count), Maximum Group Size (Count), and Desired Capacity (Count)
Pending Instances (Count)

DDB size reduction after restore

Tue, 25 May 2021 09:19:36 +0000

Hello. We have a large datomic database (15B datoms, 20TB DDB size). We have done a backup and a restore as a test and the resulting DDB is 500GB in size. Is this expected? A couple of datom count queries match on both databases, but the reduction from 20TB to 500GB still look a bit scary

How might multiple systems serve the same database in different contexts?

Mon, 24 May 2021 15:16:46 +0000

From the guide on naming there is the following quote:

Multiple systems might serve the same primary database in different development stages. So you might name systems via the convention "[db]-[stage]", e.g."inventory-dev", "inventory-staging" and "inventory-prod".

I'm trying to understand how this would work, and how I should think about my system, application, and database name, but I'm not really sure how this would work.

My understanding was that you have one System per Storage stack, plus some number of Compute stacks, and that databases are "owned" by the Storage stack, but that doesn't make sense with the quote above.

Can anyone give some examples of how this would work? I've read the planning guide but almost certainly don't fully understand it.

Thank you!
Evan

What is a performant way to do "and" matching on attribute values?

Fri, 21 May 2021 18:04:41 +0000

So I can use

[?entity1 ?attrname1 ?attrval]
[?entity2 ?attrname2 ?attrval]

in a :where clause to get ?entity1 and ?entity2 where there exists an ?attrval that matches

I'm using

  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity1 ?attrname1) [[?attrvals1]]]
  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity2 ?attrname2) [[?attrvals2]]]
  (not-join [?attrvals1 ?attrvals2]
               [(seq ?attrvals1) [?element ...]]
               (not [(contains? ?attrvals2 ?element)]))

to get ?entity1 and ?entity2 where all ?attrvals for ?entity1 exist for ?entity2. Is there a more performant way to do this??
(This feels like a directional "and" to the implicit "or" being applied to each attrval matching in the first case)

AssertionError shows up as an Event?

Tue, 18 May 2021 15:33:51 +0000

This surprised me. In a lambda invocation, a failed assert shows up as an Event. I would have expected Alert:

"Msg": "IonLambdaException",
"Ex": {
    "Via": [
        {
            "Type": "java.lang.AssertionError",
            "Message": "Assert failed: (= (count s) 16)",
        }
    ],
    "Cause": "Assert failed: (= (count s) 16)"
},
"Type": "Event",
"Tid": 338,
"Timestamp": 162134781093

Use Launch Template instead of Launch Configuration

Fri, 07 May 2021 23:38:51 +0000

The compute Datomic CloudFormation templates (primary and query groups) create ASGs using launch configurations. Launch configurations do not support the full set of Auto Scaling group features. For example, you cannot create an Auto Scaling group that launches both Spot and On-Demand Instances or that specifies multiple instance types or multiple launch templates. In fact, AWS recommends creating ASGs using launch templates, not launch configurations.

We are interested in launch templates for its support for using Spot and On-Demand instances.

EDIT: As of 2021-10-20, launch templates will not receive any new EC2 ASG features.

Support for recent AWS instance types

Wed, 24 Mar 2021 18:36:03 +0000

AWS has released several new instance types that are much better price per performance. It'd be great to have the Datomic query group template updated to support these instance types. The Datomic query group template currently supports the following instance types: t3.medium, m5.large, i3.large, i3.xlarge. Based on this, I propose the following additions:

t3a.medium (~10% cheaper than t3.medium)
m5a.large (~10% cheaper than m5.large)

If Datomic can run on ARM based processors then further additions can be made with even better price per performance (the new AWS Graviton2 processors are fantastic. Read about Honeycomb's success).

t4g.medium (perhaps also large and xlarge, ~20% cheaper than t3.medium)
m6g.large (~20% cheaper than m5.large)
r6gd.large (has NVMe based SSD, 26% cheaper than i3.large)
I4i (released 2022-04-28)

The Datomic Solo instance type could also be configurable for some inconsequential savings. See monthly prices below:

(Current) t3.small primary + t3.nano bastion: $18.90
t3a.small + t3a.nano: $17.15
t4g.small + t4g.nano: $15.33

An example Datomic query group monthly prices. Take a query group with 2 nodes running for the full month.

(Current) m5.large: $140.16
m5a.large: $125.56 ($14.6/month savings)
m6g.large: $112.42 ($27.74/month savings)

Make DesiredCapacity an optional parameter in the query group CF template

Wed, 24 Mar 2021 17:24:25 +0000

Currently (March 2021) the Datomic query group CF template has the DesiredCapacity parameter marked as required. Because of this, during a CF template update, CF will always set the DesiredCapacity to the CF parameter for DesiredCapacity. This behavior does not work well in the following situation.

In the query group template I have the DesiredCapacity at 2, and MaxCount at 4. I have an ASG scaling policy to target track 50% average CPU utilization. My query group is currently at the max, because the ASG scaling policy scaled up the group DesiredCapacity to 4 (to match my load). My load has increased to the point where 4 nodes is no longer sufficient. I must update the query group CF template to increase the MacCount. The problem is that when you update the MacCount from 4 to 6 via a CF template update and leave the DesiredCapacity at 2, CF will actually set the DesiredCapacity on the ASG to 2 during the high load situation. This worsens the high load situation.

One option is to query the ASG for its current DesiredCapacity prior to running a CF update and set the CF parameter to the value you retrieved. This approach complicates the update procedure. It also has a "read-modify-write" race condition -- the DesiredCapacity could have changed from the time you read the DesiredCapacity to the time you are performing the CF update.

A second option is to make the DesiredCapacity parameter optional. As per the CF docs, DesiredCapacity is not a required parameter. When it is not present, it defaults to the minimum size of the group. Leaving DesiredCapacity unset will make it so during a CF update, CF will not modify the DesiredCapacity at all -- leaving the ASG "in control."

I am now seeing Iterator+offset is not available doing a an index-pull

Mon, 08 Mar 2021 16:46:35 +0000

I am doing an index-pull in Datomic Cloud. The request was timing out so I addedd :timeout 300000 and not "Iterator+offset is not available" exception starts popping up

We are on 781-9041

Stack creation fails in AWS Control-Tower enrolled account

Fri, 26 Feb 2021 13:24:53 +0000

We tried creating a Solo stack in a new AWS account provisioned through our AWS Control Tower. This would be our second Solo stack, but the only one in this account. Not being used to everything AWS, I didn't think of creating myself an admin IAM user and launched the Cloud Formation directly from my admin SSO account. Here's what happened while creating the Storage's MountTargets:

CREATE_FAILED: The IAM identity making this call has an IAM policy that is too large. Reduce...

I then realized what was happening and started afresh with a new IAM admin user. But here's what happens, again while creating the Storage:

CREATE_FAILED: Embedded stack arn:aws:cloudformation:us-east-1:<ACCOUNT>:stack/yada-demo-Storage<...> was not successfully created: The following resource(s) failed to create: [DatomicCmk, CatalogTable, FileSystem, LogGroup, LogTable].

I suspect the creation failed because of leftovers that the Cloud Formation wasn't able to remove to replace them.

More precisely, I suspect what's happening is that some of the Control Tower's Guardrails impede the stack (re)creation:

I'm currently cleaning out the leftovers and will try again one more time after that.

If anyone knows anything about this, please share some help! I'll report back with my findings too.

Are Ions the only way to develop applications with Datomic Cloud?

Wed, 17 Feb 2021 17:30:21 +0000

As I'm reading through the docs and getting a feel for how one interacts with Datomic Cloud, am I correct in thinking that app development happens with Ions, and there isn't a different method of deploying application code?

String length limit cloud vs dev-local

Tue, 05 Jan 2021 13:21:18 +0000

The page https://docs.datomic.com/cloud/schema/schema-reference.html states that:
- Strings are limited to 4096 characters.

And the dev-local page https://docs.datomic.com/cloud/dev-local.html states:
- dev-local limits strings to 4096 characters.

Is the actual cloud environment string length longer than 4k chars? And what is the real limit.

Dev-local query does not support :timeout

Wed, 30 Dec 2020 21:25:39 +0000

Dev-local 0.9.229 does not appear to support :timeout when querying. Here is a repro.

(def client (d/client {:server-type :dev-local
                       :system      "example"}))
(d/create-database client {:db-name "example"})
(def conn (d/connect client {:db-name "example"}))
(d/transact conn {:tx-data [{:db/ident       :string
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string}]})
(dotimes [n 1000]
  (d/transact conn {:tx-data (map (fn [i]
                                    {:string (str n i)}) (range 1000))}))
(first
  (d/q {:query   '[:find (pull ?e [*])
                   :where
                   [?e :string]]
        :timeout 1000
        :args    [(d/db conn)]}))

The query takes longer than 1s and does not throw. I would have expected an exception thrown.

Allow dev-local import-cloud :source and :dest to work with any :server-type

Sun, 20 Dec 2020 00:37:33 +0000

Currently import-cloud requires the :source client :server-type to be :cloud and the :dest :server-type to be :dev-local. It'd be great if this function allowed any :server-type to be passed for :source and :dest. A possible use case would be copying a dev-local database to a cloud system for testing in a staging environment.

Why is my float attribute returning double values in queries?

Thu, 17 Dec 2020 04:25:11 +0000

I have an attribute of type :db.type/float in my Datomic Cloud (solo) instance. When I query for that value, however, I get values of type java.lang.Double. (I actually have 6 float attributes and they all behave this way.)

Is this a bug? I can't see anything in the documentation that describes this behavior. In fact, the schema reference seems to specifically say that it should be float value and not a double.

This is not a big problem, but it is an annoyance because the values I'm transacting as floats are getting extra (noisy) precision added to them, so that e.g. when the user enters a 3.7, it actually shows up as 3.700000047683716. This isn't difficult to fix by casting to float, but it was surprising.

I only see this behavior in Datomic Cloud, by the way. Using dev-local correctly shows the value as a float.

Here's some relevant exchanges from a REPL session demonstrating the issue:

user> (d/q '[:find ?attr-type ?value :where
             [71587003061240196 :student/gpa-weighted-9 ?value]
             [:student/gpa-weighted-9 :db/valueType ?t]
             [?t :db/ident ?attr-type]]
           (d/db conn))
[[:db.type/float 3.700000047683716]]
user> (type (-> *1 first second))
java.lang.Double
user> (float 3.700000047683716)
3.7

Can you use d/with inside a Cloud transaction function?

Tue, 15 Dec 2020 16:58:24 +0000

Can you use d/with inside a Cloud transaction function? It's not clear from the the docs if the db passed to the tx fn is one created by d/with-db.