Datomic Knowledgebase - Questions without answers in Cloud

Is `(pull $db2 ?e [*])` supported officially?

Sat, 15 Jul 2023 04:42:45 +0000

The documentation only mentions the syntax without a src-var:

pull-expr = ['pull' variable pattern]

but found that this syntax also works:

pull-expr = ['pull' src-var variable pattern]

I have a self-contained code example here, which demonstrates a use-case, where an entity is split between 2 Datomic databases, for privacy reasons:

https://forum.datomic.com/t/pull-db-entity-pattern-is-undocumented/2299

Is this variant of the pull-expr in Datalog queries are not supported officially?

Should I use it in production or does it have some undefined behaviour, compared to its binary arity variant?

Or the documentation is just incomplete?

Datomic Cloud IAM role drift on setup - is this normal?

Wed, 12 Jul 2023 09:34:33 +0000

hi, just set up DC for the first time (ran the two AWS cloudformation templates). The storage stack is showing drift on the IAM managed policy datomic-admin-..., shows 'MODIFIED', in red. Is this normal upon setup? (checking the changes in the 'versions' tab, the difference is addition of autoscaling and s3 actions to the policy; the second version, presumably causing the 'drift', was added during the provisioning). Is this just a setup artifact nothing to worry about?

Request Entity Too Large - clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

Tue, 21 Mar 2023 12:26:20 +0000

There seems to be a bug in Datomic 990-9202 that breaks the d/tx-range function.

Earlier I could use the tx-range function the fetch data from my database.

But after upgrading to Datomic 990-9202 I get and exception when a transaction has too much data.

`
clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

cognitect.anomalies/category: :cognitect.anomalies/incorrect
 cognitect.anomalies/message: "Response body did not conform to Datomic client protocol"
                 http-result: {:status 413,
                               :headers
                               {"apigw-requestid" "Vg4UxisCDoEEJsQ=",
                                "server" "Jetty(9.4.44.v20210927)",
                                "connection" "keep-alive",
                                "content-length" "38",
                                "date" "Tue, 19 Jul 2022 12:17:47 GMT",
                                "content-type" "application/json"},
                               :body "{\"message\":\"Request Entity Too Large\"}"}

What's the state of the art in distributed Datomic?

Sun, 18 Dec 2022 00:37:33 +0000

I'd like to get an idea of the state of the art in distributed Datomic. FWIW, my wish list includes the ability for selected processing nodes to have local images and/or participate in federated global images, linked by CRDTs, etc. And a pony.

-r

Querying 2000+ entities, by passing in uuids (Really slow)

Wed, 14 Dec 2022 09:24:29 +0000

Passing in over 2000 uuids to mysql for a fairly simple query takes milliseconds to retrieve relevant data. However we're running into an issue with datomic where it takes noticeable longer. Is there a specific syntax to use when processing such long lists, or is the linked example the only way?

https://docs.datomic.com/on-prem/query/query.html#collection-binding

Inconsistency between d/pull and d/q + pull for non-existent entities

Sat, 10 Dec 2022 15:55:29 +0000

Inconsistency between d/pull and d/q + pull for non-existent entities

Datomic returns a map with the :db/id of the entity being pulled, even if the entity doesn't exist in the database. For example:

(d/pull (d/db conn) '[*] 9999999999)
=> #:db{:id 9999999999}

This behavior is somewhat confusing, especially when using d/pull with time filters. I was expecting d/pull to return nil or #:db{:id nil} if the entity wasn't created at the given point in time. However, Datomic always returns a map with the entity ID:

(d/pull (as-of (d/db conn) 1) '[*] 9999999999)
=> #:db{:id 9999999999}

Surprisingly, I see the expected behavior if I pull the entity information from a query using d/q:

(d/q '{:find [(pull ?e [*])]
        :in   [?e]}
      (d/db conn) 9999999999)
=> [[#:db{:id nil}]]

Is that difference in behavior expected? How can I determine if an entity existed at any point in time using d/pull?

How do you add relationship properties, e.g. the time a person met another person?

Mon, 07 Nov 2022 08:35:28 +0000

I have some experience with Neo4j / cypher. In cypher relationships are first-class citizens, so they can have properties, which I really like.

How do I do something similar with datalog? Do I have to have a "mapping entity" in between two entities that have a relationship?

Notifying clients of database changes in Datomic Cloud?

Sat, 04 Jun 2022 15:18:15 +0000

In your 2013 blog post The Transaction Report Queue, you write:

In Datomic, you can monitor all transactions. Any peer process in the
system can request a transaction report queue of every transaction
against a particular database.

Is there a change data capture or related mechanism in Datomic Cloud, where changes to the database can be pushed to a client?

Thank you,

Andrew

How to authenticate and authorize access to API Gateway endpoints for datomic?

Thu, 19 Aug 2021 17:05:29 +0000

I'm currently evaluating Datomic and I've deployed a production topology using a split stack. DatomicCloudVersion is 9095 and DatomicCFTVersion is 884. The CloudFormation stack created two API Gateway endpoints: datomic-[system]-client-api and datomic-[system]-ions.

From my local computer I can get a client by using datomic-[system]-client-api endpoint:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

Above, https://[endpoint-id].execute-api.us-east-1.amazonaws.com/ is the client-api API Gateway endpoint. It works, but the endpoint is open to the world. I then added an IAM authorizer to datomic-[system]-client-api and used an authorized IAM user to make a request to the now authenticated endpoint:

awscurl --service execute-api -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com --profile datomic
{:s3-auth-path "long s3 path"}

awscurl is just a tool to make authenticated calls to AWS services. As you can see, I can get a response from the endpoint by using my local profile named "datomic". If I try using regular curl without the v4 AWS signature it fails:

curl -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com
{"message":"Forbidden"}

Works as expected. The problem is now I can't get a client using:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :creds-profile "datomic"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

The response I get is:

{:status 403,
:headers
{"apigw-requestid" "EUsyNi9aIAMEPlg=",
"connection" "keep-alive",
"content-length" "23",
"date" "Thu, 19 Aug 2021 17:01:40 GMT",
"content-type" "application/json"},
:body "{\"message\":\"Forbidden\"}"}

It is as if the d/client call is ignoring the profile or not generating the proper signature. I tried setting the AWS_PROFILE environment variable as well, to no avail.

Is using an IAM Authorizer not the proper way to go? How can I protect both the client-api and ions endpoints? Finally, what's the difference between the client-api and ions endpoints?

Datomic returns empty results where "Insufficient bindings, will cause db scan" exception is expected

Wed, 07 Jul 2021 14:30:05 +0000

Hello

If we change #uuid"..." to "" (from uuid to string), we get the exception (desired behavior)

clj -Sdeps '{:deps {com.datomic/dev-local {:mvn/version "0.9.232"}}}' -M \
  -e "(require '[datomic.client.api :as d])" \
  -e '(def client (d/client {:server-type :dev-local :system "hello"}))' \
  -e '(d/create-database client {:db-name "hello"})' \
  -e '(def conn (d/connect client {:db-name "hello"}))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
=> #'user/client
=> true
=> #'user/conn
=> []

Occour on (old) versions of datomic-free too

clj -Sdeps '{:deps {com.datomic/datomic-free {:mvn/version "0.9.5697"}}}' \
  -M --report stderr \
  -e "(require '[datomic.api :as d])" \
  -e '(d/create-database "datomic:mem://hello")' \
  -e '(def conn (d/connect "datomic:mem://hello"))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
WARNING: requiring-resolve already refers to: #'clojure.core/requiring-resolve in namespace: datomic.common, being replaced by: #'datomic.common/requiring-resolve
=> true
=> #'user/conn
=> #{}

ClassCastException, d/with inside transact function

Thu, 24 Jun 2021 14:57:51 +0000

Calling a transaction function that uses d/with in a dev-local, filesystem (i.e., not :storage-dir :mem) throws a ClassCastException on internal Datomic code.

Environment

com.datomic/dev-local {:mvn/version "0.9.232"}
client-cloud 0.8.105

History

Example with exception using file system dev-local.

(def client (d/client {:server-type :dev-local
                         :storage-dir "/tmp/cce-bug3"
                         :system      "test"}))
=> #'user/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'user/conn
(d/transact conn {:tx-data [{:db/ident       :user/name
                               :db/valueType   :db.type/string
                               :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "test",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "test",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T00:49:46.400-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(defn my-tx-fn
    [db argm]
    (let [tx-report (d/with db {:tx-data [{:user/name "asd"}]})]
      []))
=> #'user/my-tx-fn
(d/transact conn {:tx-data [(list 'user/my-tx-fn {})]})
Execution error (ClassCastException) at datomic.dev-local.tx/datom-lookup-valfn (tx.clj:397).
class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')

Working memdb example.

(def client-memdb (d/client {:server-type :dev-local
                               :storage-dir :mem
                               :system      "test"}))
=> #'user/client-memdb
(d/create-database client-memdb {:db-name "test"})
=> true
(def conn-memdb (d/connect client-memdb {:db-name "test"}))
=> #'user/conn-memdb
(d/transact conn-memdb {:tx-data [{:db/ident       :user/name
                                     :db/valueType   :db.type/string
                                     :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:02:57.965-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(d/transact conn-memdb {:tx-data [(list 'user/my-tx-fn {})]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 6,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 7,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533319 50 #inst"2021-05-09T01:03:08.042-00:00" 13194139533319 true]],
 :tempids {}}

Expectation

Transaction functions that use d/with do not throw a ClassCastException. Additionally, the behavior of d/with in all Datomic environments (Cloud, File system, Memdb) is consistent. Given a previous Datomic Cloud release fixed a problem with using with-db from an Ion, it seems using d/with in a transaction function is a supported behavior.

Actual

The final d/transact transaction function throws a ClassCastException when using d/with on some internal Datomic code. This only occurs when using a dev-local, file system environment.

Full stacktrace is pasted below.

#error{:cause "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
       :via [{:type java.lang.ClassCastException,
              :message "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
              :at [datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]}],
       :trace [[datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]
               [datomic.dev_local.tx$datom_lookup_valfn invoke "tx.clj" 397]
               [datomic.dev_local.local_log.LocalLog valAt "local_log.clj" 56]
               [clojure.lang.RT get "RT.java" 760]
               [datomic.dev_local.btindex.BTIndex cons "btindex.clj" 281]
               [clojure.lang.RT conj "RT.java" 677]
               [clojure.core$conj__5390 invokeStatic "core.clj" 85]
               [clojure.core$conj__5390 invoke "core.clj" 82]
               [datomic.core.db.Db addData "db.clj" 2322]
               [datomic.core.db$add_ensured_data invokeStatic "db.clj" 3353]
               [datomic.core.db$add_ensured_data invoke "db.clj" 3351]
               [datomic.core.db$with_tx invokeStatic "db.clj" 3370]
               [datomic.core.db$with_tx invoke "db.clj" 3357]
               [datomic.core.db.Db with "db.clj" 2164]
               [datomic.core.local_db$fn__25633 invokeStatic "local_db.clj" 67]
               [datomic.core.local_db$fn__25633 invoke "local_db.clj" 24]
               [datomic.client.api.protocols$fn__11959$G__11877__11966 invoke "protocols.clj" 126]
               [datomic.client.api$with invokeStatic "api.clj" 363]
               [datomic.client.api$with invoke "api.clj" 353]
               [user$my_tx_fn invokeStatic "user.clj" 3]
               [user$my_tx_fn invoke "user.clj" 20]
               [clojure.lang.AFn applyToHelper "AFn.java" 156]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.lang.Var applyTo "Var.java" 705]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$apply invoke "core.clj" 660]
               [datomic.core.db.ProcessExpander inject "db.clj" 3229]
               [datomic.core.db.ProcessInpoint inject "db.clj" 2950]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586$fn__18587 invoke "btindex_db.clj" 440]
               [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
               [clojure.core$reduce invokeStatic "core.clj" 6827]
               [clojure.core$reduce invoke "core.clj" 6810]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586 invoke "btindex_db.clj" 440]
               [datomic.dev_local.btindex_db$expand_tx invokeStatic "btindex_db.clj" 444]
               [datomic.dev_local.btindex_db$expand_tx invoke "btindex_db.clj" 429]
               [datomic.dev_local.btindex_db$with_tx invokeStatic "btindex_db.clj" 468]
               [datomic.dev_local.btindex_db$with_tx invoke "btindex_db.clj" 461]
               [datomic.dev_local.impl.DurableConnection transact "impl.clj" 161]
               [datomic.client.api$transact invokeStatic "api.clj" 200]
               [datomic.client.api$transact invoke "api.clj" 183]
               [user$eval124260 invokeStatic "user.clj" 25]
               [user$eval124260 invoke "user.clj" 25]
               [clojure.lang.Compiler eval "Compiler.java" 7177]
               [clojure.lang.Compiler eval "Compiler.java" 7132]
               [clojure.core$eval invokeStatic "core.clj" 3214]
               [clojure.core$eval invoke "core.clj" 3210]
               [nrepl.middleware.interruptible_eval$evaluate$fn__959 invoke "interruptible_eval.clj" 91]
               [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
               [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
               [clojure.main$repl$fn__9095 invoke "main.clj" 458]
               [clojure.main$repl invokeStatic "main.clj" 458]
               [clojure.main$repl doInvoke "main.clj" 368]
               [clojure.lang.RestFn invoke "RestFn.java" 1523]
               [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
               [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
               [nrepl.middleware.interruptible_eval$interruptible_eval$fn__985$fn__989
                invoke
                "interruptible_eval.clj"
                155]
               [clojure.lang.AFn run "AFn.java" 22]
               [nrepl.middleware.session$session_exec$main_loop__1086$fn__1090 invoke "session.clj" 190]
               [nrepl.middleware.session$session_exec$main_loop__1086 invoke "session.clj" 189]
               [clojure.lang.AFn run "AFn.java" 22]
               [java.lang.Thread run "Thread.java" 829]]}

Evidence

The previously pasted example is reproducible and always throws a ClassCastException.

Impact

We are unable to test functionality in a consistent manner. Local setups using dev-local file systems will not function as expected. Developers are unable to test features in the same way they will work in production.

d/client external server communication

Thu, 24 Jun 2021 14:55:59 +0000

Environment

com.datomic/client-cloud 0.8.105

History

Get the docstring for d/client.

(clojure.repl/doc d/client)
-------------------------
datomic.client.api/client
([arg-map])
  Create a client for a Datomic system. This function does not
communicate with a server and returns immediately.

For a cloud system, arg-map requires:

  :server-type   - :cloud
  :region        - AWS region, e.g. "us-east-1"
  :system        - your system name
  :endpoint      - IP address of your system or query group

Optionally, a cloud system arg-map accepts:

  :creds-provider  - instance of com.amazonaws.auth.AWSCredentialsProvider. Defaults to DefaultAWSCredentialsProviderChain
  :creds-profile   - name of an AWS Named Profile. See http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
  :proxy-port      - local port for SSH tunnel to bastion 

  Note: :creds-provider and :creds-profile are mutually exclusive, providing both will result in an error.

For a dev-local system, arg-map comprises:

  :server-type   - :dev-local (required)
  :system        - a system name (required)
  :storage-dir   - optional, overrides :storage-dir in ~/.datomic/dev-local.edn

dev-local stores databases under ${storage-dir}/${system}/${db-name}.

For a peer-server system, arg-map requires:

  :server-type   - :peer-server
  :access-key    - access-key from peer server launch
  :secret        - secret from peer server launch
  :endpoint      - peer server host:port
  :validate-hostnames  - false

Returns a client object.

Note the second sentence in the docstring: "This function does not communicate with a server and returns immediately." Next call d/client with an endpoint that does not exist.

(d/client {:server-type :ion
             :system      "my-system"
             :endpoint    "http://i-dont-exist.datomic.net:8182"
             :region      "us-west-2"})
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to http://i-dont-exist.datomic.net:8182

Expectation

Calling d/client does not communicate with a server.

Actual

Calling d/client does communicate with a server.

Evidence

The above reproducible example indicates the d/client function call is attempted to communicate the the passing in :endpoint string given the exception thrown.

Impact

The d/client docstring misleads the user by saying no server communication will occur when it does occur. The result is calls to d/client can fail in unexpected ways when an external server communication is not accounted for. For example, we spin up Datomic Cloud query groups dynamically and attempt to communicate with the newly spun up cluster. Until the cluster is up and healthy, we expect calls to fail at d/connect with an unavailable anomaly, and instead the calls would fail at d/client with an unavailable anomaly.

Add query group ASG metrics to query group CW dashboard

Thu, 27 May 2021 17:22:27 +0000

When looking at the query group dashboard, it is helpful to know what "state" the ASG was in when referencing the various dashboard metrics. Currently, I must do one of the following: 1) reference these metrics in the ASG monitoring tab (e.g., https://us-west-2.console.aws.amazon.com/ec2autoscaling/home?region=us-west-2#/details/my-query-group?view=monitoring), 2) create a CW dashboard of my own, 3) modify the existing Datomic CW dashboard to add this chart. The ASG metrics answer questions like: 1) how many instances was my query group running? 2) was my query group trying to scale out or in? 3) why did my query group not scale out or in?

I would find the following ASG metrics (in order of importance) useful additions to the current query group dashboard.

In Service Instances (Count)
Minimum Group Size (Count), Maximum Group Size (Count), and Desired Capacity (Count)
Pending Instances (Count)

How might multiple systems serve the same database in different contexts?

Mon, 24 May 2021 15:16:46 +0000

From the guide on naming there is the following quote:

Multiple systems might serve the same primary database in different development stages. So you might name systems via the convention "[db]-[stage]", e.g."inventory-dev", "inventory-staging" and "inventory-prod".

I'm trying to understand how this would work, and how I should think about my system, application, and database name, but I'm not really sure how this would work.

My understanding was that you have one System per Storage stack, plus some number of Compute stacks, and that databases are "owned" by the Storage stack, but that doesn't make sense with the quote above.

Can anyone give some examples of how this would work? I've read the planning guide but almost certainly don't fully understand it.

Thank you!
Evan

What is a performant way to do "and" matching on attribute values?

Fri, 21 May 2021 18:04:41 +0000

So I can use

[?entity1 ?attrname1 ?attrval]
[?entity2 ?attrname2 ?attrval]

in a :where clause to get ?entity1 and ?entity2 where there exists an ?attrval that matches

I'm using

  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity1 ?attrname1) [[?attrvals1]]]
  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity2 ?attrname2) [[?attrvals2]]]
  (not-join [?attrvals1 ?attrvals2]
               [(seq ?attrvals1) [?element ...]]
               (not [(contains? ?attrvals2 ?element)]))

to get ?entity1 and ?entity2 where all ?attrvals for ?entity1 exist for ?entity2. Is there a more performant way to do this??
(This feels like a directional "and" to the implicit "or" being applied to each attrval matching in the first case)

AssertionError shows up as an Event?

Tue, 18 May 2021 15:33:51 +0000

This surprised me. In a lambda invocation, a failed assert shows up as an Event. I would have expected Alert:

"Msg": "IonLambdaException",
"Ex": {
    "Via": [
        {
            "Type": "java.lang.AssertionError",
            "Message": "Assert failed: (= (count s) 16)",
        }
    ],
    "Cause": "Assert failed: (= (count s) 16)"
},
"Type": "Event",
"Tid": 338,
"Timestamp": 162134781093

Use Launch Template instead of Launch Configuration

Fri, 07 May 2021 23:38:51 +0000

The compute Datomic CloudFormation templates (primary and query groups) create ASGs using launch configurations. Launch configurations do not support the full set of Auto Scaling group features. For example, you cannot create an Auto Scaling group that launches both Spot and On-Demand Instances or that specifies multiple instance types or multiple launch templates. In fact, AWS recommends creating ASGs using launch templates, not launch configurations.

We are interested in launch templates for its support for using Spot and On-Demand instances.

EDIT: As of 2021-10-20, launch templates will not receive any new EC2 ASG features.

Support for recent AWS instance types

Wed, 24 Mar 2021 18:36:03 +0000

AWS has released several new instance types that are much better price per performance. It'd be great to have the Datomic query group template updated to support these instance types. The Datomic query group template currently supports the following instance types: t3.medium, m5.large, i3.large, i3.xlarge. Based on this, I propose the following additions:

t3a.medium (~10% cheaper than t3.medium)
m5a.large (~10% cheaper than m5.large)

If Datomic can run on ARM based processors then further additions can be made with even better price per performance (the new AWS Graviton2 processors are fantastic. Read about Honeycomb's success).

t4g.medium (perhaps also large and xlarge, ~20% cheaper than t3.medium)
m6g.large (~20% cheaper than m5.large)
r6gd.large (has NVMe based SSD, 26% cheaper than i3.large)
I4i (released 2022-04-28)

The Datomic Solo instance type could also be configurable for some inconsequential savings. See monthly prices below:

(Current) t3.small primary + t3.nano bastion: $18.90
t3a.small + t3a.nano: $17.15
t4g.small + t4g.nano: $15.33

An example Datomic query group monthly prices. Take a query group with 2 nodes running for the full month.

(Current) m5.large: $140.16
m5a.large: $125.56 ($14.6/month savings)
m6g.large: $112.42 ($27.74/month savings)

Make DesiredCapacity an optional parameter in the query group CF template

Wed, 24 Mar 2021 17:24:25 +0000

Currently (March 2021) the Datomic query group CF template has the DesiredCapacity parameter marked as required. Because of this, during a CF template update, CF will always set the DesiredCapacity to the CF parameter for DesiredCapacity. This behavior does not work well in the following situation.

In the query group template I have the DesiredCapacity at 2, and MaxCount at 4. I have an ASG scaling policy to target track 50% average CPU utilization. My query group is currently at the max, because the ASG scaling policy scaled up the group DesiredCapacity to 4 (to match my load). My load has increased to the point where 4 nodes is no longer sufficient. I must update the query group CF template to increase the MacCount. The problem is that when you update the MacCount from 4 to 6 via a CF template update and leave the DesiredCapacity at 2, CF will actually set the DesiredCapacity on the ASG to 2 during the high load situation. This worsens the high load situation.

One option is to query the ASG for its current DesiredCapacity prior to running a CF update and set the CF parameter to the value you retrieved. This approach complicates the update procedure. It also has a "read-modify-write" race condition -- the DesiredCapacity could have changed from the time you read the DesiredCapacity to the time you are performing the CF update.

A second option is to make the DesiredCapacity parameter optional. As per the CF docs, DesiredCapacity is not a required parameter. When it is not present, it defaults to the minimum size of the group. Leaving DesiredCapacity unset will make it so during a CF update, CF will not modify the DesiredCapacity at all -- leaving the ASG "in control."

I am now seeing Iterator+offset is not available doing a an index-pull

Mon, 08 Mar 2021 16:46:35 +0000

I am doing an index-pull in Datomic Cloud. The request was timing out so I addedd :timeout 300000 and not "Iterator+offset is not available" exception starts popping up

We are on 781-9041

Dev-local query does not support :timeout

Wed, 30 Dec 2020 21:25:39 +0000

Dev-local 0.9.229 does not appear to support :timeout when querying. Here is a repro.

(def client (d/client {:server-type :dev-local
                       :system      "example"}))
(d/create-database client {:db-name "example"})
(def conn (d/connect client {:db-name "example"}))
(d/transact conn {:tx-data [{:db/ident       :string
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string}]})
(dotimes [n 1000]
  (d/transact conn {:tx-data (map (fn [i]
                                    {:string (str n i)}) (range 1000))}))
(first
  (d/q {:query   '[:find (pull ?e [*])
                   :where
                   [?e :string]]
        :timeout 1000
        :args    [(d/db conn)]}))

The query takes longer than 1s and does not throw. I would have expected an exception thrown.

Allow dev-local import-cloud :source and :dest to work with any :server-type

Sun, 20 Dec 2020 00:37:33 +0000

Currently import-cloud requires the :source client :server-type to be :cloud and the :dest :server-type to be :dev-local. It'd be great if this function allowed any :server-type to be passed for :source and :dest. A possible use case would be copying a dev-local database to a cloud system for testing in a staging environment.

What to do about "syncContext completed and\/or Request lifecycle recycled"?

Mon, 16 Nov 2020 22:03:14 +0000

We get into a state where we see a lot of syncContext completed and\/or Request lifecycle recycled

What does it mean?
What can be done about it?

The stack trace is:

"Trace": [
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "state",
                "AsyncContextState.java",
                52
            ],
            [
                "org.eclipse.jetty.server.AsyncContextState",
                "getResponse",
                "AsyncContextState.java",
                118
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invokeStatic",
                "jetty.clj",
                326
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf_STAR_",
                "invoke",
                "jetty.clj",
                323
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invokeStatic",
                "jetty.clj",
                351
            ],
            [
                "cognitect.http_endpoint.jetty$respond_bbuf",
                "invoke",
                "jetty.clj",
                348
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "fail_request",
                "http_endpoint.clj",
                126
            ],
            [
                "cognitect.http_endpoint.Endpoint",
                "respond",
                "http_endpoint.clj",
                147
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127$fn__13129",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "cognitect.http_endpoint.Endpoint$fn__13082$state_machine__8637__auto____13127",
                "invoke",
                "http_endpoint.clj",
                199
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invokeStatic",
                "ioc_macros.clj",
                978
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine",
                "invoke",
                "ioc_macros.clj",
                977
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invokeStatic",
                "ioc_macros.clj",
                982
            ],
            [
                "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped",
                "invoke",
                "ioc_macros.clj",
                980
            ],
            [
                "clojure.core.async.impl.ioc_macros$take_BANG_$fn__5769",
                "invoke",
                "ioc_macros.clj",
                991
            ],
            [
                "clojure.core.async.impl.channels.ManyToManyChannel$fn__589$fn__590",
                "invoke",
                "channels.clj",
                95
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor",
                "runWorker",
                "ThreadPoolExecutor.java",
                1149
            ],
            [
                "java.util.concurrent.ThreadPoolExecutor$Worker",
                "run",
                "ThreadPoolExecutor.java",
                624
            ],
            [
                "clojure.core.async.impl.concurrent$counted_thread_factory$reify__492$fn__493",
                "invoke",
                "concurrent.clj",
                29
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.lang.Thread",
                "run",
                "Thread.java",
                748
            ]
        ]

Provide way to interact with a datomic database before making the system available

Mon, 02 Nov 2020 15:45:42 +0000

Currently in Cloud there is no way to interact with a datomic database before making the system available for external requests.

Potential uses:

deploy the same code everywhere (dev & prod) and it does db-ensure
actions
load data and/or schema prior to the system being live
start components or other system-management tooling
provide (easy) way to integrate startup (e.g. mount/system-component)
provide status and ability to know when pieces are started so you can finish app initialization

Does dev-local/import-cloud handle failure in the middle of an import?

Fri, 30 Oct 2020 16:21:53 +0000

When running an import-cloud, does dev-local gracefully handle import-cloud failing in the middle of the import? i.e., it fails (gets killed or throws) and I can call import-cloud again without worrying about corrupt data, missing data, etc.

Allow for datomic cloud installs into pre-existing VPCs

Sun, 25 Oct 2020 19:29:15 +0000

The default Marketplace/CF config is great for getting started, more greenfield usage, etc. But it doesn't accomodate more complex (but common) situations. For instance, we've just completed a complete overhaul of our AWS infrastructure. The project had kind of shuffled into AWS. Started with one account, with lifecycle environments only segregated via security groups. After accepting that that was a VeryBadThing, went to VPC's per environment, with AWS account segregation (one for production and pre-production, another for all dev lifecycle environments), we've now moved to something a bit more sophisticated with accounts/vpcs per lifecycle environments, dedicated shared accounts/VPCs for ingress, data vault/logging, management, etc with IPSec tunnels between them all, as opposed to VPC peering.

Sorry about the excruciating detail, but just wanted to provide context. So, in this scenario, shoehorning in Datomic Cloud becomes a bit difficult. Especially in conjunction with the current app/system assumptions (bring on those multiple query groups please! :)). For instance, my plan was to give each of my devs their own sandbox Solo instance, this won't scale well (there are 40 of them ;)) of they each need their own VPC, never mind the other issues. And I guess that might be basically a separate but related request, where one could stand up multiple 'systems' in a single VPC

Datomic CLI should support AWS SSO

Sun, 25 Oct 2020 18:24:06 +0000

Datomic CLI should be able to load SSO profile from ~/.aws/config. It should be able to authorize Datomic users from AWS SSO console.

Support smaller compute instances in Datomic Cloud Production

Thu, 22 Oct 2020 16:04:44 +0000

I would love to use Ions (and perhaps Datomic itself a little as well) to run a load-light application. It needs to have HA so Solo is not an option. However Datomic Cloud only runs with i3.large or stronger, for about $90/m (w/o Datomic license) in Ireland times two (since we need at least two instances for HA). There is no way I can persuade the team to go from its two t2.small (for 2$13) to i3.large. I understand the need for i3. to have fast SSD but for application with low load and little data that is not really needed. But it would be awesome to support also some "underpowered" instances for lighter applications and/or apps that are primarily interested in leveraging Ions as a way to deploy and run code, without heavy need for a DB.

Cloud Backup and Recovery

Thu, 22 Oct 2020 15:36:26 +0000

This feature request is discussed at https://forum.datomic.com/t/cloud-backups-recovery/370
The motivation for wanting backups even in the presence of a reliable storage system is guarding against human error and data corruption bugs. Migrating the data to another AWS account might be another use case.