Datomic Knowledgebase - Recent questions and answers in Cloud

Answered: import-cloud fails with "No reader function for tag object"

Fri, 13 Sep 2024 13:37:11 +0000

One hack I tried to get around this bug is to remove the key-value pair that irreversibly encodes the credentials provider in the db-local files. The relevant file appears to be

./datomic/<system>/<db>/db-root-0000000000

This file seems to be a simple EDN file.

But even if I remove the :creds-provider key (which should not be required after the db is imported), I still can't connect to the imported db. The d/connect process gets much further along after a log of Loading....., but still ultimately fails with the same error.

Maybe the creds providers is encoded somewhere else ...

Answered: is there a "un-divert" to undo divert-system

Wed, 05 Jun 2024 11:39:51 +0000

@Phil

No we do not currently have a feature to un-divert and switch back to the original endpoint. I have logged this request with the dev team, but currently you would need to kill the process and start a new connection. You can alternatively call divert again to a new storage-dir location, but you cannot un-divert so that client calls go to the original endpoint.

Cheers,
Jaret

Answered: Equivalents for d/with; transaction function test scaffold story when using cloud

Wed, 01 Nov 2023 16:34:58 +0000

Hi Mike!

The peer api is not usable in Cloud, but after thinking about your situation I have a few points.

The client api has d/with: https://docs.datomic.com/client-api/datomic.client.api.html#var-with
I would recommend doing this kind of testing using Datomic local. https://docs.datomic.com/cloud/datomic-local.html. (added benefit you can actually import your DB and test against it with import-cloud + utilize an in-memory DB locally).

Does that cover your need for testing in Cloud?

All that being said, I have captured your question and use case here for our on-going consideration of apis for Cloud/Pro.

Cheers,
Jaret

Answered: Why do values of :db.type/uri return as com.cognitect.transit.impl.URIImpl?

Tue, 10 Oct 2023 12:04:08 +0000

I've reproduced this problem and we are working on a fix for a future release. Will updated when I have more details.

Cheers,
Jaret

Answered: Why does :db/ensure appear as an attribute in my entities?

Sat, 15 Jul 2023 04:50:52 +0000

I just described the same question in https://forum.datomic.com/t/db-ensure-doesnt-seem-virtual/2296 with my own example.

Based on the long silence, I'm guessing, it's just a mistake, imprecision or ambiguity in the documentation, that the :db/ensure is described as virtual.

It would be really helpful to clarify this in the docs, because people will just keep wasting time on wondering about it.

Is `(pull $db2 ?e [*])` supported officially?

Sat, 15 Jul 2023 04:42:45 +0000

The documentation only mentions the syntax without a src-var:

pull-expr = ['pull' variable pattern]

but found that this syntax also works:

pull-expr = ['pull' src-var variable pattern]

I have a self-contained code example here, which demonstrates a use-case, where an entity is split between 2 Datomic databases, for privacy reasons:

https://forum.datomic.com/t/pull-db-entity-pattern-is-undocumented/2299

Is this variant of the pull-expr in Datalog queries are not supported officially?

Should I use it in production or does it have some undefined behaviour, compared to its binary arity variant?

Or the documentation is just incomplete?

Datomic Cloud IAM role drift on setup - is this normal?

Wed, 12 Jul 2023 09:34:33 +0000

hi, just set up DC for the first time (ran the two AWS cloudformation templates). The storage stack is showing drift on the IAM managed policy datomic-admin-..., shows 'MODIFIED', in red. Is this normal upon setup? (checking the changes in the 'versions' tab, the difference is addition of autoscaling and s3 actions to the policy; the second version, presumably causing the 'drift', was added during the provisioning). Is this just a setup artifact nothing to worry about?

Answered: How create more one production compute?

Wed, 05 Jul 2023 14:35:00 +0000

Hi @Tatiana,

If you are looking to add more compute to a Datomic Cloud system you have a few options:

You can increase the compute size of your compute nodes. https://docs.datomic.com/cloud/operation/growing-your-system.html
You can add more compute nodes via ASG (however, you will not see much benefits to scaling compute this way on a single DB system).
This is probably, what you want... you can launch a query group against the same storage stack. (https://docs.datomic.com/cloud/operation/growing-your-system.html#task-specific-query-groups) with more docs here explaining that query groups are a kind of compute group: https://docs.datomic.com/cloud/whatis/architecture.html#compute-groups

Let me know if you have questions or more information on what specifically you are hoping to scale.

Cheers,
Jaret

Answered: Can a tuple value be added when one of the parts of the tuple is an lookup-ref?

Sun, 30 Apr 2023 15:38:18 +0000

You don't have to specify the composite key's value usually, just its constituent attributes, because those imply the composite key and Datomic will add it to the resulting transaction automatically.

So this should be sufficient:

(d/transact
 (db/get-connection)
 {:tx-data [{:molecule-part/molecule [:molecule/legacy-id 717],
             :molecule-part/number 1}]})

That being said, if you want to update an existing entity, you do have to provide an attribute in a transaction, which is a :db.unique/identity, so you would need the :molecule-part/number+molecule [1 [:molecule/legacy-id 717]] map entry, which doesn't work.

As discussed here:
https://forum.datomic.com/t/troubles-with-upsert-on-composite-tuples/1355/6
you would need to create a custom transaction function, which resolves the lookup refs used in your unique tuple attr and return entity IDs instead.

Answered: Backup Datomic Cloud to another account

Fri, 14 Apr 2023 19:03:29 +0000

Hi J,

There is currently no backup solution for Datomic Cloud. Nor is there an ability to import from s3. This is an area we are very interested in providing a solution for and frankly has been an area the community has desired for a long time. We are actively working on features around this issue.

I want to clarify what Cloud does have:

All data in Datomic Cloud is stored in S3 (as well as other highly durable storages), which provides very high durability guarantees, making secondary data loss prevention (i.e. backup) unnecessary.

However, we understand that in certain circumstances it may be desirable to ‘roll back’ a database. Datomic Cloud does not yet include the ability to ‘revert’ a database to a prior state. Nor does it have the ability to move a database across system or across account. We are working on features in this area and while I cannot provide a timeline for the features I do want to say we understand the need you have expressed.

If you have any questions or would like to discuss operational concerns like this in Cloud I would be happy to have a call and you can reach me and our team at support@datomic.com to setup such a call.

Cheers,
Jaret

Answered: Could not find artifact com.datomic:ion:jar:0.9.48 in central

Tue, 04 Apr 2023 11:40:53 +0000

For those who have stumbled upon this very same issue, here are the permissions you need:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetBucketCORS",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::*/*",
                "arn:aws:s3:::datomic-releases-1fc2183a "
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

Further steps would be to:
1. Create a dedicated Policy in AWS IAM
2. Create a dedicated User in AWS IAM specifically for Clojure CLI (e.g. tools.deps)
3. Grant the User the previously created Policy ("Permissions" tab > "Permission policies")
4. Create an Access key for the User ("Security credentials" tab > "Access keys")
5. Save the generated AWS Access Key ID and AWS Secret Access Key
6. Add an S3 repo server to your local Maven config as per Deps and CLI docs, using datomic-cloud as a server id, so that tools.deps are able to pick them up

All these steps will help you to run clojure -P (with whatever aliases you use in your project, e.g. ion-dev or simply dev) without the above error.

P.S. It also makes sense to grant this permission to all other Administrators or PowerUsers via the same Policy.

Request Entity Too Large - clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

Tue, 21 Mar 2023 12:26:20 +0000

There seems to be a bug in Datomic 990-9202 that breaks the d/tx-range function.

Earlier I could use the tx-range function the fetch data from my database.

But after upgrading to Datomic 990-9202 I get and exception when a transaction has too much data.

`
clojure.lang.ExceptionInfo: Response body did not conform to Datomic client protocol

cognitect.anomalies/category: :cognitect.anomalies/incorrect
 cognitect.anomalies/message: "Response body did not conform to Datomic client protocol"
                 http-result: {:status 413,
                               :headers
                               {"apigw-requestid" "Vg4UxisCDoEEJsQ=",
                                "server" "Jetty(9.4.44.v20210927)",
                                "connection" "keep-alive",
                                "content-length" "38",
                                "date" "Tue, 19 Jul 2022 12:17:47 GMT",
                                "content-type" "application/json"},
                               :body "{\"message\":\"Request Entity Too Large\"}"}

Answered: How can I connect to Datomic after upgrade to 990-9202, the bastion disappeared?

Mon, 20 Mar 2023 14:23:59 +0000

I got it working!

You don't need the the socks proxy anymore with the datomic client since the bastion is replaced with an AWS api gateway.

Go to your AWS console and look at the api gatewats

https://eu-central-1.console.aws.amazon.com/apigateway/main/apis

Click on datomic-YOUR-SYSTEM-NAME-client-api

Use the Invoke URL as the value for the endpoint in your configuration when connecting to datomic.

{:server-type :ion

      :region "eu-central-1" ;; e.g. us-east-1
      :system SYSTEM_NAME
      :endpoint  INVOKE_URL}

Answered: Could not find artifact com.datomic:datomic-free:jar:1.0.6610

Thu, 02 Mar 2023 15:21:54 +0000

Hi Matt,

You're correct in pointing to https://ask.datomic.com/index.php/453/datomic-free-releases-arent-uploaded-public-clojars-anymore.

Datomic 0.9.5697, datomic-free is the last version of Datomic-free uploaded. Dev-local is the successor to datomic-free and is documented at: https://docs.datomic.com/cloud/dev-local.html and you can get the latest dev-local here: https://cognitect.com/dev-tools.

The problems raised in that thread for use cases of dev-local are all captured and being worked on for future releases of Dev local.

Thanks,
Jaret

Answered: d/with crashes when called rapidly in dev-local

Tue, 27 Dec 2022 12:30:15 +0000

Thanks for the report! I've made a ticket for Dev to review after the break.

What's the state of the art in distributed Datomic?

Sun, 18 Dec 2022 00:37:33 +0000

I'd like to get an idea of the state of the art in distributed Datomic. FWIW, my wish list includes the ability for selected processing nodes to have local images and/or participate in federated global images, linked by CRDTs, etc. And a pony.

-r

Querying 2000+ entities, by passing in uuids (Really slow)

Wed, 14 Dec 2022 09:24:29 +0000

Passing in over 2000 uuids to mysql for a fairly simple query takes milliseconds to retrieve relevant data. However we're running into an issue with datomic where it takes noticeable longer. Is there a specific syntax to use when processing such long lists, or is the linked example the only way?

https://docs.datomic.com/on-prem/query/query.html#collection-binding

Inconsistency between d/pull and d/q + pull for non-existent entities

Sat, 10 Dec 2022 15:55:29 +0000

Inconsistency between d/pull and d/q + pull for non-existent entities

Datomic returns a map with the :db/id of the entity being pulled, even if the entity doesn't exist in the database. For example:

(d/pull (d/db conn) '[*] 9999999999)
=> #:db{:id 9999999999}

This behavior is somewhat confusing, especially when using d/pull with time filters. I was expecting d/pull to return nil or #:db{:id nil} if the entity wasn't created at the given point in time. However, Datomic always returns a map with the entity ID:

(d/pull (as-of (d/db conn) 1) '[*] 9999999999)
=> #:db{:id 9999999999}

Surprisingly, I see the expected behavior if I pull the entity information from a query using d/q:

(d/q '{:find [(pull ?e [*])]
        :in   [?e]}
      (d/db conn) 9999999999)
=> [[#:db{:id nil}]]

Is that difference in behavior expected? How can I determine if an entity existed at any point in time using d/pull?

How do you add relationship properties, e.g. the time a person met another person?

Mon, 07 Nov 2022 08:35:28 +0000

I have some experience with Neo4j / cypher. In cypher relationships are first-class citizens, so they can have properties, which I really like.

How do I do something similar with datalog? Do I have to have a "mapping entity" in between two entities that have a relationship?

Answered: Is there a way to create permanent aliases for reverse relationships?

Fri, 02 Sep 2022 14:59:50 +0000

No, we currently do not allow you to make a permanent alias. However, you could write a function to present this alias or are you saying that your example pull pattern does not work for displaying the alias in the results? Finally, this example was counterintuitive to me too, but perhaps that is because to my eyes the relationship between part and whole seems backwards. Is this exactly your schema or an example to try to capture the problem you were trying to solve for?

Notifying clients of database changes in Datomic Cloud?

Sat, 04 Jun 2022 15:18:15 +0000

In your 2013 blog post The Transaction Report Queue, you write:

In Datomic, you can monitor all transactions. Any peer process in the
system can request a transaction report queue of every transaction
against a particular database.

Is there a change data capture or related mechanism in Datomic Cloud, where changes to the database can be pushed to a client?

Thank you,

Andrew

Answered: Is there an easy way to migrate data from Datomic to DocumentDB (AWS MongDB version) ?

Mon, 10 Jan 2022 22:21:29 +0000

I have not seen any specialized tool to migrate from Datomic to MongoDB, no.

As always with MongoDB and similar tools, you have to know quite well exactly which queries and updates you want to perform, and how you want the exported data to be structured. If you know this, then it would be possible to create a batch job that exported the data from Datomic (make sure you use one single same db-reference/version in all the queries to get all data in sync).

I guess the best way would be to export the various documents with a query that has a pull-expression describing all the data "inside" this Document, convert this to MongoDB update data and the convert the data to JSON (this usually has to be made somewhat custom), and then import it into the Document database.

The data models and programming models are very different in Datomic and MongoDB, but I guess you are well aware of this.

Answered: How to retract entities that reference a retracted entity

Mon, 10 Jan 2022 22:10:44 +0000

I can see some different ways to solve this problem:

The first one is to make a special custom retraction function that, given a user entity eid, also looks up the other entities that should be retracted according to some rule. This requires quite a lot of discipline in the programming of the update logic.

The second solution is to add a special component attribute, only used for removing related data, preferably multiarity. When a message that refers a used is created or somehow delivered to that user, you also add a link from the user back to the message. Now, when the user entity is removed, the special component attribute linked form the user to message also removes the users messages.

The third solution is to introduce a certain reference attribute (a very generic name could be :entity/owns) that is not a component attribute, but can be introspected with a special retract-entity function (on the peer side), that looks up all the links of this special attribute and see whether related entities should be removed as well, given a root eid.

A fourth solution is to introduce a special attribute that can mark attributes like :message/user and give a retract-strategy. When your custom retract-entity function creates retraction data for a certain entity, it also looks for attributes in this entity tagged with some certain retract strategy. Like a more advanced :db/isComponent attribute.

Answered: What is the recommended way of dumping the whole datomic schema?

Fri, 05 Nov 2021 12:54:04 +0000

Recommendations may vary depending on what you want to accomplish?

If exploring an unknown schema is a goal the schema cartographer may be helpful.

Answered: Broken Web Link

Tue, 02 Nov 2021 12:37:14 +0000

Hi @ToBeJazz

These examples were moved into the Datomic Samples (https://docs.datomic.com/cloud/dev-local.html#samples) available in Dev-local. The link in the docs should likely now point to https://github.com/cognitect-labs/day-of-datomic-cloud/blob/master/tutorial/filters.repl

I will address the docs and update! Thanks for pointing this out.

-Jaret

Answered: Data model for simple shopping app

Thu, 07 Oct 2021 14:32:00 +0000

I think I found my answer and modelled it in the same way as the course registration example in https://docs.datomic.com/cloud/schema/schema-reference.html#composite-tuples

Answered: Datom "value" - content?

Thu, 30 Sep 2021 20:53:32 +0000

The value of any given attribute is determined by the schema definition for that attribute. The valid types for an attribute can be found here.

A value can certainly be an entity id. You would expect :db.type/ref to have a value referencing another entity.

:green in Clojure is a keyword

Answered: noHistory and transaction data

Fri, 27 Aug 2021 13:36:24 +0000

Hi Mitchell,

Yes, the transaction will still be present in the log. The log is a recording of all transaction data in historic order, organized for efficient access. But I would be highly skeptical of this affecting your overall performance/size significantly, and only in reads of the log. Could you help me understand what your underlying concern here is when you say "database growth issues"?

Perhaps it would be useful to have a call to discuss the specifics of your use case. If that's something you'd be interested in I can setup a call with members of our team, just shoot me an e-mail via support@cognitect.com or jaret@cognitect.com.

Cheers!
-Jaret

Answered: Highly available deployments when desired capacity is 1

Fri, 20 Aug 2021 13:46:30 +0000

In Slack, Jaret Binford answered:

That min during update is for CloudFormation updates (eg. updating to a new Datomic release.) Not Code Deploy. Code Deploy deploys to running instances. If you want HA during a Code Deploy you would need to have the desired instances set to 2.

This aligns with the config's description that I had failed to read properly:

Minimum number of instances during update: Minimum number of instances that must be in service within Auto Scaling group while AWS CloudFormation updates old instances.

So the setting is about compute group updates, not our app's updates.

Jaret wrote further:

If min during update is 1, CF should scale up to 2 during an upgrade and then kill the previous instance as long as the maximum allows for that.

How to authenticate and authorize access to API Gateway endpoints for datomic?

Thu, 19 Aug 2021 17:05:29 +0000

I'm currently evaluating Datomic and I've deployed a production topology using a split stack. DatomicCloudVersion is 9095 and DatomicCFTVersion is 884. The CloudFormation stack created two API Gateway endpoints: datomic-[system]-client-api and datomic-[system]-ions.

From my local computer I can get a client by using datomic-[system]-client-api endpoint:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

Above, https://[endpoint-id].execute-api.us-east-1.amazonaws.com/ is the client-api API Gateway endpoint. It works, but the endpoint is open to the world. I then added an IAM authorizer to datomic-[system]-client-api and used an authorized IAM user to make a request to the now authenticated endpoint:

awscurl --service execute-api -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com --profile datomic
{:s3-auth-path "long s3 path"}

awscurl is just a tool to make authenticated calls to AWS services. As you can see, I can get a response from the endpoint by using my local profile named "datomic". If I try using regular curl without the v4 AWS signature it fails:

curl -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com
{"message":"Forbidden"}

Works as expected. The problem is now I can't get a client using:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :creds-profile "datomic"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

The response I get is:

{:status 403,
:headers
{"apigw-requestid" "EUsyNi9aIAMEPlg=",
"connection" "keep-alive",
"content-length" "23",
"date" "Thu, 19 Aug 2021 17:01:40 GMT",
"content-type" "application/json"},
:body "{\"message\":\"Forbidden\"}"}

It is as if the d/client call is ignoring the profile or not generating the proper signature. I tried setting the AWS_PROFILE environment variable as well, to no avail.

Is using an IAM Authorizer not the proper way to go? How can I protect both the client-api and ions endpoints? Finally, what's the difference between the client-api and ions endpoints?

Datomic returns empty results where "Insufficient bindings, will cause db scan" exception is expected

Wed, 07 Jul 2021 14:30:05 +0000

Hello

If we change #uuid"..." to "" (from uuid to string), we get the exception (desired behavior)

clj -Sdeps '{:deps {com.datomic/dev-local {:mvn/version "0.9.232"}}}' -M \
  -e "(require '[datomic.client.api :as d])" \
  -e '(def client (d/client {:server-type :dev-local :system "hello"}))' \
  -e '(d/create-database client {:db-name "hello"})' \
  -e '(def conn (d/connect client {:db-name "hello"}))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
=> #'user/client
=> true
=> #'user/conn
=> []

Occour on (old) versions of datomic-free too

clj -Sdeps '{:deps {com.datomic/datomic-free {:mvn/version "0.9.5697"}}}' \
  -M --report stderr \
  -e "(require '[datomic.api :as d])" \
  -e '(d/create-database "datomic:mem://hello")' \
  -e '(def conn (d/connect "datomic:mem://hello"))' \
  -e "(d/q '[:find ?e :where [?e _ #uuid\"00000000-0000-0000-0000-000000000000\"]] (d/db conn))"
WARNING: requiring-resolve already refers to: #'clojure.core/requiring-resolve in namespace: datomic.common, being replaced by: #'datomic.common/requiring-resolve
=> true
=> #'user/conn
=> #{}

Answered: Query values for more than one entity in a single datomic query (Roam Alpha Api)

Fri, 02 Jul 2021 02:45:50 +0000

I believe you want something similar to the following:

(d/q '[:find ?e ?stri
       :in $ [?block-id ...]
       :where [?e :block/uid ?block-id]
              [?e :block/string ?stri]] 
     db ["YjpbFUsTx" "TgpgOssBM"])

https://docs.datomic.com/cloud/query/query-data-reference.html#collection-binding

Answered: Failure to transact a DB after delete and re-create

Fri, 25 Jun 2021 17:48:00 +0000

Prefer generating or appending unique (e.g., UUID) names to databases that are intended to be short-lived. Deleting and creating a database of the same name in a tight cycle is not a behavior that Datomic makes guarantees for or is designed around.

My guess is the deployment step is helpful only because it takes time which is allowing proper cleanup of the previous database to occur.

ClassCastException, d/with inside transact function

Thu, 24 Jun 2021 14:57:51 +0000

Calling a transaction function that uses d/with in a dev-local, filesystem (i.e., not :storage-dir :mem) throws a ClassCastException on internal Datomic code.

Environment

com.datomic/dev-local {:mvn/version "0.9.232"}
client-cloud 0.8.105

History

Example with exception using file system dev-local.

(def client (d/client {:server-type :dev-local
                         :storage-dir "/tmp/cce-bug3"
                         :system      "test"}))
=> #'user/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'user/conn
(d/transact conn {:tx-data [{:db/ident       :user/name
                               :db/valueType   :db.type/string
                               :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "test",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "test",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T00:49:46.400-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(defn my-tx-fn
    [db argm]
    (let [tx-report (d/with db {:tx-data [{:user/name "asd"}]})]
      []))
=> #'user/my-tx-fn
(d/transact conn {:tx-data [(list 'user/my-tx-fn {})]})
Execution error (ClassCastException) at datomic.dev-local.tx/datom-lookup-valfn (tx.clj:397).
class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')

Working memdb example.

(def client-memdb (d/client {:server-type :dev-local
                               :storage-dir :mem
                               :system      "test"}))
=> #'user/client-memdb
(d/create-database client-memdb {:db-name "test"})
=> true
(def conn-memdb (d/connect client-memdb {:db-name "test"}))
=> #'user/conn-memdb
(d/transact conn-memdb {:tx-data [{:db/ident       :user/name
                                     :db/valueType   :db.type/string
                                     :db/cardinality :db.cardinality/one}]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:02:57.965-00:00" 13194139533318 true]
           #datom[73 10 :user/name 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[0 13 73 13194139533318 true]],
 :tempids {}}
(d/transact conn-memdb {:tx-data [(list 'user/my-tx-fn {})]})
=>
{:db-before #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                                :basisT 6,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "394c7a65-bfeb-4dd4-b188-7aecd2057692",
                               :basisT 7,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533319 50 #inst"2021-05-09T01:03:08.042-00:00" 13194139533319 true]],
 :tempids {}}

Expectation

Transaction functions that use d/with do not throw a ClassCastException. Additionally, the behavior of d/with in all Datomic environments (Cloud, File system, Memdb) is consistent. Given a previous Datomic Cloud release fixed a problem with using with-db from an Ion, it seems using d/with in a transaction function is a supported behavior.

Actual

The final d/transact transaction function throws a ClassCastException when using d/with on some internal Datomic code. This only occurs when using a dev-local, file system environment.

Full stacktrace is pasted below.

#error{:cause "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
       :via [{:type java.lang.ClassCastException,
              :message "class datomic.core.db.Datum cannot be cast to class java.lang.Number (datomic.core.db.Datum is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')",
              :at [datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]}],
       :trace [[datomic.dev_local.tx$datom_lookup_valfn invokeStatic "tx.clj" 397]
               [datomic.dev_local.tx$datom_lookup_valfn invoke "tx.clj" 397]
               [datomic.dev_local.local_log.LocalLog valAt "local_log.clj" 56]
               [clojure.lang.RT get "RT.java" 760]
               [datomic.dev_local.btindex.BTIndex cons "btindex.clj" 281]
               [clojure.lang.RT conj "RT.java" 677]
               [clojure.core$conj__5390 invokeStatic "core.clj" 85]
               [clojure.core$conj__5390 invoke "core.clj" 82]
               [datomic.core.db.Db addData "db.clj" 2322]
               [datomic.core.db$add_ensured_data invokeStatic "db.clj" 3353]
               [datomic.core.db$add_ensured_data invoke "db.clj" 3351]
               [datomic.core.db$with_tx invokeStatic "db.clj" 3370]
               [datomic.core.db$with_tx invoke "db.clj" 3357]
               [datomic.core.db.Db with "db.clj" 2164]
               [datomic.core.local_db$fn__25633 invokeStatic "local_db.clj" 67]
               [datomic.core.local_db$fn__25633 invoke "local_db.clj" 24]
               [datomic.client.api.protocols$fn__11959$G__11877__11966 invoke "protocols.clj" 126]
               [datomic.client.api$with invokeStatic "api.clj" 363]
               [datomic.client.api$with invoke "api.clj" 353]
               [user$my_tx_fn invokeStatic "user.clj" 3]
               [user$my_tx_fn invoke "user.clj" 20]
               [clojure.lang.AFn applyToHelper "AFn.java" 156]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.lang.Var applyTo "Var.java" 705]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$apply invoke "core.clj" 660]
               [datomic.core.db.ProcessExpander inject "db.clj" 3229]
               [datomic.core.db.ProcessInpoint inject "db.clj" 2950]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586$fn__18587 invoke "btindex_db.clj" 440]
               [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
               [clojure.core$reduce invokeStatic "core.clj" 6827]
               [clojure.core$reduce invoke "core.clj" 6810]
               [datomic.dev_local.btindex_db$expand_tx$inject_all__18586 invoke "btindex_db.clj" 440]
               [datomic.dev_local.btindex_db$expand_tx invokeStatic "btindex_db.clj" 444]
               [datomic.dev_local.btindex_db$expand_tx invoke "btindex_db.clj" 429]
               [datomic.dev_local.btindex_db$with_tx invokeStatic "btindex_db.clj" 468]
               [datomic.dev_local.btindex_db$with_tx invoke "btindex_db.clj" 461]
               [datomic.dev_local.impl.DurableConnection transact "impl.clj" 161]
               [datomic.client.api$transact invokeStatic "api.clj" 200]
               [datomic.client.api$transact invoke "api.clj" 183]
               [user$eval124260 invokeStatic "user.clj" 25]
               [user$eval124260 invoke "user.clj" 25]
               [clojure.lang.Compiler eval "Compiler.java" 7177]
               [clojure.lang.Compiler eval "Compiler.java" 7132]
               [clojure.core$eval invokeStatic "core.clj" 3214]
               [clojure.core$eval invoke "core.clj" 3210]
               [nrepl.middleware.interruptible_eval$evaluate$fn__959 invoke "interruptible_eval.clj" 91]
               [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
               [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
               [clojure.main$repl$fn__9095 invoke "main.clj" 458]
               [clojure.main$repl invokeStatic "main.clj" 458]
               [clojure.main$repl doInvoke "main.clj" 368]
               [clojure.lang.RestFn invoke "RestFn.java" 1523]
               [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
               [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
               [nrepl.middleware.interruptible_eval$interruptible_eval$fn__985$fn__989
                invoke
                "interruptible_eval.clj"
                155]
               [clojure.lang.AFn run "AFn.java" 22]
               [nrepl.middleware.session$session_exec$main_loop__1086$fn__1090 invoke "session.clj" 190]
               [nrepl.middleware.session$session_exec$main_loop__1086 invoke "session.clj" 189]
               [clojure.lang.AFn run "AFn.java" 22]
               [java.lang.Thread run "Thread.java" 829]]}

Evidence

The previously pasted example is reproducible and always throws a ClassCastException.

Impact

We are unable to test functionality in a consistent manner. Local setups using dev-local file systems will not function as expected. Developers are unable to test features in the same way they will work in production.

d/client external server communication

Thu, 24 Jun 2021 14:55:59 +0000

Environment

com.datomic/client-cloud 0.8.105

History

Get the docstring for d/client.

(clojure.repl/doc d/client)
-------------------------
datomic.client.api/client
([arg-map])
  Create a client for a Datomic system. This function does not
communicate with a server and returns immediately.

For a cloud system, arg-map requires:

  :server-type   - :cloud
  :region        - AWS region, e.g. "us-east-1"
  :system        - your system name
  :endpoint      - IP address of your system or query group

Optionally, a cloud system arg-map accepts:

  :creds-provider  - instance of com.amazonaws.auth.AWSCredentialsProvider. Defaults to DefaultAWSCredentialsProviderChain
  :creds-profile   - name of an AWS Named Profile. See http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
  :proxy-port      - local port for SSH tunnel to bastion 

  Note: :creds-provider and :creds-profile are mutually exclusive, providing both will result in an error.

For a dev-local system, arg-map comprises:

  :server-type   - :dev-local (required)
  :system        - a system name (required)
  :storage-dir   - optional, overrides :storage-dir in ~/.datomic/dev-local.edn

dev-local stores databases under ${storage-dir}/${system}/${db-name}.

For a peer-server system, arg-map requires:

  :server-type   - :peer-server
  :access-key    - access-key from peer server launch
  :secret        - secret from peer server launch
  :endpoint      - peer server host:port
  :validate-hostnames  - false

Returns a client object.

Note the second sentence in the docstring: "This function does not communicate with a server and returns immediately." Next call d/client with an endpoint that does not exist.

(d/client {:server-type :ion
             :system      "my-system"
             :endpoint    "http://i-dont-exist.datomic.net:8182"
             :region      "us-west-2"})
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to http://i-dont-exist.datomic.net:8182

Expectation

Calling d/client does not communicate with a server.

Actual

Calling d/client does communicate with a server.

Evidence

The above reproducible example indicates the d/client function call is attempted to communicate the the passing in :endpoint string given the exception thrown.

Impact

The d/client docstring misleads the user by saying no server communication will occur when it does occur. The result is calls to d/client can fail in unexpected ways when an external server communication is not accounted for. For example, we spin up Datomic Cloud query groups dynamically and attempt to communicate with the newly spun up cluster. Until the cluster is up and healthy, we expect calls to fail at d/connect with an unavailable anomaly, and instead the calls would fail at d/client with an unavailable anomaly.

Answered: DDB size reduction after restore

Fri, 18 Jun 2021 14:57:15 +0000

Just curios, how much memory the peer eats? Is it the same size for both DBs?

Add query group ASG metrics to query group CW dashboard

Thu, 27 May 2021 17:22:27 +0000

When looking at the query group dashboard, it is helpful to know what "state" the ASG was in when referencing the various dashboard metrics. Currently, I must do one of the following: 1) reference these metrics in the ASG monitoring tab (e.g., https://us-west-2.console.aws.amazon.com/ec2autoscaling/home?region=us-west-2#/details/my-query-group?view=monitoring), 2) create a CW dashboard of my own, 3) modify the existing Datomic CW dashboard to add this chart. The ASG metrics answer questions like: 1) how many instances was my query group running? 2) was my query group trying to scale out or in? 3) why did my query group not scale out or in?

I would find the following ASG metrics (in order of importance) useful additions to the current query group dashboard.

In Service Instances (Count)
Minimum Group Size (Count), Maximum Group Size (Count), and Desired Capacity (Count)
Pending Instances (Count)

How might multiple systems serve the same database in different contexts?

Mon, 24 May 2021 15:16:46 +0000

From the guide on naming there is the following quote:

Multiple systems might serve the same primary database in different development stages. So you might name systems via the convention "[db]-[stage]", e.g."inventory-dev", "inventory-staging" and "inventory-prod".

I'm trying to understand how this would work, and how I should think about my system, application, and database name, but I'm not really sure how this would work.

My understanding was that you have one System per Storage stack, plus some number of Compute stacks, and that databases are "owned" by the Storage stack, but that doesn't make sense with the quote above.

Can anyone give some examples of how this would work? I've read the planning guide but almost certainly don't fully understand it.

Thank you!
Evan

What is a performant way to do "and" matching on attribute values?

Fri, 21 May 2021 18:04:41 +0000

So I can use

[?entity1 ?attrname1 ?attrval]
[?entity2 ?attrname2 ?attrval]

in a :where clause to get ?entity1 and ?entity2 where there exists an ?attrval that matches

I'm using

  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity1 ?attrname1) [[?attrvals1]]]
  [(q '[:find (seq ?attrval)
        :in $ ?entity ?attrname
        :where [?entity ?attrname ?attrval]]
      db ?entity2 ?attrname2) [[?attrvals2]]]
  (not-join [?attrvals1 ?attrvals2]
               [(seq ?attrvals1) [?element ...]]
               (not [(contains? ?attrvals2 ?element)]))

to get ?entity1 and ?entity2 where all ?attrvals for ?entity1 exist for ?entity2. Is there a more performant way to do this??
(This feels like a directional "and" to the implicit "or" being applied to each attrval matching in the first case)

AssertionError shows up as an Event?

Tue, 18 May 2021 15:33:51 +0000

This surprised me. In a lambda invocation, a failed assert shows up as an Event. I would have expected Alert:

"Msg": "IonLambdaException",
"Ex": {
    "Via": [
        {
            "Type": "java.lang.AssertionError",
            "Message": "Assert failed: (= (count s) 16)",
        }
    ],
    "Cause": "Assert failed: (= (count s) 16)"
},
"Type": "Event",
"Tid": 338,
"Timestamp": 162134781093

Use Launch Template instead of Launch Configuration

Fri, 07 May 2021 23:38:51 +0000

The compute Datomic CloudFormation templates (primary and query groups) create ASGs using launch configurations. Launch configurations do not support the full set of Auto Scaling group features. For example, you cannot create an Auto Scaling group that launches both Spot and On-Demand Instances or that specifies multiple instance types or multiple launch templates. In fact, AWS recommends creating ASGs using launch templates, not launch configurations.

We are interested in launch templates for its support for using Spot and On-Demand instances.

EDIT: As of 2021-10-20, launch templates will not receive any new EC2 ASG features.

Support for recent AWS instance types

Wed, 24 Mar 2021 18:36:03 +0000

AWS has released several new instance types that are much better price per performance. It'd be great to have the Datomic query group template updated to support these instance types. The Datomic query group template currently supports the following instance types: t3.medium, m5.large, i3.large, i3.xlarge. Based on this, I propose the following additions:

t3a.medium (~10% cheaper than t3.medium)
m5a.large (~10% cheaper than m5.large)

If Datomic can run on ARM based processors then further additions can be made with even better price per performance (the new AWS Graviton2 processors are fantastic. Read about Honeycomb's success).

t4g.medium (perhaps also large and xlarge, ~20% cheaper than t3.medium)
m6g.large (~20% cheaper than m5.large)
r6gd.large (has NVMe based SSD, 26% cheaper than i3.large)
I4i (released 2022-04-28)

The Datomic Solo instance type could also be configurable for some inconsequential savings. See monthly prices below:

(Current) t3.small primary + t3.nano bastion: $18.90
t3a.small + t3a.nano: $17.15
t4g.small + t4g.nano: $15.33

An example Datomic query group monthly prices. Take a query group with 2 nodes running for the full month.

(Current) m5.large: $140.16
m5a.large: $125.56 ($14.6/month savings)
m6g.large: $112.42 ($27.74/month savings)

Make DesiredCapacity an optional parameter in the query group CF template

Wed, 24 Mar 2021 17:24:25 +0000

Currently (March 2021) the Datomic query group CF template has the DesiredCapacity parameter marked as required. Because of this, during a CF template update, CF will always set the DesiredCapacity to the CF parameter for DesiredCapacity. This behavior does not work well in the following situation.

In the query group template I have the DesiredCapacity at 2, and MaxCount at 4. I have an ASG scaling policy to target track 50% average CPU utilization. My query group is currently at the max, because the ASG scaling policy scaled up the group DesiredCapacity to 4 (to match my load). My load has increased to the point where 4 nodes is no longer sufficient. I must update the query group CF template to increase the MacCount. The problem is that when you update the MacCount from 4 to 6 via a CF template update and leave the DesiredCapacity at 2, CF will actually set the DesiredCapacity on the ASG to 2 during the high load situation. This worsens the high load situation.

One option is to query the ASG for its current DesiredCapacity prior to running a CF update and set the CF parameter to the value you retrieved. This approach complicates the update procedure. It also has a "read-modify-write" race condition -- the DesiredCapacity could have changed from the time you read the DesiredCapacity to the time you are performing the CF update.

A second option is to make the DesiredCapacity parameter optional. As per the CF docs, DesiredCapacity is not a required parameter. When it is not present, it defaults to the minimum size of the group. Leaving DesiredCapacity unset will make it so during a CF update, CF will not modify the DesiredCapacity at all -- leaving the ASG "in control."

I am now seeing Iterator+offset is not available doing a an index-pull

Mon, 08 Mar 2021 16:46:35 +0000

I am doing an index-pull in Datomic Cloud. The request was timing out so I addedd :timeout 300000 and not "Iterator+offset is not available" exception starts popping up

We are on 781-9041

Answered: Stack creation fails in AWS Control-Tower enrolled account

Fri, 26 Feb 2021 14:40:21 +0000

After cleaning out the leftovers from my previous creation attempts, the Cloud Formation succeeded in creating the stack.

I made sure not to schedule the deletion of the KMS key for datomic, as it might impede the re-creation of the stack.

I'm not sure what made it pass this time, but it did.

So if you were wondering like me, it looks like Control Tower's default guardrails are not incompatible with Datomic Cloud.

This was our second stack under AWS Control Tower, but our previous stack had been created in an AWS account prior to its enrollment in the Organization.

Answered: Are Ions the only way to develop applications with Datomic Cloud?

Fri, 26 Feb 2021 12:41:13 +0000

I'm under the impression that any EC2 instance you stand in the datomic cloud's VPC will be able to connect to Datomic Cloud through the Datomic client-cloud lib.

I never tried it, though. Please share back if you find a definitive answer!

Answered: String length limit cloud vs dev-local

Tue, 05 Jan 2021 16:04:48 +0000

The limit in Cloud is 4,096 and are correct as documented in both cases. You should write programs that adhere to this documentation because the implementation could enforce the documented limit at any time.

Answered: Can you use d/with inside a Cloud transaction function?

Mon, 04 Jan 2021 09:12:24 +0000

Found this bug and the discussions says it has been fixed https://forum.datomic.com/t/d-with-not-working-inside-tx-fn/1243

So it seems, that it should be possible.

Dev-local query does not support :timeout

Wed, 30 Dec 2020 21:25:39 +0000

Dev-local 0.9.229 does not appear to support :timeout when querying. Here is a repro.

(def client (d/client {:server-type :dev-local
                       :system      "example"}))
(d/create-database client {:db-name "example"})
(def conn (d/connect client {:db-name "example"}))
(d/transact conn {:tx-data [{:db/ident       :string
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string}]})
(dotimes [n 1000]
  (d/transact conn {:tx-data (map (fn [i]
                                    {:string (str n i)}) (range 1000))}))
(first
  (d/q {:query   '[:find (pull ?e [*])
                   :where
                   [?e :string]]
        :timeout 1000
        :args    [(d/db conn)]}))

The query takes longer than 1s and does not throw. I would have expected an exception thrown.

Allow dev-local import-cloud :source and :dest to work with any :server-type

Sun, 20 Dec 2020 00:37:33 +0000

Currently import-cloud requires the :source client :server-type to be :cloud and the :dest :server-type to be :dev-local. It'd be great if this function allowed any :server-type to be passed for :source and :dest. A possible use case would be copying a dev-local database to a cloud system for testing in a staging environment.