Datomic Knowledgebase - Recent questions tagged client api

Error connecting to datomic cloud

Mon, 12 Dec 2022 00:30:29 +0000

I'm trying to connect with the datomic cloud, but when I connect with the client I get the following error:
Execution error (NoClassDefFoundError) at com.amazonaws.services.s3.AmazonS3ClientBuilder/standard (AmazonS3ClientBuilder.java:46).
Could not initialize class com.amazonaws.services.s3.S3CredentialsProviderChain

java.lang.IllegalArgumentException: Cannot resolve key: 24a96e20-f526-4f7f-ba38-4f684caa5607

Mon, 06 Jun 2022 19:37:30 +0000

I am trying to implement resursive rule (loglcal OR) by following this video - Lucas Cavalcanti & Edward Wible - Exploring four hidden superpowers of Datomic

Here the code

(defn owns? [cid pid db]
(d/q '{:find [?pur]

     :in [$ ?cus-id ?pur %]
     :where
     [(owns? ?cus-id ?pur)]}
db cid [:purchase/id pid] owner-rules))

(comment
(owns?

#uuid "0fb7ea94-44af-46fa-98ca-0ddb5eb23123"
#uuid "24a96e20-f526-4f7f-ba38-4f684caa5607"
(d/db bank-conn)))

While running the above example I am getting this error:
java.lang.IllegalArgumentException Cannot resolve key: 24a96e20-f526-4f7f-ba38-4f684caa5607

Here is the full code:

(def purchase-schema
[{:db/ident :purchase/id

:db/doc "The purchase ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :purchase/account

:db/doc "Purchase Account"
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}])

(def account-schema
[{:db/ident :account/id

:db/doc "The Account ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :account/customer

:db/doc "Customer who own's this account"
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}])

(def customer-schema
[{:db/ident :customer/id

:db/doc "Customer ID"
:db/unique :db.unique/identity
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one}

{:db/ident :customer/name

:db/doc "Customer name"
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one}])

;;
;; Create client
(def client (d/client {:server-type :dev-local

                   :system "dev"}))

;; Create connections
(def bank-conn (d/connect client {:db-name "bank"}))

;; Transact schemas
(comment
(d/transact bank-conn {:tx-data purchase-schema})
(d/transact bank-conn {:tx-data account-schema})
(d/transact bank-conn {:tx-data customer-schema})
(d/tx-range bank-conn {:start 6}))

(comment
(java.util.UUID/randomUUID))

;; Transact fake data
(def cid #uuid "0fb7ea94-44af-46fa-98ca-0ddb5eb23123")
(def sample-customer {:customer/id cid

                  :customer/name "Jon Snow"})

(def aid #uuid "c579a9f6-f6c3-4718-8ecf-c3003021ee9a")
(def sample-account {:account/id aid

                 :account/customer [:customer/id cid]})

(def pid #uuid "24a96e20-f526-4f7f-ba38-4f684caa5607")
(def sample-purchase {:purchase/id pid

                  :purchase/account [:account/id aid]})

(comment
(d/transact bank-conn {:tx-data [sample-customer]})
(d/transact bank-conn {:tx-data [sample-account]})
(d/transact bank-conn {:tx-data [sample-purchase]}))

(comment
(d/q '[:find ?cus

     :in $ ?cid ?pid
     :where
     [?pur :purchase/id ?pid]
     [?pur :purchase/account ?acc]
     [?acc :account/customer ?cus]
     [?cus :customer/id ?cid]]
  (d/db bank-conn)
  (:customer/id sample-customer)
  (:purchase/id sample-purchase)))

;; owns v1
(defn ownsv1? [cid pid db]
(d/q '[:find ?cus

     :in $ ?customer-id ?purchase-id
     :where
     [?pur :purchase/id ?purchase-id]
     [?pur :purchase/account ?acc]
     [?acc :account/customer ?cus]
     [?cus :customer/id ?customer-id]]
db cid pid))

(comment
(ownsv1?

(:customer/id sample-customer)
(:purchase/id sample-purchase)
(d/db bank-conn)))

;; recursive rule (logical OR)
(def owner-rules
'[[(owns? ?cus-id ?e)

 [?e :customer/id ?cus-id]]
[(owns? ?cus-id ?e)
 [?e ?ref-attr ?r]
 (owns? ?cus-id ?r)]])

(defn owns? [cid pid db]
(d/q '{:find [?pur]

     :in [$ ?cus-id ?pur %]
     :where
     [(owns? ?cus-id ?pur)]}
db cid [:purchase/id pid] owner-rules))

;;
(comment
(owns?

(:customer/id sample-customer)
(:purchase/id sample-purchase)
(d/db bank-conn)))

`Unable to read keyfile` error when trying to initialize Datomic Cloud with Client API

Mon, 23 Aug 2021 13:25:08 +0000

Howdy all.

I'm testing setting up Datomic Cloud using an IntelliJ IDE. I'm following the Client API tut from Datomic but am stuck initializing the client.

The spec from an API client is here, and the tut is here, under the step Using Datomic Cloud.

So the tut says to init a client like so:

(require '[datomic.client.api :as d])
(def cfg {:server-type :ion
      :region "<your AWS Region>" ;; e.g. us-east-1
      :system "<system name>"
      :creds-profile "<your_aws_profile_if_not_using_the_default>"
      :endpoint "<your endpoint>"})

They say to include an AWS profile if not using the default. I am using the default as far as I know--I'm not part of an org in AWS.

This is the (partially redacted) code from my tutorial.core namespace, where I'm trying to init Datomic:

(ns tutorial.core   
(:gen-class))
(require '[datomic.client.api :as d])
(def cfg {:server-type :cloud
      :region "us-east-2"
      :system "roam"
      :endpoint "https://API_ID.execute-api.us-east-2.amazonaws.com"
      })
(def client (d/client cfg))
(d/create-database client {:db-name "blocks"})
(d/connect client {:db-name "blocks"})

However, I'm getting an error from Datomic like: Forbidden to read keyfile at s3://URL/roam/datomic/access/admin/.keys. Make sure that your endpoint is correct, and that your ambient AWS credentials allow you to GetObject on the keyfile.

Do I need some sort of credential? Could anything else be causing this error? I got the endpoint URL from the ClientApiGatewayEndpoint in my CloudFormation Datomic stack.

Please let me know if I should provide more info! Thanks.

How to authenticate and authorize access to API Gateway endpoints for datomic?

Thu, 19 Aug 2021 17:05:29 +0000

I'm currently evaluating Datomic and I've deployed a production topology using a split stack. DatomicCloudVersion is 9095 and DatomicCFTVersion is 884. The CloudFormation stack created two API Gateway endpoints: datomic-[system]-client-api and datomic-[system]-ions.

From my local computer I can get a client by using datomic-[system]-client-api endpoint:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

Above, https://[endpoint-id].execute-api.us-east-1.amazonaws.com/ is the client-api API Gateway endpoint. It works, but the endpoint is open to the world. I then added an IAM authorizer to datomic-[system]-client-api and used an authorized IAM user to make a request to the now authenticated endpoint:

awscurl --service execute-api -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com --profile datomic
{:s3-auth-path "long s3 path"}

awscurl is just a tool to make authenticated calls to AWS services. As you can see, I can get a response from the endpoint by using my local profile named "datomic". If I try using regular curl without the v4 AWS signature it fails:

curl -X GET https://[endpoint-id].execute-api.us-east-1.amazonaws.com
{"message":"Forbidden"}

Works as expected. The problem is now I can't get a client using:

(d/client {:system "my-system"
           :server-type :ion
           :region "us-east-1"
           :creds-profile "datomic"
           :endpoint "https://[endpoint-id].execute-api.us-east-1.amazonaws.com/"})

The response I get is:

{:status 403,
:headers
{"apigw-requestid" "EUsyNi9aIAMEPlg=",
"connection" "keep-alive",
"content-length" "23",
"date" "Thu, 19 Aug 2021 17:01:40 GMT",
"content-type" "application/json"},
:body "{\"message\":\"Forbidden\"}"}

It is as if the d/client call is ignoring the profile or not generating the proper signature. I tried setting the AWS_PROFILE environment variable as well, to no avail.

Is using an IAM Authorizer not the proper way to go? How can I protect both the client-api and ions endpoints? Finally, what's the difference between the client-api and ions endpoints?

How to properly configure access to IonApiGateway?

Fri, 16 Jul 2021 14:50:26 +0000

How to properly configure access to IonApiGateway? Can IAM be used and how? Or is there some other approach?

I would be very glad if you can help. I read the documentation, but did not find a solution for configuring access.

d/client external server communication

Thu, 24 Jun 2021 14:55:59 +0000

Environment

com.datomic/client-cloud 0.8.105

History

Get the docstring for d/client.

(clojure.repl/doc d/client)
-------------------------
datomic.client.api/client
([arg-map])
  Create a client for a Datomic system. This function does not
communicate with a server and returns immediately.

For a cloud system, arg-map requires:

  :server-type   - :cloud
  :region        - AWS region, e.g. "us-east-1"
  :system        - your system name
  :endpoint      - IP address of your system or query group

Optionally, a cloud system arg-map accepts:

  :creds-provider  - instance of com.amazonaws.auth.AWSCredentialsProvider. Defaults to DefaultAWSCredentialsProviderChain
  :creds-profile   - name of an AWS Named Profile. See http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
  :proxy-port      - local port for SSH tunnel to bastion 

  Note: :creds-provider and :creds-profile are mutually exclusive, providing both will result in an error.

For a dev-local system, arg-map comprises:

  :server-type   - :dev-local (required)
  :system        - a system name (required)
  :storage-dir   - optional, overrides :storage-dir in ~/.datomic/dev-local.edn

dev-local stores databases under ${storage-dir}/${system}/${db-name}.

For a peer-server system, arg-map requires:

  :server-type   - :peer-server
  :access-key    - access-key from peer server launch
  :secret        - secret from peer server launch
  :endpoint      - peer server host:port
  :validate-hostnames  - false

Returns a client object.

Note the second sentence in the docstring: "This function does not communicate with a server and returns immediately." Next call d/client with an endpoint that does not exist.

(d/client {:server-type :ion
             :system      "my-system"
             :endpoint    "http://i-dont-exist.datomic.net:8182"
             :region      "us-west-2"})
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to http://i-dont-exist.datomic.net:8182

Expectation

Calling d/client does not communicate with a server.

Actual

Calling d/client does communicate with a server.

Evidence

The above reproducible example indicates the d/client function call is attempted to communicate the the passing in :endpoint string given the exception thrown.

Impact

The d/client docstring misleads the user by saying no server communication will occur when it does occur. The result is calls to d/client can fail in unexpected ways when an external server communication is not accounted for. For example, we spin up Datomic Cloud query groups dynamically and attempt to communicate with the newly spun up cluster. Until the cluster is up and healthy, we expect calls to fail at d/connect with an unavailable anomaly, and instead the calls would fail at d/client with an unavailable anomaly.

Add default :timeout to client api ns docstring

Thu, 24 Dec 2020 00:12:44 +0000

It'd be great to have the default value for :timeout (60000) included in the datomic.client.api ns docstring. Currently this value can only be found in the docs on the website. As of client-cloud 0.8.102, the ns docstring specifies the default value for :limit.

Datomic Client API equivalent to on-prem tx->t, t->tx

Mon, 21 Dec 2020 00:25:17 +0000

The Datomic client API does not provider tx->t or t->tx. These functions can be useful for very specific scenarios (e.g., you have a tx id and want to know if the database has "moved forward").

This was also requested on the forums. A workaround was provided for the user's specific case but the request is still relevant.

Datomic cloud client get-s3-auth-path: Unable to connect to localhost:8182

Thu, 19 Nov 2020 18:55:38 +0000

I could use some help debugging recent connection issues we're having with
a Datomic cloud solo topology. We were working to add analytics support so enabled it in Cloud
Formation, added and synced a config directory and adjusted our internal AWS
subaccounts to access via IAM roles.

After this we're unable to establish a connection through the client even though
it seems like all of the parts are working and wired up. I tried some
debugging by Google
and tried the suggestions of restarting the worker and gateway nodes but
unfortunately that didn't help.

If anyone could provide pointers to what to poke or try next to re-establish
authentication. it would be much appreciated.

Now all the details, first we successfully connect to the access Gateway:
`

$ datomic client access -r us-east-2 datomic-solo
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
[...]
debug1: Authentication succeeded (publickey).
Authenticated to 18.216.250.130 ([18.216.250.130]:22).
debug1: Local connections to LOCALHOST:8182 forwarded to remote address socks:0
debug1: Local forwarding listening on ::1 port 8182.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8182.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

`
The gateway connection seems to work:

$ curl -x socks5h://localhost:8182
http://entry.datomic-solo.us-east-2.datomic.net:8182/
{:s3-auth-path "datomic-solo-storagef7f305e7-wutene4nee-s3datomic-ci7fce3hk2gj"}

But when trying to connect in Clojure ((d/client cfg)), it times out:

Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:179).
Unable to connect to localhost:8182

ginkgo.metadata.core=> *e
#error {
 :cause "Unable to connect to localhost:8182"
 :data {:cognitect.anomalies/category :cognitect.anomalies/unavailable, :cognitect.anomalies/message "Total timeout 60000 ms elapsed", :config {:server-type :cloud, :region "us-east-2", :system "datomic-solo", :endpoint "http://entry.datomic-solo.us-east-2.datomic.net:8182", :proxy-port 8182, :endpoint-map {:headers {"host" "entry.datomic-solo.us-east-2.datomic.net:8182"}, :scheme "http", :server-name "entry.datomic-solo.us-east-2.datomic.net", :server-port 8182}}}
 :via
 [{:type java.lang.RuntimeException
   :message "could not start [#'ginkgo.metadata.ferment/conn] due to"
   :at [mount.core$up$fn__247 invoke "core.cljc" 94]}
  {:type clojure.lang.ExceptionInfo
   :message "Unable to connect to localhost:8182"
   :data {:cognitect.anomalies/category :cognitect.anomalies/unavailable, :cognitect.anomalies/message "Total timeout 60000 ms elapsed", :config {:server-type :cloud, :region "us-east-2", :system "datomic-solo", :endpoint "http://entry.datomic-solo.us-east-2.datomic.net:8182", :proxy-port 8182, :endpoint-map {:headers {"host" "entry.datomic-solo.us-east-2.datomic.net:8182"}, :scheme "http", :server-name "entry.datomic-solo.us-east-2.datomic.net", :server-port 8182}}}
   :at [datomic.client.impl.cloud$get_s3_auth_path invokeStatic "cloud.clj" 179]}]
 :trace
 [[datomic.client.impl.cloud$get_s3_auth_path invokeStatic "cloud.clj" 179]
  [datomic.client.impl.cloud$get_s3_auth_path invoke "cloud.clj" 170]
  [datomic.client.impl.cloud$create_client invokeStatic "cloud.clj" 211]
  [datomic.client.impl.cloud$create_client invoke "cloud.clj" 194]
```
On the access client side it seems to connect and forward okay:
```
debug1: Connection to port 8182 forwarding to socks port 0 requested.
debug1: channel 2: new [dynamic-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 8182 for
entry.datomic-solo.us-east-2.datomic.net port 8182, connect from 127.0.0.1 port
59870 to 127.0.0.1 port 8182, nchannels 3

This is where I'm a bit stuck and not sure where to debug next. Does anyone have
suggestions for what to poke or try next? Thanks much.

Usage of datomic client API's with in order to simulate facts without asserting.

Sun, 01 Nov 2020 16:48:56 +0000

Hello Clojurians,

According to Datomic client API the (with db tx-data) allows to make queries on a database with additionnal facts without asserting them. Afaik, it's also known as "what-if" query.

The docs says that the tx-data should be exactly the sam as in a transact.

But on my machine, this works:

(d/transact conn {:tx-data [{:token/color :blue}]})

But this doesn't:

(d/q '[:find (count ?e1)                             
       :in $                                                                                                                                         
       :where [?e1 :token/color :grey]]                                                                                                 
     (d/with (d/db conn) {:tx-data [{:token/color :grey}]}))

With this error:

  Show: Project-Only All                                                                                                                                                                                             
  Hide: Clojure Java REPL Tooling Duplicates  (11 frames hidden)                                                                                                                                                     
                                                                                                                                                                                                                     
1. Unhandled clojure.lang.ExceptionInfo                                                                                                                                                                              
   Datomic Client Exception                                                                                                                                                                                          
   {:cognitect.anomalies/category :cognitect.anomalies/incorrect,                                                                                                                                                    
    :http-result                                                                                                                                                                                                     
    {:status 400,                                                                                                                                                                                                    
     :headers                                                                                                                                                                                                        
     {"server" "Jetty(9.4.27.v20200227)",                                                                                                                                                                            
      "content-length" "247",                                                                                                                                                                                        
      "date" "Sun, 01 Nov 2020 16:13:07 GMT",                                                                                                                                                                        
      "content-type" "application/transit+msgpack"},                                                                                                                                                                 
     :body nil}}                                                                                                                                                                                                     
                 async.clj:   58  datomic.client.api.async/ares                                                                                                                                                      
                 async.clj:   54  datomic.client.api.async/ares                                                                                                                                                      
                  sync.clj:  120  datomic.client.api.sync/eval27181/fn                                                                                                                                               
             protocols.clj:  126  datomic.client.api.protocols/fn/G                                                                                                                                                  
                   api.clj:  363  datomic.client.api/with                                                                                                                                                            
                   api.clj:  353  datomic.client.api/with                                                                                                                                                            
                      REPL:  275  four.core/eval28597                                                                                                                                                                
                  core.clj: 3214  clojure.core/eval                                                                                                                                                                  
                  core.clj: 3210  clojure.core/eval                                                                                                                                                                  
                  main.clj:  437  clojure.main/repl/read-eval-print/fn                                                                                                                                               
                  main.clj:  458  clojure.main/repl/fn                                                                                                                                                               
                  main.clj:  368  clojure.main/repl                                                                                                                                                                  
               RestFn.java: 1523  clojure.lang.RestFn/invoke                                                                                                                                                         
                  AFn.java:   22  clojure.lang.AFn/run                                                                                                                                                               
                  AFn.java:   22  clojure.lang.AFn/run                                                                                                                                                               
               Thread.java:  844  java.lang.Thread/run

Any clue ?

`gcDeletedDBs` available in peer API

Sun, 25 Oct 2020 19:36:18 +0000

It would be very helpful to be able to call gcDeletedDBs programmatically from the API. Providing the felxibility of passing in a Datasource. Allows users to get GC capabilities in the app.

Store values encrypted, allowing key rotation.

Sun, 25 Oct 2020 19:14:02 +0000

In our application we have a requirement to store certain fields encrypted. This encryption is done in the application. However, we would like to be able to rotate the encryption key should it become compromised, and retain the historical query features that you would get with unencrypted storage. In order to rotate the key, we have to excise the data and re-transact it with the new key, or store the encrypted values outside of Datomic. It would be nice to have an encryption API that enabled this use case.

Ability to limit the number of results from query

Thu, 22 Oct 2020 17:42:57 +0000

Currently there is no built in way to limit the results from a query. This puts the task on the user to limit the results by first querying for entity ids, limiting that number and then pulling the data.

Client library for Python

Thu, 22 Oct 2020 16:45:51 +0000

Provide client implementation in Python

Client API: tx-report-queue equivalent

Thu, 22 Oct 2020 16:25:01 +0000

One of the features I miss a lot in the client API is TX report queues. Would it be possible to add something like this to the client? Perhaps tx-range could get an option that, when no :end is given, keeps the channel open and waits for future transactions?

Query explain for datalog

Thu, 22 Oct 2020 16:24:23 +0000

One of the places that we struggle with the most is understanding query performance. In general we get the idea that in order to write a fast query you need to reduce the search space as quickly as possible. As a result the most specific datalog clause should go first. However, we have found cases when that is not always true. For example when an attribute is a reference or when an attribute is indexed. Since we have little insight into how the query planner works its hard for us to tell what will make a query fast. We can and do optimize our queries by hand. But as our system grows and we add more queries this hand optimization becomes more burdensome. It would be great to have a tool, similar to a sql explain, that would give us the fastest ordering of the datalog clauses and provide suggestions for adding indexes.

Sorting and Pagination for Datomic Query results

Thu, 22 Oct 2020 16:08:51 +0000

Provide sorting and pagination for query results in the client and peer api's query.

Expose standby/active transactor status in on-prem metrics.

Thu, 22 Oct 2020 16:00:30 +0000

Provide a method through an API to allow monitoring of standby/active status of transactors in on-prem.