Datomic Knowledgebase - Recent questions tagged request

Pull identical var in two :find positions

Thu, 11 Nov 2021 00:15:32 +0000

Environment

com.datomic/client-cloud "1.0.117"
com.datomic/dev-local 1.0.238

History

(def client (d/client {:server-type :dev-local
                         :storage-dir :mem
                         :system      "test"}))
=> #'user/client
(d/create-database client {:db-name "a"})
=> true
(def conn (d/connect client {:db-name "a"}))
=> #'user/conn
(d/q '[:find (pull ?e [:db/doc]) (pull ?e [:db/ident])
         :where
         [?e :db/ident :db/cardinality]]
    (d/db conn))
Execution error (ArrayIndexOutOfBoundsException) at datomic.core.datalog/fn$project (datalog.clj:560).
Index 1 out of bounds for length 1

If you bind the same eid to a new var name, the exception will not be thrown.

(d/q '[:find (pull ?e [:db/doc]) (pull ?e2 [:db/ident])
       :where
       [?e :db/ident :db/cardinality]
       [(identity ?e) ?e2]]
  (d/db conn))
=>
[[#:db{:doc "Property of an attribute. Two possible values: :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes. Defaults to :db.cardinality/one."}
  #:db{:ident :db/cardinality}]]

Expectation

Pulling the same entity in multiple locations in :find will pull the relevant data and not thrown an ArrayIndexOutOfBoundsException.

Actual

Pulling the same entity in multiple locations in :find throws an ArrayIndexOutOfBoundsException.

Evidence

Full stacktrace:

#error{:cause "Index 1 out of bounds for length 1",
       :via [{:type clojure.lang.ExceptionInfo,
              :message "processing clause: [?e :db/ident :db/cardinality], message: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1",
              :data #:cognitect.anomalies{:category :cognitect.anomalies/incorrect,
                                          :message "processing clause: [?e :db/ident :db/cardinality], message: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1"},
              :at [datomic.core.datalog$throw_query_ex_BANG_ invokeStatic "datalog.clj" 50]}
             {:type java.lang.ArrayIndexOutOfBoundsException,
              :message "Index 1 out of bounds for length 1",
              :at [clojure.lang.RT aset "RT.java" 2384]}],
       :trace [[clojure.lang.RT aset "RT.java" 2384]
               [datomic.core.datalog$fn__22911$project__22994 invoke "datalog.clj" 560]
               [datomic.core.datalog$fn__22911$join__23012 invoke "datalog.clj" 659]
               [datomic.core.datalog$fn__22911$fn__23015 invoke "datalog.clj" 666]
               [datomic.core.common$pooled_mapv$fn__22667$fn__22668 invoke "common.clj" 490]
               [clojure.lang.AFn applyToHelper "AFn.java" 152]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1977]
               [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1977]
               [clojure.lang.RestFn invoke "RestFn.java" 425]
               [clojure.lang.AFn applyToHelper "AFn.java" 156]
               [clojure.lang.RestFn applyTo "RestFn.java" 132]
               [clojure.core$apply invokeStatic "core.clj" 671]
               [clojure.core$bound_fn_STAR_$fn__5767 doInvoke "core.clj" 2007]
               [clojure.lang.RestFn invoke "RestFn.java" 397]
               [clojure.lang.AFn call "AFn.java" 18]
               [java.util.concurrent.FutureTask run "FutureTask.java" 264]
               [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1128]
               [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 628]
               [java.lang.Thread run "Thread.java" 829]]}

Impact

Programmatically built queries that expect two positions in :find must apply a workaround to avoid the exception.

Add query group ASG metrics to query group CW dashboard

Thu, 27 May 2021 17:22:27 +0000

When looking at the query group dashboard, it is helpful to know what "state" the ASG was in when referencing the various dashboard metrics. Currently, I must do one of the following: 1) reference these metrics in the ASG monitoring tab (e.g., https://us-west-2.console.aws.amazon.com/ec2autoscaling/home?region=us-west-2#/details/my-query-group?view=monitoring), 2) create a CW dashboard of my own, 3) modify the existing Datomic CW dashboard to add this chart. The ASG metrics answer questions like: 1) how many instances was my query group running? 2) was my query group trying to scale out or in? 3) why did my query group not scale out or in?

I would find the following ASG metrics (in order of importance) useful additions to the current query group dashboard.

In Service Instances (Count)
Minimum Group Size (Count), Maximum Group Size (Count), and Desired Capacity (Count)
Pending Instances (Count)

Additional keys on anomaly part of the API?

Sun, 09 May 2021 01:33:17 +0000

Datomic sometimes attaches additional keys onto the anomaly in ex-data on the ExceptionInfos thrown. Is this additional data part of the API and able to be depended on? Further, could all keys returned be documented in the api docs? The returned keys are typically quite helpful in dispatching on different Datomic error "types."

Here's an example.

(def client (d/client {:server-type :dev-local
                       :storage-dir :mem
                       :system      "aa"}))
=> #'compute.http-api.public.task/client
(d/create-database client {:db-name "test"})
=> true
(def conn (d/connect client {:db-name "test"}))
=> #'compute.http-api.public.task/conn
(d/transact conn {:tx-data [{:db/ident       :user/id
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string
                             :db/unique      :db.unique/identity}
                            {:db/ident       :user/name
                             :db/cardinality :db.cardinality/one
                             :db/valueType   :db.type/string
                             :db/unique      :db.unique/identity}]})
=>
{:db-before #datomic.core.db.Db{:id "47c8f6f7-26f9-4575-97ec-7fc0948823c1",
                                :basisT 5,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "47c8f6f7-26f9-4575-97ec-7fc0948823c1",
                               :basisT 6,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533318 50 #inst"2021-05-09T01:32:25.677-00:00" 13194139533318 true]
           #datom[73 10 :user/id 13194139533318 true]
           #datom[73 41 35 13194139533318 true]
           #datom[73 40 23 13194139533318 true]
           #datom[73 42 38 13194139533318 true]
           #datom[74 10 :user/name 13194139533318 true]
           #datom[74 41 35 13194139533318 true]
           #datom[74 40 23 13194139533318 true]
           #datom[74 42 38 13194139533318 true]
           #datom[0 13 73 13194139533318 true]
           #datom[0 13 74 13194139533318 true]],
 :tempids {}}
(d/transact conn {:tx-data [{:db/id     [:user/id "a"]
                             :user/name "a"}]})
Execution error (ExceptionInfo) at datomic.core.error/raise (error.clj:55).
:db.error/not-an-entity Unable to resolve entity: [:user/id "a"] in datom [[:user/id "a"] :user/name "a"]
(ex-data *e)
=>
{:cognitect.anomalies/category :cognitect.anomalies/incorrect,
 :cognitect.anomalies/message "Unable to resolve entity: [:user/id \"a\"] in datom [[:user/id \"a\"] :user/name \"a\"]",
 :entity [:user/id "a"],
 :datom [[:user/id "a"] :user/name "a"],
 :db/error :db.error/not-an-entity}

Use Launch Template instead of Launch Configuration

Fri, 07 May 2021 23:38:51 +0000

The compute Datomic CloudFormation templates (primary and query groups) create ASGs using launch configurations. Launch configurations do not support the full set of Auto Scaling group features. For example, you cannot create an Auto Scaling group that launches both Spot and On-Demand Instances or that specifies multiple instance types or multiple launch templates. In fact, AWS recommends creating ASGs using launch templates, not launch configurations.

We are interested in launch templates for its support for using Spot and On-Demand instances.

EDIT: As of 2021-10-20, launch templates will not receive any new EC2 ASG features.

Reverse order browsing for raw indexes

Mon, 25 Jan 2021 11:39:25 +0000

Hello,
The index-pull function has the :reverse option to go through the index in reverse order.
Would it be possible to add it to the raw index browsing functions index-range and seek-datoms ?

Add default :timeout to client api ns docstring

Thu, 24 Dec 2020 00:12:44 +0000

It'd be great to have the default value for :timeout (60000) included in the datomic.client.api ns docstring. Currently this value can only be found in the docs on the website. As of client-cloud 0.8.102, the ns docstring specifies the default value for :limit.

Datomic Client API equivalent to on-prem tx->t, t->tx

Mon, 21 Dec 2020 00:25:17 +0000

The Datomic client API does not provider tx->t or t->tx. These functions can be useful for very specific scenarios (e.g., you have a tx id and want to know if the database has "moved forward").

This was also requested on the forums. A workaround was provided for the user's specific case but the request is still relevant.

Allow dev-local import-cloud :source and :dest to work with any :server-type

Sun, 20 Dec 2020 00:37:33 +0000

Currently import-cloud requires the :source client :server-type to be :cloud and the :dest :server-type to be :dev-local. It'd be great if this function allowed any :server-type to be passed for :source and :dest. A possible use case would be copying a dev-local database to a cloud system for testing in a staging environment.

Java 11+ Ion support

Tue, 03 Nov 2020 01:33:14 +0000

Datomic Ions currently run on Java 8. There are lots of great features in the modern versions of Java that we're not able to use (http client, new date methods). Since all our services run on Java 11, we must backport certain features just for Ions.

Provide way to interact with a datomic database before making the system available

Mon, 02 Nov 2020 15:45:42 +0000

Currently in Cloud there is no way to interact with a datomic database before making the system available for external requests.

Potential uses:

deploy the same code everywhere (dev & prod) and it does db-ensure
actions
load data and/or schema prior to the system being live
start components or other system-management tooling
provide (easy) way to integrate startup (e.g. mount/system-component)
provide status and ability to know when pieces are started so you can finish app initialization

Quote string values in result of Ions `{:op :deploy}`

Sun, 25 Oct 2020 21:13:20 +0000

The output of a Datomic Ions lambda deployment provides a command to check the deployment's status. For example:

{:execution-arn EXECUTION-ARN, :status-command "clojure -Adev -m datomic.ion.dev '{:op :deploy-status, :execution-arn EXECUTION-ARN}'", :doc "To check the status of your deployment, issue the :status-command."}

Calling read-string on the result chokes because EXECUTION-ARN is neither an escaped string or a symbol. Can we get an output similar to the result of :op :push where string values are quoted?

Bind Ions to IAM roles separate from their ambient compute role

Sun, 25 Oct 2020 19:58:49 +0000

Right now Ions run under the role of the underlying compute resources that they are deployed to. It would be nice to be able to give Ions a role independent of the ambient compute. This can be managed by the Ion runtime machinery by calling AssumeRole in the background, and supplying those creds to the Ions

This is important so that Ions don't have to run with the permissions of all the other ions running in the same compute.

Expose an alt-port transactor configuration property

Sun, 25 Oct 2020 19:45:14 +0000

The alt-host configuration property is great and helps in many scenarios, but not having an equivalent for the port (alt-port?) is a blocker in some other scenarios. It would be great to have such property, which would default to the same port as the transactor is listening.

`gcDeletedDBs` available in peer API

Sun, 25 Oct 2020 19:36:18 +0000

It would be very helpful to be able to call gcDeletedDBs programmatically from the API. Providing the felxibility of passing in a Datasource. Allows users to get GC capabilities in the app.

Allow for datomic cloud installs into pre-existing VPCs

Sun, 25 Oct 2020 19:29:15 +0000

The default Marketplace/CF config is great for getting started, more greenfield usage, etc. But it doesn't accomodate more complex (but common) situations. For instance, we've just completed a complete overhaul of our AWS infrastructure. The project had kind of shuffled into AWS. Started with one account, with lifecycle environments only segregated via security groups. After accepting that that was a VeryBadThing, went to VPC's per environment, with AWS account segregation (one for production and pre-production, another for all dev lifecycle environments), we've now moved to something a bit more sophisticated with accounts/vpcs per lifecycle environments, dedicated shared accounts/VPCs for ingress, data vault/logging, management, etc with IPSec tunnels between them all, as opposed to VPC peering.

Sorry about the excruciating detail, but just wanted to provide context. So, in this scenario, shoehorning in Datomic Cloud becomes a bit difficult. Especially in conjunction with the current app/system assumptions (bring on those multiple query groups please! :)). For instance, my plan was to give each of my devs their own sandbox Solo instance, this won't scale well (there are 40 of them ;)) of they each need their own VPC, never mind the other issues. And I guess that might be basically a separate but related request, where one could stand up multiple 'systems' in a single VPC

Store values encrypted, allowing key rotation.

Sun, 25 Oct 2020 19:14:02 +0000

In our application we have a requirement to store certain fields encrypted. This encryption is done in the application. However, we would like to be able to rotate the encryption key should it become compromised, and retain the historical query features that you would get with unencrypted storage. In order to rotate the key, we have to excise the data and re-transact it with the new key, or store the encrypted values outside of Datomic. It would be nice to have an encryption API that enabled this use case.

Support Multi-value parameters with Ions

Sun, 25 Oct 2020 18:59:52 +0000

API Gateway supports this via MultiValueHeaders. Consideration should be given to how best to support multi-value parameters in ions.

https://aws.amazon.com/blogs/compute/support-for-multi-value-parameters-in-amazon-api-gateway/

https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#apigateway-multivalue-headers-and-parameters

Related issues in Pedestal.ion, "Duplicate header support (applicable to the set-cookie header)."

Create DB from sequence of transactions

Sun, 25 Oct 2020 18:35:40 +0000

Given an input sequence of transactions expressed as EDN or some other format, provide a process to create a new Datomic database "offline" as quickly as possible, either in a supported storage backend or as a backup suitable for restore-db.

The input transactions will use real Entity IDs, not tempids. Provide public APIs for constructing Entity IDs correctly.

No other transactions or queries can be performed while the new database is being built. The input transactions cannot call any transaction functions.

Similarly, provide an "export" process to write out the transaction log of an existing database to a file in the same format.

This is essentially an optimization of the current process for bulk imports with a Transactor, with the added benefit of being able to control the creation of Entity IDs. This might be used, for example, to make a copy of a Datomic database with a filter or transformation applied to the transactions.

Datomic CLI should support AWS SSO

Sun, 25 Oct 2020 18:24:06 +0000

Datomic CLI should be able to load SSO profile from ~/.aws/config. It should be able to authorize Datomic users from AWS SSO console.

Allow for setting transactor properties using environment variables

Sun, 25 Oct 2020 18:08:09 +0000

Datomic should be able to read it's settings from environment variables, following a naming standard like what Spring Boot does. E.g.:

DATOMIC_LICENSE_KEY
DATOMIC_PROTOCOL
DATOMIC_PORT
…

Support AWS dynamo point in time backup in on-prem.

Sun, 25 Oct 2020 16:57:02 +0000

https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/

Currently Datomic supplies backup/restore capability here:

https://docs.datomic.com/on-prem/backup.html

Dynamo's PITR remains untested for use with Datomic DBs. Implementation of this feature would ensure that using Dynamo's PITR causes no issues on a Datomic DB.

Pass along DDB request IDs with transactions

Sun, 25 Oct 2020 16:41:05 +0000

Include the Dynamo DB request IDs with transactions.

In order to provide more information for troubleshooting, the DDB team has indicated that passing along the DDB request ID with transactions would enable further troubleshooting, down to the specific problematic transactions.

Add XRay Support

Sun, 25 Oct 2020 15:55:19 +0000

In Datomic Cloud, it would be nice to have XRay support enabled. I imagine this would entail having the daemon configured on the AMI, and perhaps optionally you guys' AWS service calls have the tracing wrappers enabled.

Delete all except the latest point in time within a backup URI

Thu, 22 Oct 2020 21:46:45 +0000

Also deleting a single point in time might be useful

Backup and Restore available via Peer API

Thu, 22 Oct 2020 18:14:38 +0000

Allow for on-prem backup and restore via the peer-api.

Provide integrity check on Datomic Backups

Thu, 22 Oct 2020 18:03:10 +0000

A tool to confirm the validity and completeness of a Datomic backup.

Custom Value Types

Thu, 22 Oct 2020 17:58:01 +0000

An attribute could accept a custom comparator and validator (as installed functions) and any fressian/edn-compatible value. We often have a complex type which datomic either forces us to serialize to a string (slow, hacky, space-inefficient), or encode into a long (hacky, tricky), or use a full entity (lost value semantics, need to declare more attributes). Custom value types would solve all this.

Add Cassandra Cluster callback to the Peer

Thu, 22 Oct 2020 17:52:42 +0000

The transactor provides a configurable cassandra cluster callback, but the peers do not. The Cassandra driver depends on old versions of metrics libraries that are incompatible with newer versions used in (for example) clojure metrics. Datastax solution for this requires use of the cluster callback https://docs.datastax.com/en/developer/java-driver/3.5/manual/metrics/#metrics-4-compatibility, but that isn't available in Peers.

Support java.time.instant

Thu, 22 Oct 2020 17:46:39 +0000

Support for other types from java.Time: Year, YearMonth, LocalTime etc.

Primary-Replica Database Replication

Thu, 22 Oct 2020 17:44:38 +0000

Provide a turnkey Primary-Replica feature for disaster recover. Alternatively provide API such that a "Replication Peer Instance" can: connect to a "primary" database simultaneously connect to a "replica" database (in a different data center) * read transactions from primary sequentially and write those transactions to the replica.

Ability to limit the number of results from query

Thu, 22 Oct 2020 17:42:57 +0000

Currently there is no built in way to limit the results from a query. This puts the task on the user to limit the results by first querying for entity ids, limiting that number and then pulling the data.

Per-db metrics

Thu, 22 Oct 2020 17:33:06 +0000

Add a "database" dimension to the CloudWatch metric to distinguish which Database is being reported on, including an "all databases" value.

Provide a progress bar/percentage on backup/restore process.

Thu, 22 Oct 2020 17:10:52 +0000

It would be nice to be able to see the percentage complete when the restoring progress is not clear.

Expose Time in Analytics

Thu, 22 Oct 2020 16:57:58 +0000

Expose t in analytics. To allow filtering by time-based predicates:
- as-of
- since

Ability to start peer without transactor.

Thu, 22 Oct 2020 16:48:41 +0000

If all transactors are down, no peers can start. Add the ability to start a peer in a read-only mode while continuing to try to connect to the transactor.

Client library for Python

Thu, 22 Oct 2020 16:45:51 +0000

Provide client implementation in Python

Support "EU-NORTH-1" (Stockholm) in Datomic Cloud

Thu, 22 Oct 2020 16:39:02 +0000

Provide ability to launch Cloud in EU-NORTH-1

Dockerize Datomic

Thu, 22 Oct 2020 16:37:12 +0000

Provide an official Docker image for Datomic on-prem

Migration tool to migrate a DB from on-prem to Cloud

Thu, 22 Oct 2020 16:34:33 +0000

Provide a path for moving an on-prem DB to the cloud

Client API: tx-report-queue equivalent

Thu, 22 Oct 2020 16:25:01 +0000

One of the features I miss a lot in the client API is TX report queues. Would it be possible to add something like this to the client? Perhaps tx-range could get an option that, when no :end is given, keeps the channel open and waits for future transactions?

Query explain for datalog

Thu, 22 Oct 2020 16:24:23 +0000

One of the places that we struggle with the most is understanding query performance. In general we get the idea that in order to write a fast query you need to reduce the search space as quickly as possible. As a result the most specific datalog clause should go first. However, we have found cases when that is not always true. For example when an attribute is a reference or when an attribute is indexed. Since we have little insight into how the query planner works its hard for us to tell what will make a query fast. We can and do optimize our queries by hand. But as our system grows and we add more queries this hand optimization becomes more burdensome. It would be great to have a tool, similar to a sql explain, that would give us the fastest ordering of the datalog clauses and provide suggestions for adding indexes.

Sorting and Pagination for Datomic Query results

Thu, 22 Oct 2020 16:08:51 +0000

Provide sorting and pagination for query results in the client and peer api's query.

S3 Storage Option

Thu, 22 Oct 2020 16:05:07 +0000

Useful for storing large amounts of data, which don't necessarily need low latency guarantees, but would want lower storage costs.

Support smaller compute instances in Datomic Cloud Production

Thu, 22 Oct 2020 16:04:44 +0000

I would love to use Ions (and perhaps Datomic itself a little as well) to run a load-light application. It needs to have HA so Solo is not an option. However Datomic Cloud only runs with i3.large or stronger, for about $90/m (w/o Datomic license) in Ireland times two (since we need at least two instances for HA). There is no way I can persuade the team to go from its two t2.small (for 2$13) to i3.large. I understand the need for i3. to have fast SSD but for application with low load and little data that is not really needed. But it would be awesome to support also some "underpowered" instances for lighter applications and/or apps that are primarily interested in leveraging Ions as a way to deploy and run code, without heavy need for a DB.

Export from console

Thu, 22 Oct 2020 16:01:20 +0000

It would be great if Datomic Console provided a way to export to CSV files

Expose standby/active transactor status in on-prem metrics.

Thu, 22 Oct 2020 16:00:30 +0000

Provide a method through an API to allow monitoring of standby/active status of transactors in on-prem.

Targeted excision

Thu, 22 Oct 2020 15:47:04 +0000

In order to comply with privacy regulations, I need a targeted ability to permanently excise particular attributes of an explicitly-identified entity.

Support backup to a database URI

Thu, 22 Oct 2020 15:40:41 +0000

Currently backup is only possible from a database URI to a file URI. Support backup from a database URI to another database URI. Maintain the incremental feature present in current backup/restore for this feature.

Cloud Backup and Recovery

Thu, 22 Oct 2020 15:36:26 +0000

This feature request is discussed at https://forum.datomic.com/t/cloud-backups-recovery/370
The motivation for wanting backups even in the presence of a reliable storage system is guarding against human error and data corruption bugs. Migrating the data to another AWS account might be another use case.

Readonly peer access

Thu, 22 Oct 2020 15:30:43 +0000

A peer that can read from the backing store, but not transact