Datomic Knowledgebase - Recent questions tagged problem

Pulling :db/id will pull the entire entity

Thu, 10 Feb 2022 16:48:24 +0000

Environment

com.datomic/client-cloud 1.0.119
com.datomic/dev-local 1.0.242

History

Reproducible in both dev-local and Cloud client implementations.

(require '[datomic.client.api :as d])

(def client (d/client {:server-type :dev-local
                       :system      "test"
                       :storage-dir :mem}))

(d/create-database client {:db-name "bug-example"})

(def conn (d/connect client {:db-name "bug-example"}))

(d/transact conn {:tx-data [{:db/ident       :name
                             :db/valueType   :db.type/string
                             :db/cardinality :db.cardinality/one}]})

(def tx-report
  (d/transact conn {:tx-data [{:db/id "a"
                               :name  "kenny"}]}))

(def db-id (get-in tx-report [:tempids "a"]))

(d/pull (d/db conn)
  [:db/id]
  db-id)
=> {:db/id 83562883711050, :name "kenny"}

Expectation

Issuing d/pull with a selection of [:db/id] will only pull the :db/id attribute.

Actual

Issuing d/pull with a selection of [:db/id] will only pull all attributes on the given entity.

Evidence

See repro in History.

Impact

This bit me when doing some debugging in the REPL. I was trying to just get the db/id of a "top-level" entity, and it pulled the whole thing. This top-level entity is massive and resulted in me needing to restart my REPL. The ability to pull the entirety of an entity is a bit frightening, especially if such a thing were to happen (on purpose or accident) in production. The use of a [*] pull pattern is almost always avoided for this reason. Tracking down a bug like this could be quite tricky since it looks so innocent by appearance.

Long blocking on event `:kv-cluster/get-val` without timeout/retry/failover semantics?

Wed, 23 Jun 2021 09:19:28 +0000

On a test environment, we're seeing this kind of log from the Datomic Peer:

;; ~5 hours CI inactive (if this is relevant)

DEBUG 2021-06-18T13:32:37.702 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :phase :begin, :pid 7, :tid 225}

;; 15 minutes pass....

DEBUG 2021-06-18T13:48:25.091 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :msec 947000.0, :phase :end, :pid 7, :tid 225}

;; This next one finishes in 21ms:

DEBUG 2021-06-18T13:48:25.094 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :phase :begin, :pid 7, :tid 225}
DEBUG 2021-06-18T13:48:25.115 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :msec 21.0, :phase :end, :pid 7, :tid 225}

;; and now we start processing the transacts that queued up when the tests started...

INFO  2021-06-18T13:48:25.117 [manifold-execute-43]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :start, :pid 7, :tid 282}
DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :begin, :pid 7, :tid 226}
DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :msec 0.462, :phase :end, :pid 7, :tid 226}
INFO  2021-06-18T13:48:25.235 [clojure-agent-send-off-pool-24]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :end, :pid 7, :tid 226}

After those 15 minutes of waiting, the peer moved on and transacted all the transactions that happened to queue in the meantime, as if nothing happened. There do not seem to be any interesting or anomalous logs in the Transactor for this time period.

Datomic Peer (1.0.6269) with PostgreSQL storage. The peer and transactor are both running on Kubernetes, but Postgres is hosted outside of the k8s cluster.

I think my followup questions are:

Any idea what could be going on with the :kv-cluster/get-val and how to go about debugging this further?
Is there some timeout we can configure to avoid this kind of situation in a production environment?
Is the probable culprit Kubernetes? If so, has someone seen or fixed similar behavior before?
These logs are happening at the DEBUG level, otherwise we would not even know that the Peer was blocking/hanging/waiting. Are there other metrics or logs we should be monitoring to identify this kind of problem?

Dates coming through incorrectly in Datomic Analytics

Tue, 01 Jun 2021 04:58:00 +0000

We are seeing some :db.type/instant values in Datomic come through as completely different dates in Datomic Analytics.

For instance

#inst "2020-01-01T00:00:00.000-00:00"

Shows up in Presto as

1970-01-19 06:17:16.800

Which suggests that at some point a unix timestamp is being misinterpreted.

(java.util.Date. (/ (.getTime #inst "2020-01-01T00:00:00.000-00:00") 1000))
;; => #inst "1970-01-19T06:17:16.800-00:00"

Example entity:

user=> (d/q '[:find (pull ?e [*]) :where [?e :accounting.fiscal-year/start-date ?y]] db)
[[{:db/id 17592186045894, :db/ensure [#:db{:id 17592186045471}], :accounting.fiscal-year/start-date #inst "2020-01-01T00:00:00.000-00:00", :accounting.fiscal-year/end-date #inst "2020-12-31T00:00:00.000-00:00", :accounting.fiscal-year/closed? false}]]

Field definition:

user=> (d/q '[:find (pull ?e [* {:db/valueType [:db/ident]}]) :where [?e :db/ident :accounting.fiscal-year/start-date]] db)
[[{:schema/boundary #:db{:id 17592186045443}, :db/unique #:db{:id 38}, :db/valueType #:db{:ident :db.type/instant}, :db.attr/preds [relvn.accounting.dbfn/date-only?], :schema/label "start date", :db/cardinality #:db{:id 35}, :db/doc "The starting date of a fiscal year.", :db/id 154, :schema/ident-type #:db{:id 17592186045420}, :db/ident :accounting.fiscal-year/start-date, :db/ensure [#:db{:id 17592186045424}], :schema/entity #:db{:id 17592186045470}}]]

In Presto:

presto:example_tenant__demo_company_14052021> select * from fiscal_year;
closing_description |       start_date        |        end_date         | closed_ |     db__id     
---------------------+-------------------------+-------------------------+---------+----------------
 NULL                | 1970-01-19 06:17:16.800 | 1970-01-19 15:02:52.800 | false   | 17592186045894

Does the transactor eagerly fill memcached with indexed segments?

Wed, 07 Apr 2021 15:21:18 +0000

The datomic transactor memcached documentation says this:

When configured to used memcached, a Datomic process will automatically write into memcached any segment it needs that is not already present in memcached. In addition, the transactor (only) will write index and log segments to memcached as the segments are produced.

(Emphasis mine.)

Based on this sentence and assuming a memcached memory pool larger than the storage size that is shared by transactor and peer, my expectation is the following:

As the transactor is building an index, it writes newly-created index segments to storage and to memcached at roughly the same time.
When the index is finished, it commits the new index root and informs all peers of the new index.
Any peers using the same memcached cluster should, at the moment the index is released:
- have a decreasing ObjectCache hitrate because newly-indexed segments are not cached
- but an increasing memcached GET count
- and a flat StorageGet and MemcachedPut count because the segments are already in memcached
- For an overall flat or increasing memcached hitrate.

Instead, what I actually see is that when peers accept a new index,

there's a dramatic decrease in Memcached hitrate
and increase in MemcachedPut counts and Storage Get counts.
The IndexWrite count reported by the transactor is very similar to the MemcachedPut count reported by the peer.

This behavior is consistent with the transactor not writing newly-indexed segments into memcached eagerly while indexing, which is contrary to the documentation.

So the question: Does the transactor really write index segments to memcached as it is indexing?

Why is my float attribute returning double values in queries?

Thu, 17 Dec 2020 04:25:11 +0000

I have an attribute of type :db.type/float in my Datomic Cloud (solo) instance. When I query for that value, however, I get values of type java.lang.Double. (I actually have 6 float attributes and they all behave this way.)

Is this a bug? I can't see anything in the documentation that describes this behavior. In fact, the schema reference seems to specifically say that it should be float value and not a double.

This is not a big problem, but it is an annoyance because the values I'm transacting as floats are getting extra (noisy) precision added to them, so that e.g. when the user enters a 3.7, it actually shows up as 3.700000047683716. This isn't difficult to fix by casting to float, but it was surprising.

I only see this behavior in Datomic Cloud, by the way. Using dev-local correctly shows the value as a float.

Here's some relevant exchanges from a REPL session demonstrating the issue:

user> (d/q '[:find ?attr-type ?value :where
             [71587003061240196 :student/gpa-weighted-9 ?value]
             [:student/gpa-weighted-9 :db/valueType ?t]
             [?t :db/ident ?attr-type]]
           (d/db conn))
[[:db.type/float 3.700000047683716]]
user> (type (-> *1 first second))
java.lang.Double
user> (float 3.700000047683716)
3.7