Regression in latest datomic when transacting data & schema in one tx

Question

Let's say you have a transaction like:

{:db/id :admin/id,
 :db/unique :db.unique/identity
 :db.install/_attribute :db.part/db}
{:db/id "admin"
  :db/ident :admin/bug
  :admin/id admin-id}

(So both schema & data in one tx)

Normally you would expect idents to work properly - e.g. after that transaction (d/entity db [:admin/id admin-id]) would return the :admin/bug entity. Unfortunately that's not the case (though it used to work in previous versions of datomic) - it returns nil. Splitting the transaction into two makes everything work.

Here's a full repro:

(require '[datomic.api :as d])
(def conn (d/connect (doto (str "datomic:mem://" (d/squuid)) (d/create-database))))
@(d/transact conn [{:db/id "new"
                    :db/ident :admin/id,
                    :db/valueType :db.type/uuid,
                    :db/cardinality :db.cardinality/one,
                    :db/doc "Id of the admin"
                    :db.install/_attribute :db.part/db}])

(def admin-id (d/squuid))

@(d/transact conn [{:db/id :admin/id,
                    :db/unique :db.unique/identity
                    :db.install/_attribute :db.part/db}
                   {:db/id "admin"
                    :db/ident :admin/bug
                    :admin/id admin-id}])

(def db (d/db conn))

(if (some? (d/entity db [:admin/id admin-id]))
  (println "All OK - admin is found by identity")
  (println "Admin identity not there? Let's see the entity: " (d/touch (d/entity db :admin/bug))))

Answer 1

@Mike

You cannot create and use an attribute in the same transaction. You have to separate them. Without installed schema resulting from a transaction, datomic cannot perform tasks like uniqueness checks on subsequent transactions or be aware of the existence of the attribute.

It would be great if you could recover what version of Datomic this ever "worked" on so I could understand the circumstances. The only thing I can think of is this appeared to work perhaps on a mem database?

Thanks,
Jaret

Comments

Yep fully understand that this shouldn't work, but it really shouldn't corrupt the database. Currently it doesn't throw an exception and corrupts the database instead. This previously worked on datomic-pro-0.9.5697 .

For context why this is something that might happen: this is an output of a schema migration compaction (i.e. concat'ing schema migrations together to minimize the number of init transactions for test runs).

---

You are saying "corrupt" the database.  I am not sure what you mean?  Can you share your full gist and expected outcome on each version of Datomic?

---

I am running through the repro you gave me.  I will create a gist and post it here.

---

Hi Mike,

Thanks for your report! After reviewing the behavior, I don't believe this is a regression, but rather an existing bug **in the mem protocol** that has been present for some time. The core issue is that an attribute must already have an AVET index before it can be made unique, as outlined in our documentation:

Schema Change - Changing :db/unique (https://docs.datomic.com/schema/schema-change.html#changing-db-unique)

>In Datomic Pro, Datomic must already be maintaining an AVET index on the attribute.

The fact that you were able to add :db/unique to an attribute without an AVET index in the mem protocol is indeed a bug, and we will be addressing it in an upcoming release.

A few key takeaways:

- I created a gist using your code snippet that demonstrates this behavior across different versions.  (https://gist.github.com/Jaretbinford/c3640e3bdf7606aa9eb54bd1b0608f7b)
- The behavior was consistent for mem but correctly enforced in dev and all durable storages.
- In the upcoming release, the correct behavior will be to reject the transaction that attempts to make an attribute unique without first maintaining an index, rather than allowing that transaction.

I would be very interested if you have ever seen this not throw in any durable storage (posgres, sql, dev, cass etc)?  To my testing it is not possible in those storages.  As you will see in the aforementioned gist at the bottom, this should be the correct behavior in `mem`:

```
;@(d/transact conn [{:db/id :admin/id,
;                    :db/unique :db.unique/identity}
;                   {:db/id "admin"
;                    :db/ident :admin/bug
;                    :admin/id admin-id}])
;Execution error (Exceptions$IllegalArgumentExceptionInfo) at datomic.error/arg (error.clj:79).
;:db.error/invalid-alter-attribute Error: {:db/error :db.error/unique-without-index, :attribute :admin/id}
```


Let me know if you have any further questions!  I will circle back here once we have released a fix.