Why does Datomic store all databases in a single datomic_kvs table instead of having a separate table per database?

Question

I am curious about the reasoning behind keeping all Datomic databases in a single datomic_kvs table rather than creating a separate table for each database.

Some disadvantages of the current approach that I see:

• More complex maintenance: For example, if VACUUM FULL is needed because one database is highly active and generates a lot of garbage, it will block all Datomic databases, not just the affected one.
• Difficult to determine per-database segment size: Since the table contains entries for all databases, simply counting elements in datomic_kvs does not provide a per-database segment count.
• Database deletion requires a special utility: Instead of simply dropping a table, removing a database requires a dedicated tool.
• Disk space is not immediately freed: Because of the previous point, space is not reclaimed until VACUUM FULL is run, which again locks all Datomic databases.

Given these limitations, what are the key advantages that justify using a single datomic_kvs table instead of separate tables per database?

Also, is there any plan to introduce a mode where each Datomic database has its own separate table?

Answer 1

Hi Nikolay,

Understood your desire for a separate KV table provisioned per DB.

For Datomic Pro we tend to recommend that large mission-critical database have their own dedicated transactor and peer processes. We discuss this in the documentation here https://docs.datomic.com/operation/capacity.html#multiple-databases because garbage collection pauses impact across all databases and other shared resources can impact one databases performance over another.

I do want to point out that you can currently move your databases to their own separately provisioned systems, separate tables and transactor pairs + peers using backup and restore. Backup one DB and move to a new table. https://docs.datomic.com/operation/backup.html

I have shared your feedback with the team, and we will consider features in this space.

Thanks,
Jaret