The datomic transactor memcached documentation says this:
When configured to used memcached, a Datomic process will automatically write into memcached any segment it needs that is not already present in memcached. In addition, the transactor (only) will write index and log segments to memcached as the segments are produced.
Based on this sentence and assuming a memcached memory pool larger than the storage size that is shared by transactor and peer, my expectation is the following:
- As the transactor is building an index, it writes newly-created index segments to storage and to memcached at roughly the same time.
- When the index is finished, it commits the new index root and informs all peers of the new index.
- Any peers using the same memcached cluster should, at the moment the index is released:
- have a decreasing ObjectCache hitrate because newly-indexed segments are not cached
- but an increasing memcached GET count
- and a flat StorageGet and MemcachedPut count because the segments are already in memcached
- For an overall flat or increasing memcached hitrate.
Instead, what I actually see is that when peers accept a new index,
- there's a dramatic decrease in Memcached hitrate
- and increase in MemcachedPut counts and Storage Get counts.
IndexWrite count reported by the transactor is very similar to the MemcachedPut count reported by the peer.
This behavior is consistent with the transactor not writing newly-indexed segments into memcached eagerly while indexing, which is contrary to the documentation.
So the question: Does the transactor really write index segments to memcached as it is indexing?