Does the transactor eagerly fill memcached with indexed segments?

Wed, 07 Apr 2021 15:21:18 +0000

The datomic transactor memcached documentation says this:

When configured to used memcached, a Datomic process will automatically write into memcached any segment it needs that is not already present in memcached. In addition, the transactor (only) will write index and log segments to memcached as the segments are produced.

(Emphasis mine.)

Based on this sentence and assuming a memcached memory pool larger than the storage size that is shared by transactor and peer, my expectation is the following:

As the transactor is building an index, it writes newly-created index segments to storage and to memcached at roughly the same time.
When the index is finished, it commits the new index root and informs all peers of the new index.
Any peers using the same memcached cluster should, at the moment the index is released:
- have a decreasing ObjectCache hitrate because newly-indexed segments are not cached
- but an increasing memcached GET count
- and a flat StorageGet and MemcachedPut count because the segments are already in memcached
- For an overall flat or increasing memcached hitrate.

Instead, what I actually see is that when peers accept a new index,

there's a dramatic decrease in Memcached hitrate
and increase in MemcachedPut counts and Storage Get counts.
The IndexWrite count reported by the transactor is very similar to the MemcachedPut count reported by the peer.

This behavior is consistent with the transactor not writing newly-indexed segments into memcached eagerly while indexing, which is contrary to the documentation.

So the question: Does the transactor really write index segments to memcached as it is indexing?

What does the memcached/item-too-large log warning mean?

Tue, 27 Oct 2020 17:54:45 +0000

I'm seeing a fair frequency of :memcached/item-too-large statements in my Datomic logs.

Is this a size limitation internal to Datomic? If so, could it mean our system is missing out on some cached segments?

A recent datomic.process-monitor log statement is showing:

{:tid 23,
 :CacheRepair {:lo 1, :count 637, :sum 637, :hi 1},
 :ObjectCacheCount 16634,
 :PeerAcceptNewMsec {:lo 0, :count 348, :sum 1, :hi 1},
 :MemcachedPutSucceededMsec {:lo 0, :count 602, :sum 1276, :hi 177},
 :AvailableMB 5550.0,
 :Memcache {:lo 0, :count 12175, :sum 11537, :hi 1},
 :StorageGetMsec {:lo 2, :count 637, :sum 10089, :hi 831},
 :MemcacheItemTooLarge {:lo 1, :count 4, :sum 4, :hi 1},
 :pid 18,
 :event :metrics,
 :ObjectCache {:lo 0, :count 702668, :sum 690336, :hi 1},
 :MetricsReport {:lo 1, :count 1, :sum 1, :hi 1},
 :PeerFulltextBatch {:lo 1, :count 332, :sum 348, :hi 6},
 :DbAddFulltextMsec {:lo 0, :count 21, :sum 30, :hi 12},
 :MemcachedPutFailedMsec {:lo 0, :count 31, :sum 541, :hi 52},
 :StorageGetBytes {:lo 122, :count 637, :sum 92206399, :hi 31294091}}

Which would seem to suggest a high hit rate judging from the :Memcache :sum / :count

Is this something I can fix or would cause issues?

Thanks!!

Datomic Knowledgebase - Recent questions tagged memcached

Does the transactor eagerly fill memcached with indexed segments?

What does the memcached/item-too-large log warning mean?