Just some initial WIP code for this. Will revisit this again after
exploring some ideas as to how to reduce the cost of the
get_keys_by_segment.
The overlal idea is that there are trees of recent modifications, with
recent being some rolling time window made up of hourly blocks, and
recency being dtermined by the last-modified date on the object metadata
- which should be conistent across a cluster.
So if we were at 15:30 we would get the tree for 14:00 - 15:00 and the
tree for 15:00-16:00 from two different queries which cover the same
partitions and then compare.
Comparison may find differences, and we know what segment the difference
is in - but how to then find all keys in that segment which have been
modified in the period? Three ways:
Do it inefficeintly and infrequently using a fold_keys and a filter
(perhaps with SST files having a highest LMD in the metadata so that
they can be skipped).
Add a special index, where verye entry has a TTL, and the Key is
{$segment, Segment, Bucket, Key} so that a normal 2i query cna be used.
Align hashing for segments with hashing for penciller lookup so that a
query over the actual keys cna be optimised skipping chunks of the
in-memory part, and chunks of the SST file
Allow tictac tree sizes to be flexible.
Tested lots of different sizes. Having both level 1 and level 2 the
same size seemed to be consistently quicker than trying to make either
of the levels relatively wider.
There's an 8% performance improvement if the SegmentCount is reduced by
a quarter.
Naming is now confusing now we have TicTac Trees. This query builds a
list of keys and hashes not a tree - so it was misleading anyaway. Now
renamed hashlist_query.
Need to allow specific settings to be passed into unit tests.
Also, too much journal compaction may lead to intermittent failures on
the basic_SUITE space_clear_on_delete test. think this is because
there are less “deletes” to reload in on startup to trigger the cascade
down and clear up?
The CDB FSM process can be blocked by get_positions for all positions,
so loop around the index outside of the FSM process to allow for other
messages to interleave.
This at least checks the file is present, and the Key exists in the
index of that file. If the value is corrupt it will be removed by
compation, and then this will fail (unless the file is never compacted).
TODO: resolve issus of files which are corrupt - but never compacted
- a job for backup?
If the clerk updates the manifest - it might not recognise changes to
the manifest made since the clerk took the manifest. So the penciller
must merge its view of the snapshots back in to the updated manifest
fold objects which snaps in the fold was implemented incorrectly - it
took information from the LedgeCache at the point of the request, not
at the point of the fold. So the LedgerCache SQN may have been
surpassed in the Penciller by the time the fold was called.
riak_kv_sweeper gets the async fold function, then determines if the
function can be called. If the system is busy the fold may be queued,
and may never be acted upon.
This may cause issues with snapshot timeouts etc.
Increase this to 90 minutes. The first time all the snapshots are
rebuilt it may take a long time, but they all get scheduled together -
and queued until concurrency limits allow it to be completed.
currently the snapshot is made on initialisation, and only released
when completed (which may be after the queue). so the last couple of
snapshots were over-shooting the 1 hour.
