The CDB FSM process can be blocked by get_positions for all positions,
so loop around the index outside of the FSM process to allow for other
messages to interleave.
This at least checks the file is present, and the Key exists in the
index of that file. If the value is corrupt it will be removed by
compation, and then this will fail (unless the file is never compacted).
TODO: resolve issus of files which are corrupt - but never compacted
- a job for backup?
If the clerk updates the manifest - it might not recognise changes to
the manifest made since the clerk took the manifest. So the penciller
must merge its view of the snapshots back in to the updated manifest
fold objects which snaps in the fold was implemented incorrectly - it
took information from the LedgeCache at the point of the request, not
at the point of the fold. So the LedgerCache SQN may have been
surpassed in the Penciller by the time the fold was called.
riak_kv_sweeper gets the async fold function, then determines if the
function can be called. If the system is busy the fold may be queued,
and may never be acted upon.
This may cause issues with snapshot timeouts etc.
Increase this to 90 minutes. The first time all the snapshots are
rebuilt it may take a long time, but they all get scheduled together -
and queued until concurrency limits allow it to be completed.
currently the snapshot is made on initialisation, and only released
when completed (which may be after the queue). so the last couple of
snapshots were over-shooting the 1 hour.
Cmpaction is overly aggressive. It is a lot of work to compact a run of
files for just 20% reduction in disk space, when disk space for the
Journal (i.e. low IOPS disk space should be relatively inexpensive).
Require at least a 40% reduction for a compaction job.
Change to 5 blocks is intended to make the blocks in lookup slots
fractionally smaller, but more importantly to introduce a middle block
that cna be opened in a binary-split style fashion to reduce the number
of blocks that need to be opened for range queries. Worst case for
full slots is 3 blocks now not 4.
Still not clear if yielding is the cause of memory problems, but taking
it away universally has impacted throughput. At the very least we
should continue to yield on high-contention files (those at higher
levels), where the processes are more likely to be quickly terminated
anyway allowing GC to be invoked.
There was complicated and confusing code that achieved nothing for
effiency when trimming slots. the expensive part (binary_to_term) was
still needed on every block, and it was hard to get code coverage and
make sense of what it was really trying to achieve.
This is now much simpler - and may set us up for potential further
indexing help.
