Not within the fold fun of the leveled_runner.
This should avoid constantly having to re-merge and filter the penciller memory when running list_buckets and hitting inactive keys
In production scale testing, placing te check_modified call on get_kvrange not get_slots made the performance difference.
It should help in get_lots as well, but unable to reliably get coverage in tests with this. So for now, will leave off until a proper test can be constructed which demonstrates any benefits.
When scanning over a leveled store with a helper (e.g. segment filter and last modified date range), applying the filter will speed up the query when the block index cache is available to get_slots.
If it is not available, previously the leveled_sst did not then promote the cache after it had accessed the underlying blocks.
Now the code does this, and also when the cache has all been added, it extracts the largest last modified date so that sst files older than the passed in date can be immediately dismissed
Use the same function to decide for both scoring and compaction - and avoid the situation where somethig is scored for cmpaction, but doesnt change (which was the case previously with tombstones that were still in the ledger).
Move the average towards the current score if not scoring each run. Score from more keys to get a better score (as overheads of scoring are now better sorted by setting score_onein rather than by reducing the sample size).
Potentially reduce the overheads of scoring each file on every run.
The change also alters the default thresholds for compaction to favour longer runs (which will tend towards greater storage efficiency).
In giving an empty file a score of 0, a race condition was exposed. A file might not be active, but might still be rolling - and then cna get scored as 0, and immediately compacted. It will then be removed from the journal manifest.
Check each file is not rolling before making it a candidate for rolling.
Initial test included for running with recallc, and also transition from retain to recalc.
Moves all logic for startup fold into leveled_bookie - avoid the Inker requiring any direct knowledge about implementation of the Penciller.
