In giving an empty file a score of 0, a race condition was exposed. A file might not be active, but might still be rolling - and then cna get scored as 0, and immediately compacted. It will then be removed from the journal manifest.
Check each file is not rolling before making it a candidate for rolling.
Initial test included for running with recallc, and also transition from retain to recalc.
Moves all logic for startup fold into leveled_bookie - avoid the Inker requiring any direct knowledge about implementation of the Penciller.
Also:
Sort the output from an 'all' fetch one loop at a time
Make sure the test of scoring na empty file is scoring an empty file
If it is an emtpy file we want to compact the fragment away - in which case it should score 0.0 not 100.0
Change the penciller check so that it returns current/replaced/missing not just true/false.
Reduce unnecessary penciller checks for non-standard keys that will always be retained - and remove redunandt code.
Expand tests of retain and recover to make sure that compaction on delete is well covered.
Also move the SQN number laong during initial loads - to stop aggressive loop to find starting SQN every file.
When we want to be sure a file has been written before proceeding - we need a safer (that `file:write_file/2`) mechanism to be sure that it is written before proceeding.
This will:
open, write, sync, rename and then optionally read-back.
Changed so that manifest writing uses the safest form (including read back), and that sst writing uses a slightly looser form (with no read back to avoid performance issues).
Two reasons for logging this:
- to assist in sizing the ledger cache;
- to resolve the mystery when there appear to be no fetches from the penciller (as the penciller does not report fetches from the ledger cache)
Improve the speed of leveled_cdb tests by disabling sync on write.
Improve the strength of check of the correct behaviour when compacting with a reduced journal size.
Use hibernation rather than manual garbage_collect calls as per standard recommendation. Hibernate will be default gabage_collect anyway. Maybe help with SST files that naturally go quiet.
Plus typos from previous commit in leveled_cdb.
CDB files may be opened or rolled then left untouched for a period, so clean up any memory. Been awoken from hibernate has a cost, but it is a rare event.
Now that the SST files will fadvise on load (to force load into the page cache). The load should take place in reverse order, so that if th eledger is > page_cache, it is the higher levels that will end up in the cache at the expense of the lower levels.
Two helpers for memory management:
1 - a scan over the cdb file may lead to a lot of binary references being made. So force a GC fater the scan.
2 - the penciller files contain slots that will be frequently read - so advice the page cache to pre-load them on startup.
This is in response to unexpected memory mangement issues in a potentially non-conventional setup - where the erlang VM held a lot of memory (that could be GC'd , in preference to the page cache - and consequently disk I/O and request latency were higher than expected.
Extracting binary from within a binary leaves a reference to the whole of the original binary.
If there are a lot of very large objects received abck toback - this can explode the amount of memory the penciller appears to hold (and gc cannot resolve this).
To dereference from the larger binary, need to do a binary copy
When there is hevay PUT load, leveled_sst files could go into the delete-pending state befre the GC message is receieved - and the GC message would then interrupt the timeout cycle and lead ot the file not being GC'd until close.
