Some initial work to get snapshots going.
Changes required, as need to snapshot through the Bookie to ensure that
there is no race between extracting the Bookie's in-memory view and the
Penciller's view if a push_to_mem has occurred inbetween.
A lot still outstanding, especially around Inker snapshots, and handling
timeouts
Two aspects of pushing to the penciller have been refactored:
1 - Allow the penciller to respond before the ETS table has been updated
to unlock the Bookie sooner.
2 - Change the way the copy of the memtable is stored to work more
effectively with snapshots wihtout locking the Penciller any further on
a snapshot or push request
Makes the ability to get positions and the fetch directly by position
more generic - supporting the fetch of different flavours of
combinations, and requesting a sample of positions not just all
CDB did many "bitty" reads/writes when scanning or writing hash tables -
change these to bult reads and writes to speed up.
CDB also added capabilities to fetch positions and get keys by position
to help with iclerk role.
Additional bookie test revealed that the persisting/reading of inker
manifests was inconsistent and buggy.
Also, the CDB files were inffeciently writing the top index table -
needed to be improved as this is blokicng on a roll
Make scanning over a CDB file generic rather than specific to read-in of
active nursery log - open to be called as an external function to
support other scanning behaviour.
Add test to show inker rolling journal. to achieve needs to make CDB
size an option, and also alter the manifest sorting so that
find_in_manifest actually works!
An attempt to get a first inker that can build a ledger from a manifest
as well as support simple get and put operations. Basic tests surround
the building of manifests only at this stage - more work required for
get and put.
Two issues looked at
- There shouldn't be a remainder after writing the L0 file, as this
could have overlapping sequence numbers which will be missed on restart
- There should be a safety-check to stop the Clerk from doing a fake
push too soon after a background L0 file ahs been written (as the fake
push would lock the ledger waiting for the L0 file write to finish)
Added support for startup and shutdown of a Ledger. As aprt of this
will now start tracking the highest sequence number. This also adds a
safety check on pcl_pushmem to make sure that only keys with a higher
sequenc enumber are being pushed in - and hence we can happily insert
into the in-memory view without checking the sequence number.
Allow for the clerk to merge continuously is no activity for the
penciller to prompt.
The penciller now must also correctly lock the manifest - to stop races
between the creation of ne wL0 files and the completion of work by the
clerk
Standardise on record definitions between modules to make easier - then
add functionality to pushing to penciller as bookie would do. Some
initial manual testing of this seems OK.
Add further descriptions of roles following name changes. Attempt to
simplify manifest management in the Penciller by assuming there is only
one Penciller's Clerk active - and so only one piece of work can be
ongoing
CDB was failing tests (was it always this way?). There has been a
little bit of a patch-up of the test, but there are still some
potentially outstanding issues with scanning over a file when attempting
to read beyond the end of the file.
Tabbing reformatting and general tidy.
Concierge documentation development ongoing.
Delete some old working files and adopt a new naming convention. The
keymanager actor has now been replaced by a concierge, to reflect that
this management role is performed at the front of house
