The object tag "o" which was taken from eleveldb has been an extended to
allow for specific functions to be triggered for different object types,
in particular when extracting metadata for stroing in the Ledger.
There is now a riak tag (o_rkv@v1), and in theory other tags can be
added and used, as long as their is an appropriate set of functions in
the leveled_codec.
An attempt to refactor out more complex code.
The Penciller clerk and Penciller have been re-shaped so that there
relationship is much simpler, and also to make sure that they shut down
much more neatly when the clerk is busy to avoid crashdumps in ct tests.
The CDB now has a binary_mode - so that we don't do binary_to_term twice
... although this may have made things slower ??!!? Perhaps the
is_binary check now required on read is an overhead. Perhaps it is some
other mystery.
There is now a more effiicient fetching of the size on pcl_load now as
well.
This exposed another off-by-one error on startup.
This commit also includes an unsafe change to reply early from a rolling
CDB file (with lots of objects writing the hash table can take too
long). This is bad, but will be resolved through a refactor of the
manifest writing: essentially we deferred writing of the manifest
update which was an unnecessary performance optimisation. If instead we
wait on this, the process is made substantially simpler, and it is safer
to perform the roll of the complete CDB journal asynchronously. If the
manifest update takes too long, an append-only log may be used instead.
Add some initial system tests. This highlighted issues:
- That files deleted by compaction would be left orphaned and not close,
and would not in fact delete (now deleted by closure only)
- There was an issue on stratup that the first few keys in each journal
would not be re-loaded into the ledger
Largely untested work at this stage to allow for the Inker to request
the Inker's clerk to perform a single round of compact based on the best
run of files it can find.
Some initial work to get snapshots going.
Changes required, as need to snapshot through the Bookie to ensure that
there is no race between extracting the Bookie's in-memory view and the
Penciller's view if a push_to_mem has occurred inbetween.
A lot still outstanding, especially around Inker snapshots, and handling
timeouts
Two aspects of pushing to the penciller have been refactored:
1 - Allow the penciller to respond before the ETS table has been updated
to unlock the Bookie sooner.
2 - Change the way the copy of the memtable is stored to work more
effectively with snapshots wihtout locking the Penciller any further on
a snapshot or push request
Add test to show inker rolling journal. to achieve needs to make CDB
size an option, and also alter the manifest sorting so that
find_in_manifest actually works!
Standardise on record definitions between modules to make easier - then
add functionality to pushing to penciller as bookie would do. Some
initial manual testing of this seems OK.
