Commit graph

116 commits

Author SHA1 Message Date
martinsumner
d24b100aa6 Initial work on Journal Compaction
Largely untested work at this stage to allow for the Inker to request
the Inker's clerk to perform a single round of compact based on the best
run of files it can find.
2016-09-27 14:58:26 +01:00
martinsumner
e2bb09b873 Snapshot testing
Work to test the checking of sequence numbers in snapshots as required
by the inkers clerk to calculate the percentage of a file which is
compactable
2016-09-26 10:55:08 +01:00
martinsumner
c64d67d9fb Snapshot Work - Interim Commit
Some initial work to get snapshots going.

Changes required, as need to snapshot through the Bookie to ensure that
there is no race between extracting the Bookie's in-memory view and the
Penciller's view if a push_to_mem has occurred inbetween.

A lot still outstanding, especially around Inker snapshots, and handling
timeouts
2016-09-23 18:50:29 +01:00
martinsumner
4e28e4173c Rebar and eunit changes
Initial rebar compile - which exposed eunit tets failures associated
with changes to file structures and filename references
2016-09-19 18:50:11 +01:00
martinsumner
a1c970a66a Manifest ordering
Be more explicit about manifest ordering to stop keys being laoded in
incorrect order
2016-09-19 15:56:35 +01:00
martinsumner
7c28ffbd96 Further bookie test - CDB optimisation and Inker manifest correction
Additional bookie test revealed that the persisting/reading of inker
manifests was inconsistent and buggy.

Also, the CDB files were inffeciently writing the top index table -
needed to be improved as this is blokicng on a roll
2016-09-19 15:31:26 +01:00
martinsumner
b452fbe27c End-to-end test
Changes to ensure working of first end-to-end test (with a single Key
and Value)
2016-09-15 18:38:23 +01:00
martinsumner
e73a5bbf31 WIP - First draft of Bookie code
First draft of untested bookie code
2016-09-15 10:53:24 +01:00
martinsumner
86666b1cb6 Scan over CDB file
Make scanning over a CDB file generic rather than specific to read-in of
active nursery log - open to be called as an external function to
support other scanning behaviour.
2016-09-09 15:58:19 +01:00
martinsumner
edfe9e3bed Improved testing
Improve testing of Penciller to show startup and shutdown with push,
merging and fetch
2016-09-08 14:21:30 +01:00
martinsumner
0d905639be Testing of Inker rolling Journal
Add test to show inker rolling journal.  to achieve needs to make CDB
size an option, and also alter the manifest sorting so that
find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
martinsumner
f0e1c1d7ea Basic GET/PUT and rolling in Inker
Add support to roll file on PUT in the inker
2016-09-06 17:17:31 +01:00
martinsumner
f3a40e106d Inker improvements
Resolve issue in CDB file when we have cached the index.  Allow for
Inker to find keys in the active journal
2016-09-05 20:22:16 +01:00
martinsumner
2a76eb364e Inker - Initial Code
An attempt to get a first inker that can build a ledger from a manifest
as well as support simple get and put operations.  Basic tests surround
the building of manifests only at this stage - more work required for
get and put.
2016-09-05 15:01:23 +01:00
martinsumner
33f1efd576 Work on descriptions
Add further descriptions of roles following name changes.  Attempt to
simplify manifest management in the Penciller by assuming there is only
one Penciller's Clerk active - and so only one piece of work can be
ongoing
2016-08-02 17:51:43 +01:00
martinsumner
2bdb5fba6c Re-naming
Naming things is hard.  This change renames things based on the
Bookie/Inker/Penciller terminology
2016-08-02 13:44:48 +01:00