Commit graph

1085 commits

Author SHA1 Message Date
martinsumner
86666b1cb6 Scan over CDB file
Make scanning over a CDB file generic rather than specific to read-in of
active nursery log - open to be called as an external function to
support other scanning behaviour.
2016-09-09 15:58:19 +01:00
martinsumner
edfe9e3bed Improved testing
Improve testing of Penciller to show startup and shutdown with push,
merging and fetch
2016-09-08 14:21:30 +01:00
martinsumner
0d905639be Testing of Inker rolling Journal
Add test to show inker rolling journal.  to achieve needs to make CDB
size an option, and also alter the manifest sorting so that
find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
martinsumner
f0e1c1d7ea Basic GET/PUT and rolling in Inker
Add support to roll file on PUT in the inker
2016-09-06 17:17:31 +01:00
martinsumner
f3a40e106d Inker improvements
Resolve issue in CDB file when we have cached the index.  Allow for
Inker to find keys in the active journal
2016-09-05 20:22:16 +01:00
martinsumner
2a76eb364e Inker - Initial Code
An attempt to get a first inker that can build a ledger from a manifest
as well as support simple get and put operations.  Basic tests surround
the building of manifests only at this stage - more work required for
get and put.
2016-09-05 15:01:23 +01:00
martinsumner
ce0c55a2ec Resolve issue of Remainders
Two issues looked at
- There shouldn't be a remainder after writing the L0 file, as this
could have overlapping sequence numbers which will be missed on restart
- There should be a safety-check to stop the Clerk from doing a fake
push too soon after a background L0 file ahs been written (as the fake
push would lock the ledger waiting for the L0 file write to finish)
2016-08-16 12:45:48 +01:00
martinsumner
4586e2340a Startup and Shutdown Support
Added support for startup and shutdown of a Ledger.  As aprt of this
will now start tracking the highest sequence number.  This also adds a
safety check on pcl_pushmem to make sure that only keys with a higher
sequenc enumber are being pushed in - and hence we can happily insert
into the in-memory view without checking the sequence number.
2016-08-15 16:43:39 +01:00
martinsumner
6e56b569b8 Auto-merge
Allow for the clerk to merge continuously is no activity for the
penciller to prompt.

The penciller now must also correctly lock the manifest - to stop races
between the creation of ne wL0 files and the completion of work by the
clerk
2016-08-12 01:05:59 +01:00
martinsumner
c269eb3c52 Initial level merging
Some basic merging in the lsm tree.
2016-08-10 13:02:08 +01:00
martinsumner
718425633a Penciller accepting push
Standardise on record definitions between modules to make easier - then
add functionality to pushing to penciller as bookie would do.  Some
initial manual testing of this seems OK.
2016-08-09 16:09:29 +01:00
martinsumner
75996b90ca Remove file
Not needed yet
2016-08-02 17:54:13 +01:00
martinsumner
33f1efd576 Work on descriptions
Add further descriptions of roles following name changes.  Attempt to
simplify manifest management in the Penciller by assuming there is only
one Penciller's Clerk active - and so only one piece of work can be
ongoing
2016-08-02 17:51:43 +01:00
martinsumner
2bdb5fba6c Re-naming
Naming things is hard.  This change renames things based on the
Bookie/Inker/Penciller terminology
2016-08-02 13:44:48 +01:00
martinsumner
04da891272 Tidy up removing old files 2016-07-29 17:52:44 +01:00
martinsumner
b5db1b4e14 CDB to gen_server
First draft to make CDB a gen_Server
2016-07-29 17:48:11 +01:00
martinsumner
28f612426a Reformat of CDB
CDB was failing tests (was it always this way?).  There has been a
little bit of a patch-up of the test, but there are still some
potentially outstanding issues with scanning over a file when attempting
to read beyond the end of the file.

Tabbing reformatting and general tidy.

Concierge documentation development ongoing.
2016-07-29 17:19:30 +01:00
martinsumner
c1f6a042d9 Renaming
Delete some old working files and adopt a new naming convention.  The
keymanager actor has now been replaced by a concierge, to reflect that
this management role is performed at the front of house
2016-07-28 17:22:50 +01:00
martinsumner
a2d873a06d Add first draft of manager
Start to build up functions for the keymanager
2016-07-27 18:03:44 +01:00
martinsumner
a07ea27dd8 Add merging of files
Add first function of worker - the ability to merge multiple files
together from different levels
2016-07-22 16:57:28 +01:00
martinsumner
392830c839 Improve test clean-up
Remove file created in a test
2016-07-12 19:44:16 +01:00
martinsumner
45c10613e7 Iterator support added
Initial support for iterators
2016-07-12 19:42:50 +01:00
martinsumner
9dae893958 Read-only opening
Completing the file also closes it and leads to a read-only opening of
the file.
2016-07-07 11:43:26 +01:00
martinsumner
3b954aea43 Some refinements
Minor amendments to formatting and outputs
2016-07-06 18:09:40 +01:00
martinsumner
71a6538288 Improved testing of file creation
Rsolved some off-by-one errors, and ability to support KeyLists larger
than the keys supported in a file
2016-07-06 16:09:08 +01:00
martinsumner
27dc026176 Write a SFT File
With some initial test support
2016-07-06 10:52:47 +01:00
martinsumner
eedc296314 Completing KeyLists on a block boundary
Handle when writing a block empties the Key Lists but the block is full
- don't go-on and create a second empty block
2016-06-17 15:14:25 +01:00
martinsumner
cc16f90c9c SFT file continued
Writing of a slot
2016-06-10 19:09:55 +01:00
martinsumner
a95d77607e Initial work on sft files
Working on the delta-encoded segment filter, plus some initial
performance testing.
2016-05-31 17:21:14 +01:00
Martin Sumner
b7ae91fb71 Write initial block to bst file - part 2 2015-07-02 14:22:45 +01:00
Martin Sumner
46de8ad6a2 Write initial block to bst file 2015-06-10 08:14:37 +01:00
Martin Sumner
c5f50c613d Ongoing improvements - in particular CDB now supports general erlang terms not just lists 2015-06-04 21:15:31 +01:00
Martin Sumner
647a7f44dc Tidy-up initial files and add testing to optimise bst bloom filters 2015-05-31 23:31:31 +01:00
Martin Sumner
b09246ef04 Removing test files and binaries 2015-05-25 23:45:35 +01:00
Martin Sumner
e2099d0c14 Initial files proving concepts
WIP - nothing currently workable
2015-05-25 22:45:45 +01:00