umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1328 commits 1 branch 0 tags 25 MiB
Commit graph

62 commits

Author SHA1 Message Date
Martin Sumner
082eabb65b Switch to start_link 
Start all processes linked - to collapse the whole tree if one process fails
 2018-06-28 12:16:43 +01:00
Martin Sumner
f88f511df3 leveled_codec spec/doc 
Try and make this code a little bit more organised andd easier to follow
 2018-05-03 17:18:13 +01:00
Martin Sumner
c2f19d8825 Switch to using bloom at penciller 
Previouslythe tinybloom was used within the SST file as an extra check to remove false fetches.

However the SST already has a low FPR check in the slot_index.  If the newebloom was used (which is no longer per slot, but per sst), this can be shared with the penciller and then the penciller could use it and avoid the message pass.

the message pass may be blocked by a 2i query or a slot fetch request for a merge.  So this should make performance within the Penciller snappier.

This is as a result of taking sst_timings within a volume test - where there was an average of + 100microsecs for each level that was dropped down.  Given the bloom/slot checks were < 20 microsecs - there seems to be some further delay.

The bloom is a binary of > 64 bytes - so passing it around should not require a copy.
 2017-11-28 01:19:30 +00:00
Martin Sumner
f55cbbeac3 OTP 19 requires defaults in dialyzer 2017-11-13 14:02:39 +00:00
Martin Sumner
61b7be5039 Make compression algorithm an option 
Compression can be switched between LZ4 and zlib (native).

The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
 2017-11-06 15:54:58 +00:00
Martin Sumner
a128dcdadf Change hash algorithm for penciller 
Switch from magic hash to md5 - to hopefully remove the need for some
of the artificial jumps required to get expected fall positive ratios.

Also split the hash into two 16-bit integers.  We assume that SegmentID
(from the perspective of AAE merkle/tictac trees) will always be at
least 16 bits.  the idea is that hashes should be used in blooms and
indexes such that some advantage can be gained from just knowing the
segmentID - in particular when folding over all the keys in a bucket.

Performance testing has been difficult so far - I think due to “cloud”
mysteries.
 2017-10-20 23:04:29 +01:00
Heinz N. Gies
25389893cf Add compatibility for old and new random / rand functions 2017-08-01 11:24:12 +02:00
Heinz N. Gies
44fd603474 Cleanup dialyzer errrors in leveled_pclerk 2017-07-31 19:41:26 +02:00
martinsumner
8b3ca78d49 spec help for SST file 2017-05-18 12:29:56 +01:00
martinsumner
bfcf981485 Correct root path setting in pclerk 2017-03-09 21:32:36 +00:00
martinsumner
4c59342600 Change SST reference to split filename 
The manifest and the logs are bloated by having the full file path for
every filename in there - given the root path is constant.

Could also cause issues if the mount point is ever changed.
 2017-03-09 21:23:09 +00:00
martinsumner
8077c70486 More conservative approach to ongoing work monitoring 
As per comments though - if we auto-restart pclerk in the future this
will have to be re-considered.

Perhaps a re-starting pclerk should force some reset of this boolean on
startup perhaps by making a different work_for_clerk if in a virgin
state.
 2017-02-09 23:49:12 +00:00
martinsumner
05b2d8faaf is_empty not in OTP16 
bloody OTP16 strikes again - is_empty isn't there
 2017-02-09 21:12:01 +00:00
martinsumner
adb78f5c5a Resolve pclerk crash 
Need to add extra logging to understand why pclerk crashes in some
volume tests.

Penciller's Clerk <0.813.0> shutdown now complete for reason
{badarg,[{dict,fetch,[63,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,[100,105,99,116,46,101,114,108]},{line,126}]},{leveled_pclerk,handle_cast,2,[{file,[115,114,99,47,108,101,118,101,108,101,100,95,112,99,108,101,114,107,46,101,114,108]},{line,96}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,604}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,239}]}]}

Should only be prompted after prompt deletions had bene updated.
Perhaps a race whereby somehow it is prompted again after it is emptied.
 2017-02-09 14:33:39 +00:00
martinsumner
5105df1cd6 Add replace capability to manifest 2017-01-23 11:02:54 +00:00
martinsumner
59c0c06b60 OTP16 type again 2017-01-17 19:01:59 +00:00
martinsumner
7086717765 Rename to pmanifest 
There may be a seperate imanifest in the future
 2017-01-17 14:11:50 +00:00
martinsumner
ec08d1ab97 Must remove before we insert - cannot safely if overlapping with insertions 2017-01-17 10:37:46 +00:00
martinsumner
59dc451e70 Address confusion over type in DeletionsList 2017-01-17 10:21:29 +00:00
martinsumner
9832ecc369 Manifest now back to a simple list 
This has refactored code with the implementation of the manifest
isolated in to a seperate module, and the pure async relationship
between penciller and their clerk.  However, the manifest is just a
simple list at each level.
 2017-01-17 10:12:15 +00:00
martinsumner
13c5cc8899 Add timing points 
Add some timing points to manifest updates
 2017-01-15 10:36:50 +00:00
martinsumner
8e92a2c563 Fix pclerk unit test 2017-01-15 01:47:23 +00:00
Martin Sumner
f868be8086 Change Penciller to Clerk handoff 
correct the previous change to make sure that deletes are not confirmed
when work is outstanding, but also make sure that the clerk only ever
casts the Penciller so no deadlocks can ensue.
 2017-01-15 00:52:43 +00:00
martinsumner
9577d24be0 Wait on close of penciller clerk 
The clerk never calls the Penciller, so cannot deadlock.  Will wait for
a merge to be complete
 2017-01-14 22:59:04 +00:00
martinsumner
dcf3afc056 Log basement setting when creating files 2017-01-14 21:19:51 +00:00
martinsumner
13c81f0ed1 Basic working 
Some basic tests working - but still outstanding issues.
 2017-01-14 19:41:09 +00:00
martinsumner
76bdd83346 Manifest refactor - STILL BROKEN 
Some working tests now, but sitll broken
 2017-01-14 16:36:05 +00:00
martinsumner
0204a23a58 Refactor - STILL BROKEN 
Will at least compile, but in need of a massive eunit rewrite and
associated debug to get back to a potentially verifiable state again
 2017-01-13 18:23:57 +00:00
martinsumner
08641e05cf Manifest changes - BROKEN 
Going to abandond this branch for now.  The change is beoming
excessively time consuming, and it is not clear that a smaller change
might not achieve more of the objectives.

All this is broken - but perhaps could get picke dup another day.
 2017-01-12 13:48:43 +00:00
martinsumner
7049aaf5ca Better attempt to handle empty file being generated 2016-12-29 09:35:58 +00:00
martinsumner
e01b310d20 Handle production of empty file 2016-12-29 05:09:47 +00:00
martinsumner
dc28388c76 Removed SFT 
Now moved over to SST on this branch
 2016-12-29 02:07:14 +00:00
martinsumner
c664483f03 Add basic merge support 
No generates KV list first, and then creates a new SST
 2016-12-28 21:47:05 +00:00
martinsumner
86bdfdeaf0 Reverted back out the additional bloom check 
This is desirable to add back in going forward, but wasn't implemented
in a safe or clear way.

The way the bloom was or was not on the LoopState was clumsy, and it got
persisted in multiple places without a CRC check.

Intention to implement back in wherby it is requested on-demand by the
Penciller, and then the SFT worker lifts it off disk and CRC checks it.
So it is never on the SFT LoopState.  Also it will be easier to control
the logic over which levels have the bloom in the Penciller.
 2016-12-11 21:01:10 +00:00
martinsumner
f96d148073 Make the merge_test a more sensible size 
On the verge of a timeout.  Rather than keep battling with the timeout,
make it do less work
 2016-12-11 20:17:05 +00:00
martinsumner
5cfe9a71e1 Wrap test with non-default timeout 2016-12-11 15:25:14 +00:00
martinsumner
523716e8f2 Add tiny bloom to Penciller Manifest 
This is an attempt to save on unnecessary message transfers, and
slightly more expensive GCS checks in the SFT file itself.
 2016-12-11 04:48:50 +00:00
martinsumner
2d3a40e6f1 Magic Hash - and no L0 Index 
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).

The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.

It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest.  This will then reduce the message passing required for a GET
not in the cache or higher levels
 2016-12-11 01:02:56 +00:00
martinsumner
8cbe2ef93a Coverage cheats 
You juke the stats, and majors become colonels.  I've been here before
 2016-11-14 20:43:38 +00:00
Martin Sumner
2d590c3077 More aggressive clerk 2016-11-05 17:50:28 +00:00
martinsumner
2299e1ce9d Prune unreachbale branches 2016-11-04 19:14:03 +00:00
martinsumner
2b8a37439d Log refinement - logging process IDs 2016-11-04 17:28:04 +00:00
martinsumner
7147ec0470 Logging - Phase 1 
Abstract out logging and introduce a logbase
 2016-11-02 18:14:46 +00:00
martinsumner
3b05874b8a Add initial timestamp support 
Covered only by basic unit test at present.
 2016-10-31 12:12:06 +00:00
martinsumner
95609702bd Penciller Memory Refactor 
Plugged the ne wpencille rmemory into the Penciller, and took advantage
of the increased speed to simplify the callbacks involved.

The outcome is much simpler code
 2016-10-30 18:25:30 +00:00
martinsumner
20cc17f916 Penciller Refactor 
Removed o(100) lines of code by refactoring the Penciller to no longer
use ETS tables.  The code is less confusing, and probably not an awful
lot slower.
 2016-10-27 20:56:18 +01:00
martinsumner
254183369e CDB - switch to gen_fsm 
The CDB file management server has distinct states, and was growing case
logic to prevent certain messages from being handled in ceratin states,
and to handle different messages differently.  So this has now been
converted to a gen_fsm.

As part of resolving this, the space_clear_ondelete test has been
completed, and completing this revealed that the Penciller could not
cope with a change which emptied the ledger.  So a series of changes has
been handled to allow it to smoothly progress to an empty manifest.
 2016-10-26 20:39:16 +01:00
martinsumner
e9c568a8b3 Test fix-up 
There was a test that failed to close down a bookie and that caused some
issues.  The issues are double-reoslved, the close down was tidied as
well as the forgotten close being added back in.

There is some generla tidy around in anticipation of TTL support.
 2016-10-21 21:26:28 +01:00
martinsumner
c431bf3b0a Broken snapshot test 
The test confirming that deleting sft files wer eheld open whilst
snapshots were registered was actually broken.  This test has now been
fixed, as well as the logic in registring snapshots which had used
ledger_sqn mistakenly rather than manifest_sqn.
 2016-10-21 11:38:30 +01:00
martinsumner
caa8d26e3e Stop file check 
File check now covered by measure in the sft_new path, whihc will backup
any existing file before moving.

This gets triggered by incomplete changes on shutdown.
 2016-10-20 19:18:49 +01:00