Commit graph

86 commits

Author SHA1 Message Date
Martin Sumner
61b7be5039 Make compression algorithm an option
Compression can be switched between LZ4 and zlib (native).

The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
Martin Sumner
bfaed921e6 Split code for folders - introduce runner actor
Introduce a dedicated module for all the different fold types.  Also simplify the list of folders by deprecating those folds that should eb achieveable by fold_heads/fold_objects type folds but with smarter functions.

Makes sure that the fold functiosn also have better spec coverage, and are dialyzer checked.
2017-10-17 20:39:11 +01:00
Heinz N. Gies
25389893cf Add compatibility for old and new random / rand functions 2017-08-01 11:24:12 +02:00
Heinz N. Gies
eece253222 Cleanup most dialyzer errrors in leveled_inker 2017-07-31 19:47:58 +02:00
martinsumner
f23072fb54 Merge remote-tracking branch 'refs/remotes/origin/master' into mas-specs-i61b 2017-05-26 13:45:20 +01:00
martinsumner
c664176247 Release penciller snapshot after journal compaction
As otherwise memory consumption beocmes an issue, as they will take an
hour to timeout naturally.
2017-05-26 10:51:30 +01:00
martinsumner
366253831a Add dpecs to inker
Give the dialyzer a hand
2017-05-22 15:31:42 +01:00
Martin Sumner
8e334c0b5a Merge pull request #63 from martinsumner/mas-compactscore-i35
Mas compactscore i35
2017-05-22 09:57:32 +01:00
martinsumner
11ff3129f3 Reduce compaction targets
Cmpaction is overly aggressive.  It is a lot of work to compact a run of
files for just 20% reduction in disk space, when disk space for the
Journal (i.e. low IOPS disk space should be relatively inexpensive).
Require at least a 40% reduction for a compaction job.
2017-03-30 12:15:36 +01:00
martinsumner
6143fcb664 Remove binary_to_term
when fetching don't need to binary_to_term key changes
2017-03-29 15:37:04 +01:00
martinsumner
f3ffa920af Trying to standardise binary manipulation of value
Looking into theory that use of term_to_binary is imperfect.  Also may
be better to compress values only when they are compacted?
2017-03-20 15:43:54 +00:00
martinsumner
9ad6969b0d Seed randomnes at Actor startup 2017-03-06 21:35:02 +00:00
martinsumner
c92107e4b4 2i order of events
When running a load of mainly 2i queries, there is a huge cost in the
previous snapshot code.  The time taken to create a clone of the
Penciller (duplicating all the LoopState) varied between 1 and 200ms
depedning on the size of the LoopState.

For 2i queries, most of that LoopState was then being thrown away after
running the query against the levelzero_cache.  This was taking < 1ms on
average.  It would be better to avoid the o(100)ms of CPU  burning and
block for o(1)ms - so th eorder of events have been changed ot filter
first so only the small part of the LoopState actually required is
copied to the clone.
2017-03-06 18:42:32 +00:00
martinsumner
27fdbf2ac8 Syntax 2017-02-26 22:46:57 +00:00
martinsumner
44471ecc0a ManEntry must be a ManEntry?
What else could it be?
2017-02-26 22:42:44 +00:00
martinsumner
fb54d29873 Timestamps no longer required 2017-02-26 19:09:56 +00:00
martinsumner
fad9bfbff1 Trigger log points, deprecate lower PUT log points
Only record PUT timings at the Bookie, and trigger them correctly on the
right log point
2017-02-26 19:07:55 +00:00
martinsumner
266e851a96 Merge pull request #18 from martinsumner/mas-leveledtree
Mas leveledtree
2017-01-24 02:39:24 +00:00
martinsumner
3d99036093 Switch the LM1 cache to be a tree
Use a tree of lists not a skiplist
2017-01-20 16:36:20 +00:00
martinsumner
853f113ee5 Inker Manifest to be a two-level list
To eas seraching from front to back, change the inker manifest to be a
two-level list
2017-01-19 12:23:28 +00:00
Martin Sumner
1f406c76dd Resolve test issues 2017-01-19 09:47:56 +00:00
martinsumner
3ca928629c Split out inker manifest - load last key from manifest
The process of re-opening file included an expensive search for the
LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
martinsumner
3712c62a50 Broken WIP 2017-01-17 16:30:04 +00:00
Martin Sumner
676e8fa494 Add Get Timing 2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909 put-timing default
Remove need for individual actors to know the defaults for put_timing
tuple
2016-12-22 14:41:43 +00:00
martinsumner
130fb36ddd Add head timings
Include log breaking down timings of HEAD requests by result and level
2016-12-22 14:03:31 +00:00
martinsumner
c193962c92 Sort out different timestamps 2016-12-20 23:16:52 +00:00
martinsumner
060ce2e263 Add put timing points 2016-12-20 23:11:50 +00:00
martinsumner
2d3a40e6f1 Magic Hash - and no L0 Index
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).

The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.

It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest.  This will then reduce the message passing required for a GET
not in the cache or higher levels
2016-12-11 01:02:56 +00:00
martinsumner
c4e4cf67fe Add bloom to loaded skiplist 2016-12-10 11:39:00 +00:00
martinsumner
03d025d581 Replace ledger-side gb_trees
Try to make minimal change to replace gb_trees with gb_tree API-like
skiplists
2016-11-25 14:50:13 +00:00
martinsumner
0f7e421371 Add destruction
Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
martinsumner
f40ecdd529 Pick-up test misses
There were some coverage misses in tests, so check in unit test coverage
or remove branches not currently needed.
2016-11-18 21:35:45 +00:00
martinsumner
8cbe2ef93a Coverage cheats
You juke the stats, and majors become colonels.  I've been here before
2016-11-14 20:43:38 +00:00
martinsumner
630f802780 Inker Close nastiness
Try to stop some of the potential deadlocking around Inker close and
prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
martinsumner
44738f7c75 Deferred Deletion of Journals
This allows for deleted journals to be retained for a period (the
waste_retnetion_period).  The idea being that a backup strategy can
ensure that all journals are backed up, even ones created and removed
from within a backup period - so that any restore pont is possible.

This is also a pre-cursor to removing some of the PromptDelete
complexity from the Inker Clerk - all compactions can prompt deletion as
deletion is now deferred.
2016-11-14 11:17:14 +00:00
martinsumner
cc3cbc983b Tidy up closure of CDB Files 2016-11-08 22:43:22 +00:00
martinsumner
70dc637c97 Add slow offer support on write pressure
When there is write pressure on the penciller and it returns to the
bookie, the bookie will now punish the next PUT (and itself) with a
pause.  The longer the back-pressure state has been in place, the more
frequent the pauses
2016-11-05 14:31:10 +00:00
martinsumner
479dc3ac80 Registering and releasing of Journal snapshots
Added a test of journal compaction with a registered snapshot and it
showed that the deleting of files did not correctly check the list of
registerd snapshots.  Corrected.
2016-11-04 15:56:57 +00:00
martinsumner
ad9c886b65 Inker test for pending manifest
Should ignore the corrupted pending manifest file
2016-11-03 20:00:55 +00:00
martinsumner
4e46c9735d Log improvements
Continuation of log review and conversion to using central log function.

Fixup of convoluted shutdown process between Bookie, Inker and Inker's
Clerk
2016-11-03 16:05:43 +00:00
martinsumner
37e78dcdc9 Expanded AAE tests to include busted hashtable
Busted the hashtable in a Journal file, and demonstrated it can be fixed
by changing the extension name (no need to recover from backup if only
the hashtable is bust)
2016-11-03 12:11:50 +00:00
martinsumner
73004328e1 Recovery Tests
Some initial entropy tests showing loss of data from a corrupted CDB
file.
2016-10-31 20:58:19 +00:00
martinsumner
7d3a04428b Refactor snapshot
Better reuse snapshotting fucntions in the Bookie, and use it to support
doing Inker clone checks
2016-10-31 17:26:28 +00:00
martinsumner
311179964a Quality review
Minor test fix-up and quality changes
2016-10-30 22:06:44 +00:00
martinsumner
95609702bd Penciller Memory Refactor
Plugged the ne wpencille rmemory into the Penciller, and took advantage
of the increased speed to simplify the callbacks involved.

The outcome is much simpler code
2016-10-30 18:25:30 +00:00
martinsumner
cdb01cd24f Quality Review
Looked through test coverage and dialyzer output and attempted to fill
test gaps and strip out untestable code (to let it crash).
2016-10-29 00:52:49 +01:00
martinsumner
20cc17f916 Penciller Refactor
Removed o(100) lines of code by refactoring the Penciller to no longer
use ETS tables.  The code is less confusing, and probably not an awful
lot slower.
2016-10-27 20:56:18 +01:00
martinsumner
254183369e CDB - switch to gen_fsm
The CDB file management server has distinct states, and was growing case
logic to prevent certain messages from being handled in ceratin states,
and to handle different messages differently.  So this has now been
converted to a gen_fsm.

As part of resolving this, the space_clear_ondelete test has been
completed, and completing this revealed that the Penciller could not
cope with a change which emptied the ledger.  So a series of changes has
been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
martinsumner
2a47acc758 Rolback hash|no_hash and batch journal compaction
The no_hash option in CDB files became too hard to manage, in particular
the need to scan the whole file to find the last_key rather than cheat
and use the index.  It has been removed for now.

The writing to the journal during journal compaction has now been
enhanced by a mput option on the CDB file write - so it can write each
batch as one pwrite operation.
2016-10-26 11:39:27 +01:00