umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
802 commits 1 branch 0 tags 25 MiB
Commit graph

588 commits

Author SHA1 Message Date
martinsumner
adb78f5c5a Resolve pclerk crash 
Need to add extra logging to understand why pclerk crashes in some
volume tests.

Penciller's Clerk <0.813.0> shutdown now complete for reason
{badarg,[{dict,fetch,[63,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,[100,105,99,116,46,101,114,108]},{line,126}]},{leveled_pclerk,handle_cast,2,[{file,[115,114,99,47,108,101,118,101,108,101,100,95,112,99,108,101,114,107,46,101,114,108]},{line,96}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,604}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,239}]}]}

Should only be prompted after prompt deletions had bene updated.
Perhaps a race whereby somehow it is prompted again after it is emptied.
 2017-02-09 14:33:39 +00:00
martinsumner
7320b34681 Comment update 2017-01-25 12:38:33 +00:00
martinsumner
684f1bcd99 Increase bloom size 2017-01-25 01:07:15 +00:00
martinsumner
10a64b86be Remove rogue timestamp 2017-01-24 21:53:43 +00:00
martinsumner
d57b74d967 Re-introduce tinybloom to SST 
This had been removed due to the CPU cost of adding - however then the
tinybloom wa simplemented by directly manipulating bits through binary
comprehension - rather than applying bor band bsl bsr operations.

With these operations the cost of producing and checking the bloom is
<10% by comparison.
 2017-01-24 21:51:12 +00:00
Martin Sumner
f8f2e02d92 Use bel for powers of two 
D’oh.  That’s much much faster of course!
 2017-01-24 18:09:51 +00:00
martinsumner
8c3d0fc493 Alternate implementation 2017-01-24 17:15:39 +00:00
martinsumner
9d95057518 Do it daft way - will it be faster? 2017-01-24 17:00:30 +00:00
martinsumner
93448b76ba Add some extra length checks 2017-01-24 15:49:19 +00:00
martinsumner
a8488663c7 Alternate tinybloom 
Previously the code had used a tiny bloom - but this proved to be
expensive to build.  Looking at the alternative of a slot-size only tiny
bloom
 2017-01-24 15:48:12 +00:00
martinsumner
266e851a96 Merge pull request #18 from martinsumner/mas-leveledtree 
Mas leveledtree
 2017-01-24 02:39:24 +00:00
martinsumner
d225f4d7f5 Add use of leveled_tree to sst summary 2017-01-23 22:58:51 +00:00
martinsumner
94762c79ca Improve test coverage 
Add tests for other scenarios
 2017-01-23 21:43:33 +00:00
martinsumner
861cedf45e Add back missed space in logs 2017-01-23 19:16:17 +00:00
martinsumner
fb896f13b1 Improve logging - add timestamp to logs 2017-01-23 18:56:01 +00:00
martinsumner
90c920fe86 Additional unit test work 
Reverts a previous ct test fix
 2017-01-23 15:15:40 +00:00
martinsumner
5105df1cd6 Add replace capability to manifest 2017-01-23 11:02:54 +00:00
Martin Sumner
2c4c5c9597 Corrections 
Support an empty list of entries being added.  Also specify a tree
correctly in all from_orderedlist scenarios
 2017-01-23 00:22:53 +00:00
martinsumner
cc9494735b Use an index tree for the lower levels of penciller manifest 2017-01-22 23:36:16 +00:00
Martin Sumner
58cda7d157 Switch to using skip lists from leveled_tree 
Remove now unused leveled_skiplist and leveled_tinybloom
 2017-01-21 22:34:56 +00:00
martinsumner
6d2eb1d57c Added skiplist to datatypes 2017-01-21 21:51:35 +00:00
martinsumner
efec232e71 Adjust test size to match SST summary 2017-01-21 16:26:23 +00:00
martinsumner
0fa22ab4b3 Add skiplist into leveled_tree 
Getting ready to remove seperate skiplist module.  Need to add the
skiplist support in leveled_tree for relative testing.
 2017-01-21 16:23:24 +00:00
martinsumner
f031558b28 Make idxt active implementation 
Use idxt for ledger_cache - ready for volume test
 2017-01-21 13:45:19 +00:00
martinsumner
917473dfdc Alternative tree 
As an alternatuve to a tree - try a tree where the list of elements and
the tree to lookup the list of elements are kept in seperate objects
 2017-01-21 13:41:41 +00:00
martinsumner
d6761259e1 Experiment with seperating index from elements 2017-01-21 12:54:32 +00:00
martinsumner
ed96d0ca7a Change so that type of cache is parameterised 
The Tree doesn't seem to be better than the skiplist.  Get ready to make
this switchable
 2017-01-21 11:38:26 +00:00
Martin Sumner
4846272393 Build more efficiently 
Don’t use ++ to build the tree
 2017-01-21 01:48:38 +00:00
Martin Sumner
51f0402292 Change Skip Width 2017-01-20 21:31:17 +00:00
Martin Sumner
9e32e228bb Include gb_trees code 
The gb_trees in OTP16 doesn’t have iterative from support.  Included
here so that we can continue to use it regardless in Riak.
 2017-01-20 17:34:57 +00:00
martinsumner
3d99036093 Switch the LM1 cache to be a tree 
Use a tree of lists not a skiplist
 2017-01-20 16:36:20 +00:00
Martin Sumner
1745ba6863 Improve from_list speed 2017-01-20 16:35:38 +00:00
martinsumner
220d493b5a Tree changes 2017-01-20 16:15:00 +00:00
Martin Sumner
860a8a95e3 Width of 32 may be better 
With 4000 keys - build time dropped significantly
 2017-01-19 22:59:53 +00:00
Martin Sumner
12d06efa10 Set log to debug 2017-01-19 22:49:59 +00:00
martinsumner
c722f3132d Initial version of potential new tree module 
with basic Unit Tests.  Intended to replace skiplist
 2017-01-19 22:49:32 +00:00
martinsumner
853f113ee5 Inker Manifest to be a two-level list 
To eas seraching from front to back, change the inker manifest to be a
two-level list
 2017-01-19 12:23:28 +00:00
Martin Sumner
1f406c76dd Resolve test issues 2017-01-19 09:47:56 +00:00
martinsumner
3ca928629c Split out inker manifest - load last key from manifest 
The process of re-opening file included an expensive search for the
LastKey - now the LastKey can be provided out of the manifest.
 2017-01-18 15:23:06 +00:00
martinsumner
4e53128a2d Merge remote-tracking branch 'refs/remotes/origin/master' into mas-manifesttidy-inker 2017-01-18 10:13:21 +00:00
martinsumner
f97e02975a Add comments 2017-01-17 21:09:42 +00:00
martinsumner
59c0c06b60 OTP16 type again 2017-01-17 19:01:59 +00:00
martinsumner
5345242e27 Take out type definition for OTP 16 2017-01-17 19:00:30 +00:00
martinsumner
6298886592 Empty non-persisted parts of record on save 
Some of the manifest should not be preserved between restarts - so empty
these parts on save
 2017-01-17 17:15:04 +00:00
martinsumner
850a524ad3 Close Pending Deletes on shutdown 
Pending deletes are not in the Manifest, and so were not being closed on
shutdown
 2017-01-17 16:54:54 +00:00
martinsumner
3712c62a50 Broken WIP 2017-01-17 16:30:04 +00:00
martinsumner
7086717765 Rename to pmanifest 
There may be a seperate imanifest in the future
 2017-01-17 14:11:50 +00:00
martinsumner
2e734ff501 Corrected snapshot releases 2017-01-17 14:02:21 +00:00
martinsumner
c99c50ce6e Fix-up message exchange on confirm delete 2017-01-17 11:18:58 +00:00
martinsumner
14ebf94e56 Correct botched manifest entry removal 2017-01-17 10:53:13 +00:00