umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1632 commits 1 branch 0 tags 25 MiB
Commit graph

1115 commits

Author SHA1 Message Date
martinsumner
c92107e4b4 2i order of events 
When running a load of mainly 2i queries, there is a huge cost in the
previous snapshot code.  The time taken to create a clone of the
Penciller (duplicating all the LoopState) varied between 1 and 200ms
depedning on the size of the LoopState.

For 2i queries, most of that LoopState was then being thrown away after
running the query against the levelzero_cache.  This was taking < 1ms on
average.  It would be better to avoid the o(100)ms of CPU  burning and
block for o(1)ms - so th eorder of events have been changed ot filter
first so only the small part of the LoopState actually required is
copied to the clone.
 2017-03-06 18:42:32 +00:00
Martin Sumner
c1dc92720c Random, random, random 
well random had me foxed.  As the clone was short-lived process it only
called random once - and so always got the same answer.

random has to be seeded to give different answers when called once from
a process - so this is now seeded in leveed_log
 2017-03-06 13:51:38 +00:00
martinsumner
eb6f668fcd Use log at random 
Easy way to sample frequent things - especially when they'r ein ocverag
equeries
 2017-03-06 10:34:56 +00:00
martinsumner
94a39e4c4d Fix double logging of B0004 2017-03-06 10:27:48 +00:00
martinsumner
5c2f05858d Alter logging to help understand performance factors 
Change logging of the snapshots to better understand performance
 2017-03-06 10:17:51 +00:00
martinsumner
bc5388710b Update SST comments 2017-03-04 20:47:46 +00:00
martinsumner
426f5df75c Fix dialyzer warnings 2017-03-02 21:24:40 +00:00
martinsumner
30e23fe514 Add no_lookup feature to snapshots 2017-03-02 18:23:47 +00:00
martinsumner
b01f7d23df Add pre-filter for 2i queries 2017-03-02 17:49:43 +00:00
martinsumner
480a029135 Renaming 
Renaming to leveled

Would prefer to call it PontusDB - as a database optimised for HEAD
requests.  But our heroes often leave us.
 2017-02-28 01:38:47 +00:00
martinsumner
35b6e16590 Clear dialyzer warnings 2017-02-27 20:23:36 +00:00
martinsumner
fce7cbd2ef Add sync_strategy comments 2017-02-27 17:32:38 +00:00
martinsumner
2db28c3028 Tidy line length 2017-02-27 12:24:26 +00:00
martinsumner
8d3f4bd4dc Comment API on Bookie 
Make it easier to understand how to work the store
 2017-02-27 11:58:16 +00:00
martinsumner
27fdbf2ac8 Syntax 2017-02-26 22:46:57 +00:00
martinsumner
a9101e4781 SibCount must be non-zero 2017-02-26 22:45:20 +00:00
martinsumner
44471ecc0a ManEntry must be a ManEntry? 
What else could it be?
 2017-02-26 22:42:44 +00:00
martinsumner
aba385869f Test long running clause 2017-02-26 22:41:25 +00:00
martinsumner
4e6c22d7e6 Dumb test 
Can't test for slow not_present outside of simple unit test ... will be
very rare event
 2017-02-26 21:48:04 +00:00
martinsumner
19534122a2 Coverage checks 2017-02-26 21:37:47 +00:00
martinsumner
100d675472 iManifest -test empty journal 2017-02-26 21:13:07 +00:00
martinsumner
41e5686a66 Test coverage - missing 1% 2017-02-26 20:52:40 +00:00
martinsumner
fb54d29873 Timestamps no longer required 2017-02-26 19:09:56 +00:00
martinsumner
fad9bfbff1 Trigger log points, deprecate lower PUT log points 
Only record PUT timings at the Bookie, and trigger them correctly on the
right log point
 2017-02-26 19:07:55 +00:00
martinsumner
8077c70486 More conservative approach to ongoing work monitoring 
As per comments though - if we auto-restart pclerk in the future this
will have to be re-considered.

Perhaps a re-starting pclerk should force some reset of this boolean on
startup perhaps by making a different work_for_clerk if in a virgin
state.
 2017-02-09 23:49:12 +00:00
martinsumner
793977b76c Prevent double work ongoing 
This was the cause of the pclerk crash.  The concurrent work would lead
to the attempt to re-fetch the prompted deletions after they had been
deleted.
 2017-02-09 23:41:28 +00:00
martinsumner
05b2d8faaf is_empty not in OTP16 
bloody OTP16 strikes again - is_empty isn't there
 2017-02-09 21:12:01 +00:00
martinsumner
adb78f5c5a Resolve pclerk crash 
Need to add extra logging to understand why pclerk crashes in some
volume tests.

Penciller's Clerk <0.813.0> shutdown now complete for reason
{badarg,[{dict,fetch,[63,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,[100,105,99,116,46,101,114,108]},{line,126}]},{leveled_pclerk,handle_cast,2,[{file,[115,114,99,47,108,101,118,101,108,101,100,95,112,99,108,101,114,107,46,101,114,108]},{line,96}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,604}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,239}]}]}

Should only be prompted after prompt deletions had bene updated.
Perhaps a race whereby somehow it is prompted again after it is emptied.
 2017-02-09 14:33:39 +00:00
martinsumner
7320b34681 Comment update 2017-01-25 12:38:33 +00:00
martinsumner
684f1bcd99 Increase bloom size 2017-01-25 01:07:15 +00:00
martinsumner
10a64b86be Remove rogue timestamp 2017-01-24 21:53:43 +00:00
martinsumner
d57b74d967 Re-introduce tinybloom to SST 
This had been removed due to the CPU cost of adding - however then the
tinybloom wa simplemented by directly manipulating bits through binary
comprehension - rather than applying bor band bsl bsr operations.

With these operations the cost of producing and checking the bloom is
<10% by comparison.
 2017-01-24 21:51:12 +00:00
Martin Sumner
f8f2e02d92 Use bel for powers of two 
D’oh.  That’s much much faster of course!
 2017-01-24 18:09:51 +00:00
martinsumner
8c3d0fc493 Alternate implementation 2017-01-24 17:15:39 +00:00
martinsumner
9d95057518 Do it daft way - will it be faster? 2017-01-24 17:00:30 +00:00
martinsumner
93448b76ba Add some extra length checks 2017-01-24 15:49:19 +00:00
martinsumner
a8488663c7 Alternate tinybloom 
Previously the code had used a tiny bloom - but this proved to be
expensive to build.  Looking at the alternative of a slot-size only tiny
bloom
 2017-01-24 15:48:12 +00:00
martinsumner
266e851a96 Merge pull request #18 from martinsumner/mas-leveledtree 
Mas leveledtree
 2017-01-24 02:39:24 +00:00
martinsumner
d225f4d7f5 Add use of leveled_tree to sst summary 2017-01-23 22:58:51 +00:00
martinsumner
94762c79ca Improve test coverage 
Add tests for other scenarios
 2017-01-23 21:43:33 +00:00
martinsumner
861cedf45e Add back missed space in logs 2017-01-23 19:16:17 +00:00
martinsumner
fb896f13b1 Improve logging - add timestamp to logs 2017-01-23 18:56:01 +00:00
martinsumner
90c920fe86 Additional unit test work 
Reverts a previous ct test fix
 2017-01-23 15:15:40 +00:00
martinsumner
5105df1cd6 Add replace capability to manifest 2017-01-23 11:02:54 +00:00
Martin Sumner
2c4c5c9597 Corrections 
Support an empty list of entries being added.  Also specify a tree
correctly in all from_orderedlist scenarios
 2017-01-23 00:22:53 +00:00
martinsumner
cc9494735b Use an index tree for the lower levels of penciller manifest 2017-01-22 23:36:16 +00:00
Martin Sumner
58cda7d157 Switch to using skip lists from leveled_tree 
Remove now unused leveled_skiplist and leveled_tinybloom
 2017-01-21 22:34:56 +00:00
martinsumner
6d2eb1d57c Added skiplist to datatypes 2017-01-21 21:51:35 +00:00
martinsumner
efec232e71 Adjust test size to match SST summary 2017-01-21 16:26:23 +00:00
martinsumner
0fa22ab4b3 Add skiplist into leveled_tree 
Getting ready to remove seperate skiplist module.  Need to add the
skiplist support in leveled_tree for relative testing.
 2017-01-21 16:23:24 +00:00