umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
525 commits 1 branch 0 tags 25 MiB
Commit graph

68 commits

Author SHA1 Message Date
martinsumner
adb78f5c5a Resolve pclerk crash 
Need to add extra logging to understand why pclerk crashes in some
volume tests.

Penciller's Clerk <0.813.0> shutdown now complete for reason
{badarg,[{dict,fetch,[63,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,[100,105,99,116,46,101,114,108]},{line,126}]},{leveled_pclerk,handle_cast,2,[{file,[115,114,99,47,108,101,118,101,108,101,100,95,112,99,108,101,114,107,46,101,114,108]},{line,96}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,604}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,239}]}]}

Should only be prompted after prompt deletions had bene updated.
Perhaps a race whereby somehow it is prompted again after it is emptied.
 2017-02-09 14:33:39 +00:00
martinsumner
d57b74d967 Re-introduce tinybloom to SST 
This had been removed due to the CPU cost of adding - however then the
tinybloom wa simplemented by directly manipulating bits through binary
comprehension - rather than applying bor band bsl bsr operations.

With these operations the cost of producing and checking the bloom is
<10% by comparison.
 2017-01-24 21:51:12 +00:00
martinsumner
861cedf45e Add back missed space in logs 2017-01-23 19:16:17 +00:00
martinsumner
fb896f13b1 Improve logging - add timestamp to logs 2017-01-23 18:56:01 +00:00
martinsumner
90c920fe86 Additional unit test work 
Reverts a previous ct test fix
 2017-01-23 15:15:40 +00:00
Martin Sumner
2c4c5c9597 Corrections 
Support an empty list of entries being added.  Also specify a tree
correctly in all from_orderedlist scenarios
 2017-01-23 00:22:53 +00:00
martinsumner
14ebf94e56 Correct botched manifest entry removal 2017-01-17 10:53:13 +00:00
martinsumner
5076187545 Reduce log level of troubleshooting log 2017-01-17 10:39:31 +00:00
martinsumner
5adfb5c5ef Extra logging for troubleshooting 2017-01-17 10:32:15 +00:00
martinsumner
e6c4c9eff8 Log deletions from mnaifest (via GC) 2017-01-15 11:04:26 +00:00
martinsumner
13c5cc8899 Add timing points 
Add some timing points to manifest updates
 2017-01-15 10:36:50 +00:00
martinsumner
7b0b3e9b83 Add logging of Manifest SQN at startup 2017-01-14 22:26:26 +00:00
martinsumner
dcf3afc056 Log basement setting when creating files 2017-01-14 21:19:51 +00:00
martinsumner
85c03a61f9 Log changes 2017-01-14 20:11:01 +00:00
martinsumner
13c81f0ed1 Basic working 
Some basic tests working - but still outstanding issues.
 2017-01-14 19:41:09 +00:00
martinsumner
0204a23a58 Refactor - STILL BROKEN 
Will at least compile, but in need of a massive eunit rewrite and
associated debug to get back to a potentially verifiable state again
 2017-01-13 18:23:57 +00:00
martinsumner
08641e05cf Manifest changes - BROKEN 
Going to abandond this branch for now.  The change is beoming
excessively time consuming, and it is not clear that a smaller change
might not achieve more of the objectives.

All this is broken - but perhaps could get picke dup another day.
 2017-01-12 13:48:43 +00:00
martinsumner
ed27a53452 New manifest code 
The manifest had previously been a list for eveyr leevl of the manifest,
and keys were found by folding over the list.  By Level 4 the list will
be 4096 items long, and so the fold would be expensive, and would be
required many times.

To make this less expensive an ETS table is to use.  However, the ETS
table needs to be shared between snapshots and so in order to use the
ETS the entries to the table need to support multi-versioning - whereby
each clone can see a version of the table at the Manifest SQN the clone
is supporting.
 2017-01-09 14:52:26 +00:00
martinsumner
70c6e52fa7 Remove logs for slot_cache 2017-01-03 15:27:28 +00:00
martinsumner
b6ae0e1af5 Fix broken SST cache 2017-01-03 13:03:59 +00:00
martinsumner
31d4346806 Log improvements 
Log on bad CRC, and also not seeing SST timing logs, so log these more
frequently
 2017-01-02 18:54:19 +00:00
Martin Sumner
2079fff7f8 Switched to indexed blocks as slot implementation 
Prior to this refactor, the slot and been made up of four blocks with
an external binary index.  Although the form of the index has changed
again, micro-benchmarking once again showed that this was a relatively
efficient mechanism.
 2017-01-02 10:47:04 +00:00
martinsumner
0c543ae3ec Remove legacy logs 2016-12-29 05:10:11 +00:00
martinsumner
e01b310d20 Handle production of empty file 2016-12-29 05:09:47 +00:00
martinsumner
dc28388c76 Removed SFT 
Now moved over to SST on this branch
 2016-12-29 02:07:14 +00:00
martinsumner
3716de1c82 Revert back to sampling 
timing logs should be based on a sample
 2016-12-28 15:49:58 +00:00
martinsumner
cbad375373 Refactoring of skiplist ranges and support for sst ranges 
the Skiplist range code was needlessly complicated.  It may be faster
than the new code, but the complexity delta cannot be support for such a
small change.

This was incovered whilst troubleshooting the initial kv range test.
 2016-12-28 15:48:04 +00:00
martinsumner
0d0ab32653 Some end-to-end testing 2016-12-24 17:48:31 +00:00
martinsumner
58d8e60994 Some basic code layout work 2016-12-24 15:12:24 +00:00
martinsumner
0ddaaf9ac3 Stopped unnecessary seek for last_key 
When rolling we already know the last_key - no need to seek for it on
startup.

The time it takes for this seek needs to be considered with regards to
startup time.  Can we do without knowing lastkey?
 2016-12-22 19:51:39 +00:00
martinsumner
a131b99082 Randomising logging of PUT timings 2016-12-22 17:33:14 +00:00
martinsumner
353fb08e21 Randomise logging of GET/HEAD samples 2016-12-22 17:28:41 +00:00
martinsumner
ee534081c3 Reduce log levels 
Remove some log noise to debug level
 2016-12-22 17:15:42 +00:00
martinsumner
151dd3ab70 Sample only for HEAD/GET response times 
Report regularly but only on a sample
 2016-12-22 16:47:36 +00:00
Martin Sumner
676e8fa494 Add Get Timing 2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909 put-timing default 
Remove need for individual actors to know the defaults for put_timing
tuple
 2016-12-22 14:41:43 +00:00
Martin Sumner
e9e0a7b323 Set higher logpoint 
Expectation is for many HEAD requests - so only log every 100K
 2016-12-22 14:36:57 +00:00
martinsumner
df350e1e6f Add unit test for head timing 2016-12-22 14:09:17 +00:00
martinsumner
130fb36ddd Add head timings 
Include log breaking down timings of HEAD requests by result and level
 2016-12-22 14:03:31 +00:00
martinsumner
be775127e8 Improve logging of merge activity and slow GETs 
Look into speculation that collisions between fetch_rnage and fetch may
be an issue
 2016-12-21 12:45:27 +00:00
martinsumner
f3e16dcd10 Add long-running logs 2016-12-21 01:56:12 +00:00
martinsumner
060ce2e263 Add put timing points 2016-12-20 23:11:50 +00:00
martinsumner
9e28287231 Resolve failing recovery test 
Now passing consistently with a number of different corruptions catered
for (including corruption of the Tag in the Inker Key)
 2016-12-16 23:18:55 +00:00
martinsumner
c8be3bfa46 Slot hash corrected 
When building the hashtree the incorrect IndexLength was being used to
calculate the slot - causing many queries to loop all the way round the
Index
 2016-12-13 17:02:45 +00:00
martinsumner
8f775a88fd Investigate performance regression 
Performance has regressed following the hashtable change.  Speculation
that the hashtable format might not be right, and so there is more
cycling around the hashtree.  Logging added.
 2016-12-13 14:06:19 +00:00
martinsumner
52499170c0 Tidy logging following changes 
Include detailed timings in a permanent log
 2016-12-13 12:41:44 +00:00
martinsumner
8bcb49479d Re-introduce ETS Index 
Add ETS Index back in to avoid having to check each skip list in turn.
Also this helps keep a lower skip list size.
 2016-12-11 05:23:24 +00:00
martinsumner
ccc993383d Stop second hash on fetch_head 
The bookie should magic_hash for fetch_head, and now passes the hash to
the Penciller so second hash not required.
 2016-12-11 01:21:53 +00:00
martinsumner
2d3a40e6f1 Magic Hash - and no L0 Index 
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).

The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.

It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest.  This will then reduce the message passing required for a GET
not in the cache or higher levels
 2016-12-11 01:02:56 +00:00
martinsumner
c40e5d2d30 Reduce log noise 2016-12-08 16:43:35 +00:00