umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1636 commits 1 branch 0 tags 25 MiB
Commit graph

157 commits

Author SHA1 Message Date
Martin Sumner
bde188e691 Constrain keys 
Rather than supporting any() - constrain at least to binary()/integer() or string().
 2018-09-01 12:10:56 +01:00
Martin Sumner
11ba7029aa de-terminate penciller 2018-04-10 09:51:21 +01:00
Martin Sumner
8113aebdcf Add timings for Level 3 
Level 3 readings now relatively common - so time the separately
 2018-02-09 08:59:21 +00:00
Martin Sumner
bb39498ec5 Missed log line added back 
.. and covered in test
 2017-12-04 15:26:01 +00:00
Martin Sumner
3b42bc28d1 Add build timing info to merge_list log 
Help to determine what the expensive part of the operation is
 2017-11-30 16:15:38 +00:00
Martin Sumner
eb90541a85 Add a small cache to SST file 
so that a HEAD which folllows a HEAD (e.g. when a GET follows a HEAD) has a chance of avoiding the binary_to_term CPU load
 2017-11-28 14:56:40 +00:00
Martin Sumner
c2f19d8825 Switch to using bloom at penciller 
Previouslythe tinybloom was used within the SST file as an extra check to remove false fetches.

However the SST already has a low FPR check in the slot_index.  If the newebloom was used (which is no longer per slot, but per sst), this can be shared with the penciller and then the penciller could use it and avoid the message pass.

the message pass may be blocked by a 2i query or a slot fetch request for a merge.  So this should make performance within the Penciller snappier.

This is as a result of taking sst_timings within a volume test - where there was an average of + 100microsecs for each level that was dropped down.  Given the bloom/slot checks were < 20 microsecs - there seems to be some further delay.

The bloom is a binary of > 64 bytes - so passing it around should not require a copy.
 2017-11-28 01:19:30 +00:00
Martin Sumner
f436cfd03e Add consistent timing points 
Now all timing points should be made in a consistent fashion
 2017-11-21 23:13:24 +00:00
Martin Sumner
3ef550d9f8 Refactor timing point management 
For Penciller and timing head requests.
 2017-11-21 19:58:36 +00:00
Martin Sumner
58946a7f98 Amend SST Timing Capture 
Use sampling mechansm from CDB timing capture.  Do it less though - as far more SST fetches in comparison to CDB fetches.
 2017-11-21 17:00:23 +00:00
Martin Sumner
52c7a023a1 Stop using list 
Producing the list of all slots to try appeared to be expensive.  In volume tests taking 150 - 250 microseconds per GET.  Perhaps the list could be long (>1000), with a split and append, so not surprising.

Instead loop and count.
 2017-11-20 20:01:21 +00:00
Martin Sumner
5b4bc1ce59 Merge branch 'master' into mas-i108-cdbtimings 2017-11-20 17:34:50 +00:00
Martin Sumner
faa5ef82aa Test logging of samples 
To prompt the log the journal size needs to be reduced
 2017-11-20 15:31:31 +00:00
Martin Sumner
8a43539090 Take sample timings from CDB files 
Periodically get a CDB file process to take samples of how long fetching keys/values takes - and record  those samples
 2017-11-20 14:58:43 +00:00
Martin Sumner
0e071d078e fold_objects in SQN order 
This adds a test that fold_objects works in SQN order
 2017-11-17 18:30:51 +00:00
Martin Sumner
53afd70174 Remove unnecessary logs 2017-11-09 11:17:50 +00:00
Martin Sumner
22e894c928 Allow waste retnetion to be ignored 
If wast retention period is undefined, then it should be ignored - and no waste retained (rather than retaining waste for 24 hours as at present).

This wasn't working anyway - as reopen reader didn't get the cdb options (which didn't have the waste path on anyway) - so waste would not eb retained if the file had been opened after a stop/start.
 2017-11-08 12:58:09 +00:00
Martin Sumner
e8bd712fb8 Tidy up test shutdown 2017-11-08 11:20:22 +00:00
Martin Sumner
36264eb416 Search range failure 
Discovered a bug with search ranges in leveled_tree - this was uncovered by an intermittently fialing 19.3 test.

Test case added and bug fixed.  It was due to a fialure to use end_key passed causing issues with particular manifests and full bucket ranges.
 2017-10-24 13:19:30 +01:00
Martin Sumner
3950942da3 Roll in fix for intermittently failing test 
As descibed in https://github.com/martinsumner/leveled/issues/92

Only the first fix was made.

Just to eb safe - archiving means renaming to another file with a different extension.  Assumption is that renamed files cna be manually reaped if necessary.
 2017-09-27 23:52:49 +01:00
Heinz N. Gies
25389893cf Add compatibility for old and new random / rand functions 2017-08-01 11:24:12 +02:00
martinsumner
569b498727 Resolve dialyzer warnings 
Botched switch to leveled_log in list - so reoslved dialyzer warnings
 2017-06-01 22:03:51 +01:00
martinsumner
204a5a8caa Change log to keyfind 
Save doing dict:to_list each call to log
 2017-05-31 10:56:48 +01:00
martinsumner
4d12dfe0ab Returning snapshots 
If the clerk updates the manifest - it might not recognise changes to
the manifest made since the clerk took the manifest.  So the penciller
must merge its view of the snapshots back in to the updated manifest
 2017-04-19 22:46:37 +01:00
Martin Sumner
4e9fa2a206 Timeout long-running snapshots 
Add logic to timeout long-running snapshots.
 2017-04-05 09:16:01 +01:00
martinsumner
80b62cbff2 Debug excessive log 
Logs excessively during 2i tests.  Set to debug for now, until can think
further about this
 2017-03-16 20:03:18 +00:00
martinsumner
8b2091cef7 Remaining ledger snapshots log -> debug 
This log under 2i load appears thousands of times per second.  Not
sustainable as an info log.  Will need to think about how to manage
this, but setting back to debug for now
 2017-03-16 19:37:39 +00:00
martinsumner
4f0622d2ac Merge remote-tracking branch 'refs/remotes/origin/mas-sstblock-i42' into mas-sstblockv2-i42 2017-03-13 21:09:13 +00:00
martinsumner
c5bb150f97 Drop some logs 
Not found to be interesting so far
 2017-03-13 20:30:33 +00:00
martinsumner
5311a157d5 Merge remote-tracking branch 'refs/remotes/origin/mas-sstblock-i42' into mas-sstblockv2-i42 2017-03-13 19:22:41 +00:00
martinsumner
f3e962c43a Add level to SST slow fetch log 2017-03-13 12:16:36 +00:00
martinsumner
b2f3d882a9 Draft of branch to condense range_only keys 2017-03-10 20:43:37 +00:00
martinsumner
9ad6969b0d Seed randomnes at Actor startup 2017-03-06 21:35:02 +00:00
martinsumner
87f2c5d7ae Merge remote-tracking branch 'origin/mas-2iphase2-i34' into mas-2iphase2-i34 
# Conflicts:
#	src/leveled_log.erl
 2017-03-06 18:44:22 +00:00
martinsumner
c92107e4b4 2i order of events 
When running a load of mainly 2i queries, there is a huge cost in the
previous snapshot code.  The time taken to create a clone of the
Penciller (duplicating all the LoopState) varied between 1 and 200ms
depedning on the size of the LoopState.

For 2i queries, most of that LoopState was then being thrown away after
running the query against the levelzero_cache.  This was taking < 1ms on
average.  It would be better to avoid the o(100)ms of CPU  burning and
block for o(1)ms - so th eorder of events have been changed ot filter
first so only the small part of the LoopState actually required is
copied to the clone.
 2017-03-06 18:42:32 +00:00
Martin Sumner
c1dc92720c Random, random, random 
well random had me foxed.  As the clone was short-lived process it only
called random once - and so always got the same answer.

random has to be seeded to give different answers when called once from
a process - so this is now seeded in leveed_log
 2017-03-06 13:51:38 +00:00
martinsumner
eb6f668fcd Use log at random 
Easy way to sample frequent things - especially when they'r ein ocverag
equeries
 2017-03-06 10:34:56 +00:00
martinsumner
5c2f05858d Alter logging to help understand performance factors 
Change logging of the snapshots to better understand performance
 2017-03-06 10:17:51 +00:00
martinsumner
fad9bfbff1 Trigger log points, deprecate lower PUT log points 
Only record PUT timings at the Bookie, and trigger them correctly on the
right log point
 2017-02-26 19:07:55 +00:00
martinsumner
adb78f5c5a Resolve pclerk crash 
Need to add extra logging to understand why pclerk crashes in some
volume tests.

Penciller's Clerk <0.813.0> shutdown now complete for reason
{badarg,[{dict,fetch,[63,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,[100,105,99,116,46,101,114,108]},{line,126}]},{leveled_pclerk,handle_cast,2,[{file,[115,114,99,47,108,101,118,101,108,101,100,95,112,99,108,101,114,107,46,101,114,108]},{line,96}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,604}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,239}]}]}

Should only be prompted after prompt deletions had bene updated.
Perhaps a race whereby somehow it is prompted again after it is emptied.
 2017-02-09 14:33:39 +00:00
martinsumner
d57b74d967 Re-introduce tinybloom to SST 
This had been removed due to the CPU cost of adding - however then the
tinybloom wa simplemented by directly manipulating bits through binary
comprehension - rather than applying bor band bsl bsr operations.

With these operations the cost of producing and checking the bloom is
<10% by comparison.
 2017-01-24 21:51:12 +00:00
martinsumner
861cedf45e Add back missed space in logs 2017-01-23 19:16:17 +00:00
martinsumner
fb896f13b1 Improve logging - add timestamp to logs 2017-01-23 18:56:01 +00:00
martinsumner
90c920fe86 Additional unit test work 
Reverts a previous ct test fix
 2017-01-23 15:15:40 +00:00
Martin Sumner
2c4c5c9597 Corrections 
Support an empty list of entries being added.  Also specify a tree
correctly in all from_orderedlist scenarios
 2017-01-23 00:22:53 +00:00
martinsumner
14ebf94e56 Correct botched manifest entry removal 2017-01-17 10:53:13 +00:00
martinsumner
5076187545 Reduce log level of troubleshooting log 2017-01-17 10:39:31 +00:00
martinsumner
5adfb5c5ef Extra logging for troubleshooting 2017-01-17 10:32:15 +00:00
martinsumner
e6c4c9eff8 Log deletions from mnaifest (via GC) 2017-01-15 11:04:26 +00:00
martinsumner
13c5cc8899 Add timing points 
Add some timing points to manifest updates
 2017-01-15 10:36:50 +00:00