umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1594 commits 1 branch 0 tags 25 MiB
Commit graph

1086 commits

Author SHA1 Message Date
martinsumner
c787e0cd78 Handle corrupted Ledger Key when applying recovery strategy 
Otherwise may blow up in journal_compaction_bustedjournal test
 2017-03-13 14:32:46 +00:00
martinsumner
f3e962c43a Add level to SST slow fetch log 2017-03-13 12:16:36 +00:00
martinsumner
62c3ba8b6f Passing ETS reference not tree 
ETS reference gets converted to tree by Penciller
 2017-03-13 11:54:46 +00:00
martinsumner
f2cd9b3f33 Consistency of empty slotlist references 
Need to return an empty slotlist in a consistent way
 2017-03-11 13:04:55 +00:00
martinsumner
1f8de798bd Fix empty slot issue 2017-03-11 12:41:30 +00:00
martinsumner
a07770a3df Unit tets of lookup over-size issue 
A mistake meant resetting to lookup on a skipped key would cause issues
if the skipped key ocurred under a no_lookup slot after the ?SLOT_SIZE
had been reached.  This caused the slot to switch to lookup, but beyond
the maximum size
 2017-03-11 00:03:55 +00:00
martinsumner
4e4f498f20 Correctly set no_lookup on skip_key 
Otherwise could change to lookup after the size limit has been reached
 2017-03-10 23:48:17 +00:00
martinsumner
1813317121 Correctly identify empty slotlist 2017-03-10 22:49:00 +00:00
martinsumner
b2f3d882a9 Draft of branch to condense range_only keys 2017-03-10 20:43:37 +00:00
martinsumner
730ab2ec48 tidy out io:format 2017-03-10 11:10:15 +00:00
martinsumner
601f43de3d Merge remote-tracking branch 'refs/remotes/origin/master' into mas-sstblock-i42 2017-03-10 10:24:51 +00:00
martinsumner
d7eee2f9c9 Remove rogue log 2017-03-09 22:24:11 +00:00
martinsumner
39a005a8d0 Try and be consistent in flilename format 
i.e. ./filename.sst
 2017-03-09 21:52:29 +00:00
martinsumner
bfcf981485 Correct root path setting in pclerk 2017-03-09 21:32:36 +00:00
martinsumner
4c59342600 Change SST reference to split filename 
The manifest and the logs are bloated by having the full file path for
every filename in there - given the root path is constant.

Could also cause issues if the mount point is ever changed.
 2017-03-09 21:23:09 +00:00
martinsumner
3b2d8691b2 Make sure selection is just the manifest entry 2017-03-08 15:10:08 +00:00
martinsumner
90b587b083 Allow manifest entry to be selected from level 2 2017-03-08 12:09:10 +00:00
martinsumner
04cfb453c4 Fetch specific block only 
Rely on CRC check in zlib.  Still need to catch on failure
 2017-03-07 20:19:11 +00:00
martinsumner
cd82bba6cd Use cache size - don't recalc length 2017-03-06 21:44:20 +00:00
martinsumner
9ad6969b0d Seed randomnes at Actor startup 2017-03-06 21:35:02 +00:00
martinsumner
87f2c5d7ae Merge remote-tracking branch 'origin/mas-2iphase2-i34' into mas-2iphase2-i34 
# Conflicts:
#	src/leveled_log.erl
 2017-03-06 18:44:22 +00:00
martinsumner
c92107e4b4 2i order of events 
When running a load of mainly 2i queries, there is a huge cost in the
previous snapshot code.  The time taken to create a clone of the
Penciller (duplicating all the LoopState) varied between 1 and 200ms
depedning on the size of the LoopState.

For 2i queries, most of that LoopState was then being thrown away after
running the query against the levelzero_cache.  This was taking < 1ms on
average.  It would be better to avoid the o(100)ms of CPU  burning and
block for o(1)ms - so th eorder of events have been changed ot filter
first so only the small part of the LoopState actually required is
copied to the clone.
 2017-03-06 18:42:32 +00:00
Martin Sumner
c1dc92720c Random, random, random 
well random had me foxed.  As the clone was short-lived process it only
called random once - and so always got the same answer.

random has to be seeded to give different answers when called once from
a process - so this is now seeded in leveed_log
 2017-03-06 13:51:38 +00:00
martinsumner
eb6f668fcd Use log at random 
Easy way to sample frequent things - especially when they'r ein ocverag
equeries
 2017-03-06 10:34:56 +00:00
martinsumner
94a39e4c4d Fix double logging of B0004 2017-03-06 10:27:48 +00:00
martinsumner
5c2f05858d Alter logging to help understand performance factors 
Change logging of the snapshots to better understand performance
 2017-03-06 10:17:51 +00:00
martinsumner
bc5388710b Update SST comments 2017-03-04 20:47:46 +00:00
martinsumner
426f5df75c Fix dialyzer warnings 2017-03-02 21:24:40 +00:00
martinsumner
30e23fe514 Add no_lookup feature to snapshots 2017-03-02 18:23:47 +00:00
martinsumner
b01f7d23df Add pre-filter for 2i queries 2017-03-02 17:49:43 +00:00
martinsumner
480a029135 Renaming 
Renaming to leveled

Would prefer to call it PontusDB - as a database optimised for HEAD
requests.  But our heroes often leave us.
 2017-02-28 01:38:47 +00:00
martinsumner
35b6e16590 Clear dialyzer warnings 2017-02-27 20:23:36 +00:00
martinsumner
fce7cbd2ef Add sync_strategy comments 2017-02-27 17:32:38 +00:00
martinsumner
2db28c3028 Tidy line length 2017-02-27 12:24:26 +00:00
martinsumner
8d3f4bd4dc Comment API on Bookie 
Make it easier to understand how to work the store
 2017-02-27 11:58:16 +00:00
martinsumner
27fdbf2ac8 Syntax 2017-02-26 22:46:57 +00:00
martinsumner
a9101e4781 SibCount must be non-zero 2017-02-26 22:45:20 +00:00
martinsumner
44471ecc0a ManEntry must be a ManEntry? 
What else could it be?
 2017-02-26 22:42:44 +00:00
martinsumner
aba385869f Test long running clause 2017-02-26 22:41:25 +00:00
martinsumner
4e6c22d7e6 Dumb test 
Can't test for slow not_present outside of simple unit test ... will be
very rare event
 2017-02-26 21:48:04 +00:00
martinsumner
19534122a2 Coverage checks 2017-02-26 21:37:47 +00:00
martinsumner
100d675472 iManifest -test empty journal 2017-02-26 21:13:07 +00:00
martinsumner
41e5686a66 Test coverage - missing 1% 2017-02-26 20:52:40 +00:00
martinsumner
fb54d29873 Timestamps no longer required 2017-02-26 19:09:56 +00:00
martinsumner
fad9bfbff1 Trigger log points, deprecate lower PUT log points 
Only record PUT timings at the Bookie, and trigger them correctly on the
right log point
 2017-02-26 19:07:55 +00:00
martinsumner
8077c70486 More conservative approach to ongoing work monitoring 
As per comments though - if we auto-restart pclerk in the future this
will have to be re-considered.

Perhaps a re-starting pclerk should force some reset of this boolean on
startup perhaps by making a different work_for_clerk if in a virgin
state.
 2017-02-09 23:49:12 +00:00
martinsumner
793977b76c Prevent double work ongoing 
This was the cause of the pclerk crash.  The concurrent work would lead
to the attempt to re-fetch the prompted deletions after they had been
deleted.
 2017-02-09 23:41:28 +00:00
martinsumner
05b2d8faaf is_empty not in OTP16 
bloody OTP16 strikes again - is_empty isn't there
 2017-02-09 21:12:01 +00:00
martinsumner
adb78f5c5a Resolve pclerk crash 
Need to add extra logging to understand why pclerk crashes in some
volume tests.

Penciller's Clerk <0.813.0> shutdown now complete for reason
{badarg,[{dict,fetch,[63,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,[100,105,99,116,46,101,114,108]},{line,126}]},{leveled_pclerk,handle_cast,2,[{file,[115,114,99,47,108,101,118,101,108,101,100,95,112,99,108,101,114,107,46,101,114,108]},{line,96}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,604}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,239}]}]}

Should only be prompted after prompt deletions had bene updated.
Perhaps a race whereby somehow it is prompted again after it is emptied.
 2017-02-09 14:33:39 +00:00
martinsumner
7320b34681 Comment update 2017-01-25 12:38:33 +00:00