umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1638 commits 1 branch 0 tags 25 MiB
Commit graph

1638 commits

This branch
This branch
All branches
Author SHA1 Message Date
martinsumner
05ddcadbf9 Merge pull request #13 from martinsumner/mas-staggerhashtreewrite 
Stopped unnecessary seek for last_key
 2016-12-22 21:34:06 +00:00
martinsumner
0ddaaf9ac3 Stopped unnecessary seek for last_key 
When rolling we already know the last_key - no need to seek for it on
startup.

The time it takes for this seek needs to be considered with regards to
startup time.  Can we do without knowing lastkey?
 2016-12-22 19:51:39 +00:00
martinsumner
44cf5788ab Merge pull request #12 from martinsumner/mas-puttiming 
Timing Points
 2016-12-22 18:07:28 +00:00
martinsumner
a131b99082 Randomising logging of PUT timings 2016-12-22 17:33:14 +00:00
martinsumner
353fb08e21 Randomise logging of GET/HEAD samples 2016-12-22 17:28:41 +00:00
martinsumner
ee534081c3 Reduce log levels 
Remove some log noise to debug level
 2016-12-22 17:15:42 +00:00
martinsumner
151dd3ab70 Sample only for HEAD/GET response times 
Report regularly but only on a sample
 2016-12-22 16:47:36 +00:00
Martin Sumner
676e8fa494 Add Get Timing 2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909 put-timing default 
Remove need for individual actors to know the defaults for put_timing
tuple
 2016-12-22 14:41:43 +00:00
Martin Sumner
e9e0a7b323 Set higher logpoint 
Expectation is for many HEAD requests - so only log every 100K
 2016-12-22 14:36:57 +00:00
martinsumner
df350e1e6f Add unit test for head timing 2016-12-22 14:09:17 +00:00
martinsumner
130fb36ddd Add head timings 
Include log breaking down timings of HEAD requests by result and level
 2016-12-22 14:03:31 +00:00
martinsumner
ea20fc07f4 Maybe not 2016-12-21 21:56:33 +00:00
martinsumner
39d634c95b And again 2016-12-21 21:49:08 +00:00
martinsumner
3de146043b Its not like there's more than two hard things 
D'oh
 2016-12-21 21:41:54 +00:00
martinsumner
b2835aeaec Improve fetching efficiency 
Experiment to see if parsing all keys in block can be avoided - and if
so does this make the range scan more efficient.

Unproven change.
 2016-12-21 18:28:14 +00:00
martinsumner
be775127e8 Improve logging of merge activity and slow GETs 
Look into speculation that collisions between fetch_rnage and fetch may
be an issue
 2016-12-21 12:45:27 +00:00
martinsumner
f3e16dcd10 Add long-running logs 2016-12-21 01:56:12 +00:00
martinsumner
c193962c92 Sort out different timestamps 2016-12-20 23:16:52 +00:00
martinsumner
060ce2e263 Add put timing points 2016-12-20 23:11:50 +00:00
martinsumner
299e8e6de3 Initial phash test 
phash does not appear to be a potential causer of delay
 2016-12-20 20:55:56 +00:00
martinsumner
2a8e1afe41 Merge pull request #11 from martinsumner/mas-journalcorruptfail 
Resolve failing recovery test
 2016-12-16 23:58:37 +00:00
martinsumner
9e28287231 Resolve failing recovery test 
Now passing consistently with a number of different corruptions catered
for (including corruption of the Tag in the Inker Key)
 2016-12-16 23:18:55 +00:00
martinsumner
1684f0a913 Merge pull request #10 from martinsumner/mas_riakmetadata 
Full Riak Metadata
 2016-12-15 11:59:14 +00:00
martinsumner
4798dc1148 Write block theory 
With riak metadata the sft files are about 2MB - so a group write coun
of 64 is trying to write in 500 KB chunks.  Is this too big?
 2016-12-14 22:03:38 +00:00
Martin Sumner
b92b511166 Revert "Experiment" 
This reverts commit fe907eb479.
 2016-12-14 18:49:47 +00:00
martinsumner
fe907eb479 Experiment 
Perhaps the results ar epaging related - change settings to hold less in
memory and see
 2016-12-14 11:38:42 +00:00
martinsumner
f4e2e274e0 Reintroduce riak metadata extraction 
The full riak metadata had been stripped from the Ledger update for
performance reasons.  However, the full metadata is required in order to
save a GET before a PUT.  Therefore we want to do isolated testing on
this change to establish the relative cost value in that cost saving.
 2016-12-14 10:27:11 +00:00
martinsumner
bb1221a918 Merge pull request #9 from martinsumner/mas-addpclbloom 
Mas addpclbloom
 2016-12-14 10:11:18 +00:00
Martin Sumner
5efed94a1e Try slightly larger cache 2016-12-13 22:29:55 +00:00
martinsumner
b2c36ae541 Merge remote-tracking branch 'refs/remotes/origin/master' into mas-addpclbloom 2016-12-13 21:04:15 +00:00
martinsumner
baf4ca252f Revert "Experiment with temporary us eof ETS table" 
This reverts commit 2a106d0dc5.
 2016-12-13 20:24:29 +00:00
martinsumner
2a106d0dc5 Experiment with temporary us eof ETS table 
Rather than expensive lists:ukeymerge, try use a temporary ETS table.
 2016-12-13 19:38:14 +00:00
martinsumner
bc5190a9bd Merge pull request #8 from martinsumner/mas-cdb-hashtree-refactor 
Mas cdb hashtree refactor
 2016-12-13 18:48:37 +00:00
martinsumner
c8be3bfa46 Slot hash corrected 
When building the hashtree the incorrect IndexLength was being used to
calculate the slot - causing many queries to loop all the way round the
Index
 2016-12-13 17:02:45 +00:00
martinsumner
8f775a88fd Investigate performance regression 
Performance has regressed following the hashtable change.  Speculation
that the hashtable format might not be right, and so there is more
cycling around the hashtree.  Logging added.
 2016-12-13 14:06:19 +00:00
martinsumner
52499170c0 Tidy logging following changes 
Include detailed timings in a permanent log
 2016-12-13 12:41:44 +00:00
Martin Sumner
cfc6a67638 Switch to ordered_set 
Improved performance by a combination of switching to an ordered_set
(so a list can be extracted in a sane way), and building the binary
from an ordered list.
 2016-12-13 12:35:30 +00:00
martinsumner
aa2d19df1d Revert back to handling list of binaries (but differently) 
Performance from last commit got worse not better :-(

Perhaps better handling all as lists, and then building a binary at the
end.
 2016-12-13 03:22:40 +00:00
martinsumner
972a0ee0b9 Refactor hash table write 
Less looping and re-looping over list.  Uses ordering to build more
naturally.
 2016-12-13 02:15:13 +00:00
martinsumner
52e21de298 Initial switch to using ETS 
No real refactor of building hashtables at this stage - just using ETS
not an arrary of skiplists
 2016-12-12 21:47:09 +00:00
martinsumner
8ccd02e893 Merge Tree issue 
The attempt to refcator the writer meant that files were never reaching
the max slots - and so we were only ever stopping when the lists were
exhausted.  This meant that the merge tree just had a C0 and a C1 file!
 2016-12-12 18:30:12 +00:00
martinsumner
ff0bf15c8f Fix the fix 2016-12-12 18:18:37 +00:00
martinsumner
1537334fbd Ensure fetch still works when delete_pending 2016-12-12 18:17:53 +00:00
martinsumner
cf6a1eb513 Add extra bloom check 
Add extra bloom check - but get the SFT process to perform not the chekc
not the Penciller.  This avoids complexity of negotiating the transfer
of the bloom to the Penciller - but doesn't avoid the potentially
unecessary message pass between processes.
 2016-12-12 18:01:37 +00:00
martinsumner
1f56501499 Refactor writing SFT Files 
Previously the code had involved veyr high arity functions which were
hard to follow.  This has been simplified somewhat with the addition of
a writer record to make things easier to track, as well as a general
refactoring to better logically seperate the building of things.
 2016-12-12 16:12:31 +00:00
martinsumner
addd4c89d0 Merge pull request #7 from martinsumner/mas-stripmeta 
Mas stripmeta
 2016-12-11 22:17:03 +00:00
martinsumner
f28c7e02bf Remove unnecessary clause 
As the intention is to change the way the tiny bloom is called, the
unnecessary clause of handling an undefined bloom can be removed.
 2016-12-11 21:24:04 +00:00
martinsumner
86bdfdeaf0 Reverted back out the additional bloom check 
This is desirable to add back in going forward, but wasn't implemented
in a safe or clear way.

The way the bloom was or was not on the LoopState was clumsy, and it got
persisted in multiple places without a CRC check.

Intention to implement back in wherby it is requested on-demand by the
Penciller, and then the SFT worker lifts it off disk and CRC checks it.
So it is never on the SFT LoopState.  Also it will be easier to control
the logic over which levels have the bloom in the Penciller.
 2016-12-11 21:01:10 +00:00
martinsumner
4b48ed14c6 Correct Mistyped 2 ^ 32 2016-12-11 20:38:20 +00:00