umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
374 commits 1 branch 0 tags 25 MiB
Commit graph

374 commits

This branch
This branch
All branches
Author SHA1 Message Date
martinsumner
3de146043b Its not like there's more than two hard things 
D'oh
 2016-12-21 21:41:54 +00:00
martinsumner
b2835aeaec Improve fetching efficiency 
Experiment to see if parsing all keys in block can be avoided - and if
so does this make the range scan more efficient.

Unproven change.
 2016-12-21 18:28:14 +00:00
martinsumner
be775127e8 Improve logging of merge activity and slow GETs 
Look into speculation that collisions between fetch_rnage and fetch may
be an issue
 2016-12-21 12:45:27 +00:00
martinsumner
f3e16dcd10 Add long-running logs 2016-12-21 01:56:12 +00:00
martinsumner
c193962c92 Sort out different timestamps 2016-12-20 23:16:52 +00:00
martinsumner
060ce2e263 Add put timing points 2016-12-20 23:11:50 +00:00
martinsumner
299e8e6de3 Initial phash test 
phash does not appear to be a potential causer of delay
 2016-12-20 20:55:56 +00:00
martinsumner
2a8e1afe41 Merge pull request #11 from martinsumner/mas-journalcorruptfail 
Resolve failing recovery test
 2016-12-16 23:58:37 +00:00
martinsumner
9e28287231 Resolve failing recovery test 
Now passing consistently with a number of different corruptions catered
for (including corruption of the Tag in the Inker Key)
 2016-12-16 23:18:55 +00:00
martinsumner
1684f0a913 Merge pull request #10 from martinsumner/mas_riakmetadata 
Full Riak Metadata
 2016-12-15 11:59:14 +00:00
martinsumner
4798dc1148 Write block theory 
With riak metadata the sft files are about 2MB - so a group write coun
of 64 is trying to write in 500 KB chunks.  Is this too big?
 2016-12-14 22:03:38 +00:00
Martin Sumner
b92b511166 Revert "Experiment" 
This reverts commit fe907eb479.
 2016-12-14 18:49:47 +00:00
martinsumner
fe907eb479 Experiment 
Perhaps the results ar epaging related - change settings to hold less in
memory and see
 2016-12-14 11:38:42 +00:00
martinsumner
f4e2e274e0 Reintroduce riak metadata extraction 
The full riak metadata had been stripped from the Ledger update for
performance reasons.  However, the full metadata is required in order to
save a GET before a PUT.  Therefore we want to do isolated testing on
this change to establish the relative cost value in that cost saving.
 2016-12-14 10:27:11 +00:00
martinsumner
bb1221a918 Merge pull request #9 from martinsumner/mas-addpclbloom 
Mas addpclbloom
 2016-12-14 10:11:18 +00:00
Martin Sumner
5efed94a1e Try slightly larger cache 2016-12-13 22:29:55 +00:00
martinsumner
b2c36ae541 Merge remote-tracking branch 'refs/remotes/origin/master' into mas-addpclbloom 2016-12-13 21:04:15 +00:00
martinsumner
baf4ca252f Revert "Experiment with temporary us eof ETS table" 
This reverts commit 2a106d0dc5.
 2016-12-13 20:24:29 +00:00
martinsumner
2a106d0dc5 Experiment with temporary us eof ETS table 
Rather than expensive lists:ukeymerge, try use a temporary ETS table.
 2016-12-13 19:38:14 +00:00
martinsumner
bc5190a9bd Merge pull request #8 from martinsumner/mas-cdb-hashtree-refactor 
Mas cdb hashtree refactor
 2016-12-13 18:48:37 +00:00
martinsumner
c8be3bfa46 Slot hash corrected 
When building the hashtree the incorrect IndexLength was being used to
calculate the slot - causing many queries to loop all the way round the
Index
 2016-12-13 17:02:45 +00:00
martinsumner
8f775a88fd Investigate performance regression 
Performance has regressed following the hashtable change.  Speculation
that the hashtable format might not be right, and so there is more
cycling around the hashtree.  Logging added.
 2016-12-13 14:06:19 +00:00
martinsumner
52499170c0 Tidy logging following changes 
Include detailed timings in a permanent log
 2016-12-13 12:41:44 +00:00
Martin Sumner
cfc6a67638 Switch to ordered_set 
Improved performance by a combination of switching to an ordered_set
(so a list can be extracted in a sane way), and building the binary
from an ordered list.
 2016-12-13 12:35:30 +00:00
martinsumner
aa2d19df1d Revert back to handling list of binaries (but differently) 
Performance from last commit got worse not better :-(

Perhaps better handling all as lists, and then building a binary at the
end.
 2016-12-13 03:22:40 +00:00
martinsumner
972a0ee0b9 Refactor hash table write 
Less looping and re-looping over list.  Uses ordering to build more
naturally.
 2016-12-13 02:15:13 +00:00
martinsumner
52e21de298 Initial switch to using ETS 
No real refactor of building hashtables at this stage - just using ETS
not an arrary of skiplists
 2016-12-12 21:47:09 +00:00
martinsumner
8ccd02e893 Merge Tree issue 
The attempt to refcator the writer meant that files were never reaching
the max slots - and so we were only ever stopping when the lists were
exhausted.  This meant that the merge tree just had a C0 and a C1 file!
 2016-12-12 18:30:12 +00:00
martinsumner
ff0bf15c8f Fix the fix 2016-12-12 18:18:37 +00:00
martinsumner
1537334fbd Ensure fetch still works when delete_pending 2016-12-12 18:17:53 +00:00
martinsumner
cf6a1eb513 Add extra bloom check 
Add extra bloom check - but get the SFT process to perform not the chekc
not the Penciller.  This avoids complexity of negotiating the transfer
of the bloom to the Penciller - but doesn't avoid the potentially
unecessary message pass between processes.
 2016-12-12 18:01:37 +00:00
martinsumner
1f56501499 Refactor writing SFT Files 
Previously the code had involved veyr high arity functions which were
hard to follow.  This has been simplified somewhat with the addition of
a writer record to make things easier to track, as well as a general
refactoring to better logically seperate the building of things.
 2016-12-12 16:12:31 +00:00
martinsumner
addd4c89d0 Merge pull request #7 from martinsumner/mas-stripmeta 
Mas stripmeta
 2016-12-11 22:17:03 +00:00
martinsumner
f28c7e02bf Remove unnecessary clause 
As the intention is to change the way the tiny bloom is called, the
unnecessary clause of handling an undefined bloom can be removed.
 2016-12-11 21:24:04 +00:00
martinsumner
86bdfdeaf0 Reverted back out the additional bloom check 
This is desirable to add back in going forward, but wasn't implemented
in a safe or clear way.

The way the bloom was or was not on the LoopState was clumsy, and it got
persisted in multiple places without a CRC check.

Intention to implement back in wherby it is requested on-demand by the
Penciller, and then the SFT worker lifts it off disk and CRC checks it.
So it is never on the SFT LoopState.  Also it will be easier to control
the logic over which levels have the bloom in the Penciller.
 2016-12-11 21:01:10 +00:00
martinsumner
4b48ed14c6 Correct Mistyped 2 ^ 32 2016-12-11 20:38:20 +00:00
martinsumner
f96d148073 Make the merge_test a more sensible size 
On the verge of a timeout.  Rather than keep battling with the timeout,
make it do less work
 2016-12-11 20:17:05 +00:00
martinsumner
5cfe9a71e1 Wrap test with non-default timeout 2016-12-11 15:25:14 +00:00
martinsumner
24a5347bec Revert 2016-12-11 15:19:34 +00:00
martinsumner
a86686d621 Remove unnecessary reverse 2016-12-11 15:17:58 +00:00
martinsumner
1b63845050 Bring compression back to SFT 
It is expensive on the CPU - but it leads to a 4 x increase in the cache
coverage.

Try and make some small micro gains in list handling in create_block
 2016-12-11 15:02:33 +00:00
martinsumner
44cee5a6e8 Experiemnt with no compression 
Does compression hurt CPU more than the benefit gaine din some cases?
 2016-12-11 12:33:09 +00:00
martinsumner
71cf7a3a51 Setting change led to idle CPU 2016-12-11 08:37:03 +00:00
martinsumner
fb069666dc Export module 2016-12-11 08:16:00 +00:00
martinsumner
16c704551b Revert to original SFT build settings 
Leveled is always CPU bound during tests, and it is the merge in the
ledger that drains the CPU hardest,
 2016-12-11 07:35:23 +00:00
martinsumner
6f06c6fdeb ETS delete 
Delete the objects rather than starting a new table each time
 2016-12-11 07:07:30 +00:00
martinsumner
2758498fad More Jitter! 
Having reduced the size of the ledger cache (again) we can now tolerate
more jitter here
 2016-12-11 06:54:41 +00:00
martinsumner
32ac305c67 Compaction test error 
Compaction tests now throwing up different corruption points
 2016-12-11 06:53:25 +00:00
martinsumner
8bcb49479d Re-introduce ETS Index 
Add ETS Index back in to avoid having to check each skip list in turn.
Also this helps keep a lower skip list size.
 2016-12-11 05:23:24 +00:00
martinsumner
f848500eff Tinker, tinker, tinker, tinker 2016-12-11 04:53:36 +00:00