umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1638 commits 1 branch 0 tags 25 MiB
Commit graph

1638 commits

This branch
This branch
All branches
Author SHA1 Message Date
martinsumner
f96d148073 Make the merge_test a more sensible size 
On the verge of a timeout.  Rather than keep battling with the timeout,
make it do less work
 2016-12-11 20:17:05 +00:00
martinsumner
5cfe9a71e1 Wrap test with non-default timeout 2016-12-11 15:25:14 +00:00
martinsumner
24a5347bec Revert 2016-12-11 15:19:34 +00:00
martinsumner
a86686d621 Remove unnecessary reverse 2016-12-11 15:17:58 +00:00
martinsumner
1b63845050 Bring compression back to SFT 
It is expensive on the CPU - but it leads to a 4 x increase in the cache
coverage.

Try and make some small micro gains in list handling in create_block
 2016-12-11 15:02:33 +00:00
martinsumner
44cee5a6e8 Experiemnt with no compression 
Does compression hurt CPU more than the benefit gaine din some cases?
 2016-12-11 12:33:09 +00:00
martinsumner
71cf7a3a51 Setting change led to idle CPU 2016-12-11 08:37:03 +00:00
martinsumner
fb069666dc Export module 2016-12-11 08:16:00 +00:00
martinsumner
16c704551b Revert to original SFT build settings 
Leveled is always CPU bound during tests, and it is the merge in the
ledger that drains the CPU hardest,
 2016-12-11 07:35:23 +00:00
martinsumner
6f06c6fdeb ETS delete 
Delete the objects rather than starting a new table each time
 2016-12-11 07:07:30 +00:00
martinsumner
2758498fad More Jitter! 
Having reduced the size of the ledger cache (again) we can now tolerate
more jitter here
 2016-12-11 06:54:41 +00:00
martinsumner
32ac305c67 Compaction test error 
Compaction tests now throwing up different corruption points
 2016-12-11 06:53:25 +00:00
martinsumner
8bcb49479d Re-introduce ETS Index 
Add ETS Index back in to avoid having to check each skip list in turn.
Also this helps keep a lower skip list size.
 2016-12-11 05:23:24 +00:00
martinsumner
f848500eff Tinker, tinker, tinker, tinker 2016-12-11 04:53:36 +00:00
martinsumner
523716e8f2 Add tiny bloom to Penciller Manifest 
This is an attempt to save on unnecessary message transfers, and
slightly more expensive GCS checks in the SFT file itself.
 2016-12-11 04:48:50 +00:00
martinsumner
ea8f3c07a7 oops 2016-12-11 02:00:19 +00:00
martinsumner
2c7fdc74d4 Setting fiddling 
Try to find a happy medium
 2016-12-11 01:58:25 +00:00
martinsumner
5d11bc051f Allow for more fluctuation in L0 write time 
Try to alleviate existing co-ordination issue when all vnodes tend to
try and write L0 files concurrently
 2016-12-11 01:49:03 +00:00
martinsumner
1f38bcb328 Magic Hash vs phash2 
Magic Hash broke Skip List organisation
 2016-12-11 01:32:32 +00:00
martinsumner
ccc993383d Stop second hash on fetch_head 
The bookie should magic_hash for fetch_head, and now passes the hash to
the Penciller so second hash not required.
 2016-12-11 01:21:53 +00:00
martinsumner
2d3a40e6f1 Magic Hash - and no L0 Index 
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).

The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.

It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest.  This will then reduce the message passing required for a GET
not in the cache or higher levels
 2016-12-11 01:02:56 +00:00
martinsumner
95d5e12ce7 Switch to using ets set as index of L0 cache 
Hope is that this will cause less garbage collection, and also will be
slightly faster.

Note that snapshots don't now get an index - they get the special index
'snap'.  However, the SkipLists have bloom protection, and most
snapshots are iterators not fetchers.
 2016-12-10 14:15:35 +00:00
martinsumner
06c58bf84b Split out hashtree implementation 
Split out hashtree implementation functions in leveled_cdb to make it
easier to swap this out.  Currently using an array of skiplists - may be
better with an ets ordered_set
 2016-12-10 13:03:38 +00:00
martinsumner
c4e4cf67fe Add bloom to loaded skiplist 2016-12-10 11:39:00 +00:00
martinsumner
626a8e63f9 Experiment converting CDB to use skiplist not gb_tree 
Might insertion time be faster?
 2016-12-10 10:55:35 +00:00
martinsumner
a3f60e3609 OTP version shenanigans 2016-12-09 18:55:13 +00:00
martinsumner
d2bd01eaf1 Add fast fail to skiplist 
Add a bloom filter to the skiplist, to make it faster at returning not
found.  The SkipList is now encapsulated within a dict().
 2016-12-09 18:30:40 +00:00
martinsumner
f0db730f07 Adjust jitter settings 2016-12-09 16:34:15 +00:00
martinsumner
82cb49638a Attempt at performance improvement 
Try to add some extra jitter in to the process of L0 writes, and also
make L0 writes delayed to help with bufferring
 2016-12-09 14:36:03 +00:00
martinsumner
349d194a7c Increase jitter slightly 2016-12-09 09:52:31 +00:00
martinsumner
5bdb7fd7fa Alter Riak HEAD 
Change the extract of Riak metadata.

In Riak-based volume tests hte writing of SFT files is tanking.  Could
this be the "extra" metadata.  i.e. There are only current plans to look
at the vclock.  Sibling count is free to fetch, what if we just get
these two items, will it be less CPU to extract the metadata, but also
will the reduced weight reduce the downstream impact?
 2016-12-08 23:38:50 +00:00
martinsumner
2f4013f430 Set Jitter correctly this time 2016-12-08 21:02:39 +00:00
martinsumner
b03ef664c8 Experiment with SFT settings 2016-12-08 20:52:17 +00:00
martinsumner
f07ab85a81 Oops 2016-12-08 18:36:57 +00:00
martinsumner
34e7ce170a Jitter co-ordination issues 
Experiment to try and reoslve jitter co-orsination issue
 2016-12-08 18:35:20 +00:00
martinsumner
ac45f95559 Investigate jitter coordination 2016-12-08 18:30:45 +00:00
martinsumner
c40e5d2d30 Reduce log noise 2016-12-08 16:43:35 +00:00
martinsumner
e2af60f72b Add cache size jitter 
First pan-riak volume requests showed coordination issues
 2016-12-08 16:38:44 +00:00
martinsumner
441fff13e5 Merge pull request #6 from martinsumner/OTP16 
OTP16
 2016-12-03 15:01:28 +00:00
martinsumner
3417baa3b8 Simple test 
To try and pinpoint any issue with _int index (as seen in Riak
integrtaion testing)
 2016-12-02 17:39:28 +00:00
Martin Sumner
d07cad34b7 Why lists reverse? 
It doesn’t make sense as the Acc may not be a list.  Not failing any
tests.
 2016-12-02 00:40:00 +00:00
martinsumner
3ec144d33d Remove unnecessary branch 2016-12-01 11:31:25 +00:00
martinsumner
dd0e9de050 Edit comment 2016-11-30 23:24:12 +00:00
martinsumner
c0384d7a86 Extend small skiplist test further 2016-11-30 23:23:10 +00:00
martinsumner
39f61db92b Further testing of skiplists 
Suspected at some stage there was a problem with small lists that never
hit a marker.  Can't find it yet through test
 2016-11-30 23:21:39 +00:00
martinsumner
364527f3b8 Add potential support for deeper layers 
Unproven attempt to make skiplist code more generic
 2016-11-30 22:35:23 +00:00
martinsumner
743d59c71b Make SkipList Two-Level 
Read and write times were increasing as the siz eof skiplist got larger
(e.g. over 2000).  Tried instead with a smaller skip width and a
two-tier skiplist and this gave much more regular eprformance as the
size of the skiplist went up.
 2016-11-30 15:36:27 +00:00
martinsumner
e3783485de Minor SkipList speed-up 2016-11-29 11:13:08 +00:00
martinsumner
dd6201b34b Try to avoid crashing on invalid key length 
May occurr in corrupted files.
 2016-11-29 00:27:23 +00:00
martinsumner
b62b38caa6 Tidy up test at end 
May cause issues otherwise if there is an attempt to corrupt the journal
running in another bookie
 2016-11-28 23:00:12 +00:00