umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
274 commits 1 branch 0 tags 25 MiB
Commit graph

274 commits

This branch
This branch
All branches
Author SHA1 Message Date
martinsumner
523716e8f2 Add tiny bloom to Penciller Manifest 
This is an attempt to save on unnecessary message transfers, and
slightly more expensive GCS checks in the SFT file itself.
 2016-12-11 04:48:50 +00:00
martinsumner
ea8f3c07a7 oops 2016-12-11 02:00:19 +00:00
martinsumner
2c7fdc74d4 Setting fiddling 
Try to find a happy medium
 2016-12-11 01:58:25 +00:00
martinsumner
5d11bc051f Allow for more fluctuation in L0 write time 
Try to alleviate existing co-ordination issue when all vnodes tend to
try and write L0 files concurrently
 2016-12-11 01:49:03 +00:00
martinsumner
1f38bcb328 Magic Hash vs phash2 
Magic Hash broke Skip List organisation
 2016-12-11 01:32:32 +00:00
martinsumner
ccc993383d Stop second hash on fetch_head 
The bookie should magic_hash for fetch_head, and now passes the hash to
the Penciller so second hash not required.
 2016-12-11 01:21:53 +00:00
martinsumner
2d3a40e6f1 Magic Hash - and no L0 Index 
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).

The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.

It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest.  This will then reduce the message passing required for a GET
not in the cache or higher levels
 2016-12-11 01:02:56 +00:00
martinsumner
95d5e12ce7 Switch to using ets set as index of L0 cache 
Hope is that this will cause less garbage collection, and also will be
slightly faster.

Note that snapshots don't now get an index - they get the special index
'snap'.  However, the SkipLists have bloom protection, and most
snapshots are iterators not fetchers.
 2016-12-10 14:15:35 +00:00
martinsumner
06c58bf84b Split out hashtree implementation 
Split out hashtree implementation functions in leveled_cdb to make it
easier to swap this out.  Currently using an array of skiplists - may be
better with an ets ordered_set
 2016-12-10 13:03:38 +00:00
martinsumner
c4e4cf67fe Add bloom to loaded skiplist 2016-12-10 11:39:00 +00:00
martinsumner
626a8e63f9 Experiment converting CDB to use skiplist not gb_tree 
Might insertion time be faster?
 2016-12-10 10:55:35 +00:00
martinsumner
a3f60e3609 OTP version shenanigans 2016-12-09 18:55:13 +00:00
martinsumner
d2bd01eaf1 Add fast fail to skiplist 
Add a bloom filter to the skiplist, to make it faster at returning not
found.  The SkipList is now encapsulated within a dict().
 2016-12-09 18:30:40 +00:00
martinsumner
f0db730f07 Adjust jitter settings 2016-12-09 16:34:15 +00:00
martinsumner
82cb49638a Attempt at performance improvement 
Try to add some extra jitter in to the process of L0 writes, and also
make L0 writes delayed to help with bufferring
 2016-12-09 14:36:03 +00:00
martinsumner
349d194a7c Increase jitter slightly 2016-12-09 09:52:31 +00:00
martinsumner
5bdb7fd7fa Alter Riak HEAD 
Change the extract of Riak metadata.

In Riak-based volume tests hte writing of SFT files is tanking.  Could
this be the "extra" metadata.  i.e. There are only current plans to look
at the vclock.  Sibling count is free to fetch, what if we just get
these two items, will it be less CPU to extract the metadata, but also
will the reduced weight reduce the downstream impact?
 2016-12-08 23:38:50 +00:00
martinsumner
2f4013f430 Set Jitter correctly this time 2016-12-08 21:02:39 +00:00
martinsumner
b03ef664c8 Experiment with SFT settings 2016-12-08 20:52:17 +00:00
martinsumner
f07ab85a81 Oops 2016-12-08 18:36:57 +00:00
martinsumner
34e7ce170a Jitter co-ordination issues 
Experiment to try and reoslve jitter co-orsination issue
 2016-12-08 18:35:20 +00:00
martinsumner
ac45f95559 Investigate jitter coordination 2016-12-08 18:30:45 +00:00
martinsumner
c40e5d2d30 Reduce log noise 2016-12-08 16:43:35 +00:00
martinsumner
e2af60f72b Add cache size jitter 
First pan-riak volume requests showed coordination issues
 2016-12-08 16:38:44 +00:00
martinsumner
441fff13e5 Merge pull request #6 from martinsumner/OTP16 
OTP16
 2016-12-03 15:01:28 +00:00
martinsumner
3417baa3b8 Simple test 
To try and pinpoint any issue with _int index (as seen in Riak
integrtaion testing)
 2016-12-02 17:39:28 +00:00
Martin Sumner
d07cad34b7 Why lists reverse? 
It doesn’t make sense as the Acc may not be a list.  Not failing any
tests.
 2016-12-02 00:40:00 +00:00
martinsumner
3ec144d33d Remove unnecessary branch 2016-12-01 11:31:25 +00:00
martinsumner
dd0e9de050 Edit comment 2016-11-30 23:24:12 +00:00
martinsumner
c0384d7a86 Extend small skiplist test further 2016-11-30 23:23:10 +00:00
martinsumner
39f61db92b Further testing of skiplists 
Suspected at some stage there was a problem with small lists that never
hit a marker.  Can't find it yet through test
 2016-11-30 23:21:39 +00:00
martinsumner
364527f3b8 Add potential support for deeper layers 
Unproven attempt to make skiplist code more generic
 2016-11-30 22:35:23 +00:00
martinsumner
743d59c71b Make SkipList Two-Level 
Read and write times were increasing as the siz eof skiplist got larger
(e.g. over 2000).  Tried instead with a smaller skip width and a
two-tier skiplist and this gave much more regular eprformance as the
size of the skiplist went up.
 2016-11-30 15:36:27 +00:00
martinsumner
e3783485de Minor SkipList speed-up 2016-11-29 11:13:08 +00:00
martinsumner
dd6201b34b Try to avoid crashing on invalid key length 
May occurr in corrupted files.
 2016-11-29 00:27:23 +00:00
martinsumner
b62b38caa6 Tidy up test at end 
May cause issues otherwise if there is an attempt to corrupt the journal
running in another bookie
 2016-11-28 23:00:12 +00:00
martinsumner
e8c1d39df9 Switch to binary format Riak object 
Initial change to try and test assuming that leveled received the binary
format of Riak objects (and parses that for metadata).
 2016-11-28 22:26:09 +00:00
martinsumner
b0a515553f Merge remote-tracking branch 'origin/OTP16' into OTP16 
# Conflicts:
#	src/leveled_cdb.erl
 2016-11-27 00:18:20 +00:00
martinsumner
45f7fcb41a Datasync added for riak_sync strategy 2016-11-27 00:16:49 +00:00
Martin Sumner
0cb5fa39d9 Basho Erlang didn't like o_sync 
Or some options, trying sync.
 2016-11-26 23:03:07 +00:00
Martin Sumner
8f83c5226d Add log of write ops 2016-11-26 22:59:33 +00:00
Martin Sumner
feb7149a8d Add schema file 2016-11-26 22:10:51 +00:00
martinsumner
2fdf70ddf6 Add further skiplist test 2016-11-26 12:52:10 +00:00
Martin Sumner
fe080895fd Revert type definition 
Can’t find a type definition support din both OTP 16 and OTP 18, so
reverting to not defining type
 2016-11-25 18:22:35 +00:00
Martin Sumner
ec32a7e3eb OTP16 compliance - array type 2016-11-25 18:20:17 +00:00
martinsumner
196c807b5e Pass through sync_strategy 
Allow to switch for Riak to use o_sync as the sync flag rather than sync
 2016-11-25 17:41:08 +00:00
martinsumner
03d025d581 Replace ledger-side gb_trees 
Try to make minimal change to replace gb_trees with gb_tree API-like
skiplists
 2016-11-25 14:50:13 +00:00
martinsumner
c9afe34fea Missing key tming test 2016-11-24 23:31:13 +00:00
martinsumner
54eedb8282 Add out of range tests for skiplist range 2016-11-24 23:22:19 +00:00
martinsumner
3bbfd8b923 Tighten SkipList range 
Make SkipList range accurate - trim the edges
 2016-11-24 23:09:17 +00:00