Commit graph

210 commits

Author SHA1 Message Date
martinsumner
30e23fe514 Add no_lookup feature to snapshots 2017-03-02 18:23:47 +00:00
martinsumner
b01f7d23df Add pre-filter for 2i queries 2017-03-02 17:49:43 +00:00
martinsumner
35b6e16590 Clear dialyzer warnings 2017-02-27 20:23:36 +00:00
martinsumner
fce7cbd2ef Add sync_strategy comments 2017-02-27 17:32:38 +00:00
martinsumner
2db28c3028 Tidy line length 2017-02-27 12:24:26 +00:00
martinsumner
8d3f4bd4dc Comment API on Bookie
Make it easier to understand how to work the store
2017-02-27 11:58:16 +00:00
martinsumner
aba385869f Test long running clause 2017-02-26 22:41:25 +00:00
martinsumner
ed96d0ca7a Change so that type of cache is parameterised
The Tree doesn't seem to be better than the skiplist.  Get ready to make
this switchable
2017-01-21 11:38:26 +00:00
martinsumner
3d99036093 Switch the LM1 cache to be a tree
Use a tree of lists not a skiplist
2017-01-20 16:36:20 +00:00
martinsumner
a617f8fb66 Fix snapshot_store - add index to clone
Clone was not getting the updated index
2017-01-05 22:17:30 +00:00
martinsumner
5a88565c08 Switch to binary index in pmem
Remove the ets index in pmem and use a binary index instead.  This may
be slower, but avoids the bulk upload to ets, and means that matches
know of position (so only skiplists with a match need be tried).

Also stops the discrepancy between snapshots and non-snapshots - as
previously the snapshots were always slowed by not having access to the
ETS table.
2017-01-05 21:58:33 +00:00
martinsumner
1d3fb18df7 Resolve snapshotting issue
Need to make sure the extract from ets happens at the point the snapshot
is taken.
2017-01-05 18:43:55 +00:00
martinsumner
2c828b8eca Fix snapshot issue 2017-01-05 17:55:27 +00:00
martinsumner
e6270d288f Half-way to ets for Bookie mem
A half-way implementation with use of ETS as the bookie's memory
2017-01-05 17:00:12 +00:00
martinsumner
dc28388c76 Removed SFT
Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Martin Sumner
676e8fa494 Add Get Timing 2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909 put-timing default
Remove need for individual actors to know the defaults for put_timing
tuple
2016-12-22 14:41:43 +00:00
martinsumner
130fb36ddd Add head timings
Include log breaking down timings of HEAD requests by result and level
2016-12-22 14:03:31 +00:00
martinsumner
f3e16dcd10 Add long-running logs 2016-12-21 01:56:12 +00:00
martinsumner
060ce2e263 Add put timing points 2016-12-20 23:11:50 +00:00
Martin Sumner
5efed94a1e Try slightly larger cache 2016-12-13 22:29:55 +00:00
martinsumner
ccc993383d Stop second hash on fetch_head
The bookie should magic_hash for fetch_head, and now passes the hash to
the Penciller so second hash not required.
2016-12-11 01:21:53 +00:00
martinsumner
2d3a40e6f1 Magic Hash - and no L0 Index
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).

The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.

It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest.  This will then reduce the message passing required for a GET
not in the cache or higher levels
2016-12-11 01:02:56 +00:00
martinsumner
626a8e63f9 Experiment converting CDB to use skiplist not gb_tree
Might insertion time be faster?
2016-12-10 10:55:35 +00:00
martinsumner
a3f60e3609 OTP version shenanigans 2016-12-09 18:55:13 +00:00
martinsumner
d2bd01eaf1 Add fast fail to skiplist
Add a bloom filter to the skiplist, to make it faster at returning not
found.  The SkipList is now encapsulated within a dict().
2016-12-09 18:30:40 +00:00
martinsumner
f0db730f07 Adjust jitter settings 2016-12-09 16:34:15 +00:00
martinsumner
82cb49638a Attempt at performance improvement
Try to add some extra jitter in to the process of L0 writes, and also
make L0 writes delayed to help with bufferring
2016-12-09 14:36:03 +00:00
martinsumner
349d194a7c Increase jitter slightly 2016-12-09 09:52:31 +00:00
martinsumner
2f4013f430 Set Jitter correctly this time 2016-12-08 21:02:39 +00:00
martinsumner
f07ab85a81 Oops 2016-12-08 18:36:57 +00:00
martinsumner
34e7ce170a Jitter co-ordination issues
Experiment to try and reoslve jitter co-orsination issue
2016-12-08 18:35:20 +00:00
martinsumner
ac45f95559 Investigate jitter coordination 2016-12-08 18:30:45 +00:00
martinsumner
e2af60f72b Add cache size jitter
First pan-riak volume requests showed coordination issues
2016-12-08 16:38:44 +00:00
Martin Sumner
d07cad34b7 Why lists reverse?
It doesn’t make sense as the Acc may not be a list.  Not failing any
tests.
2016-12-02 00:40:00 +00:00
martinsumner
e8c1d39df9 Switch to binary format Riak object
Initial change to try and test assuming that leveled received the binary
format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
martinsumner
196c807b5e Pass through sync_strategy
Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
martinsumner
03d025d581 Replace ledger-side gb_trees
Try to make minimal change to replace gb_trees with gb_tree API-like
skiplists
2016-11-25 14:50:13 +00:00
martinsumner
51dbad95c0 Change FoldBucketsFun to take just bucket
FoldBucketsFun does not take keys should be a 2-arity function (Bucket,
Acc).
2016-11-21 14:12:17 +00:00
martinsumner
acdc702e7d Check not snap
Make sure snapshots cannot trigger deletion
2016-11-21 12:35:20 +00:00
martinsumner
0f7e421371 Add destruction
Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
martinsumner
386d40928b Fast List Buckets
Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
martinsumner
f40ecdd529 Pick-up test misses
There were some coverage misses in tests, so check in unit test coverage
or remove branches not currently needed.
2016-11-18 21:35:45 +00:00
martinsumner
6684e8e1d3 Refine query to accept fold functions
Need to be able to pass external fold functions into different queries,
to work as a Riak backend
2016-11-18 15:53:22 +00:00
martinsumner
ac223ced68 Add FoldKeysFun
Add the capability to pass FoldKeysFun into the index_query to allow for
compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
martinsumner
e3004faa6e Add support for folding keys on a single bucket 2016-11-17 15:55:29 +00:00
martinsumner
8cbe2ef93a Coverage cheats
You juke the stats, and majors become colonels.  I've been here before
2016-11-14 20:43:38 +00:00
martinsumner
eb76a6dbcd Multi-key test
Now failing with timeout coverage enabled.  100 keys is unnecessray so
reduce to 20 per loop, but suspect this is another issue
2016-11-14 12:43:45 +00:00
martinsumner
44738f7c75 Deferred Deletion of Journals
This allows for deleted journals to be retained for a period (the
waste_retnetion_period).  The idea being that a backup strategy can
ensure that all journals are backed up, even ones created and removed
from within a backup period - so that any restore pont is possible.

This is also a pre-cursor to removing some of the PromptDelete
complexity from the Inker Clerk - all compactions can prompt deletion as
deletion is now deferred.
2016-11-14 11:17:14 +00:00
martinsumner
079e713e16 Comment fiddling 2016-11-07 11:07:12 +00:00