martinsumner
30e23fe514
Add no_lookup feature to snapshots
2017-03-02 18:23:47 +00:00
martinsumner
b01f7d23df
Add pre-filter for 2i queries
2017-03-02 17:49:43 +00:00
martinsumner
35b6e16590
Clear dialyzer warnings
2017-02-27 20:23:36 +00:00
martinsumner
fce7cbd2ef
Add sync_strategy comments
2017-02-27 17:32:38 +00:00
martinsumner
2db28c3028
Tidy line length
2017-02-27 12:24:26 +00:00
martinsumner
8d3f4bd4dc
Comment API on Bookie
...
Make it easier to understand how to work the store
2017-02-27 11:58:16 +00:00
martinsumner
aba385869f
Test long running clause
2017-02-26 22:41:25 +00:00
martinsumner
ed96d0ca7a
Change so that type of cache is parameterised
...
The Tree doesn't seem to be better than the skiplist. Get ready to make
this switchable
2017-01-21 11:38:26 +00:00
martinsumner
3d99036093
Switch the LM1 cache to be a tree
...
Use a tree of lists not a skiplist
2017-01-20 16:36:20 +00:00
martinsumner
a617f8fb66
Fix snapshot_store - add index to clone
...
Clone was not getting the updated index
2017-01-05 22:17:30 +00:00
martinsumner
5a88565c08
Switch to binary index in pmem
...
Remove the ets index in pmem and use a binary index instead. This may
be slower, but avoids the bulk upload to ets, and means that matches
know of position (so only skiplists with a match need be tried).
Also stops the discrepancy between snapshots and non-snapshots - as
previously the snapshots were always slowed by not having access to the
ETS table.
2017-01-05 21:58:33 +00:00
martinsumner
1d3fb18df7
Resolve snapshotting issue
...
Need to make sure the extract from ets happens at the point the snapshot
is taken.
2017-01-05 18:43:55 +00:00
martinsumner
2c828b8eca
Fix snapshot issue
2017-01-05 17:55:27 +00:00
martinsumner
e6270d288f
Half-way to ets for Bookie mem
...
A half-way implementation with use of ETS as the bookie's memory
2017-01-05 17:00:12 +00:00
martinsumner
dc28388c76
Removed SFT
...
Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Martin Sumner
676e8fa494
Add Get Timing
2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909
put-timing default
...
Remove need for individual actors to know the defaults for put_timing
tuple
2016-12-22 14:41:43 +00:00
martinsumner
130fb36ddd
Add head timings
...
Include log breaking down timings of HEAD requests by result and level
2016-12-22 14:03:31 +00:00
martinsumner
f3e16dcd10
Add long-running logs
2016-12-21 01:56:12 +00:00
martinsumner
060ce2e263
Add put timing points
2016-12-20 23:11:50 +00:00
Martin Sumner
5efed94a1e
Try slightly larger cache
2016-12-13 22:29:55 +00:00
martinsumner
ccc993383d
Stop second hash on fetch_head
...
The bookie should magic_hash for fetch_head, and now passes the hash to
the Penciller so second hash not required.
2016-12-11 01:21:53 +00:00
martinsumner
2d3a40e6f1
Magic Hash - and no L0 Index
...
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).
The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.
It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest. This will then reduce the message passing required for a GET
not in the cache or higher levels
2016-12-11 01:02:56 +00:00
martinsumner
626a8e63f9
Experiment converting CDB to use skiplist not gb_tree
...
Might insertion time be faster?
2016-12-10 10:55:35 +00:00
martinsumner
a3f60e3609
OTP version shenanigans
2016-12-09 18:55:13 +00:00
martinsumner
d2bd01eaf1
Add fast fail to skiplist
...
Add a bloom filter to the skiplist, to make it faster at returning not
found. The SkipList is now encapsulated within a dict().
2016-12-09 18:30:40 +00:00
martinsumner
f0db730f07
Adjust jitter settings
2016-12-09 16:34:15 +00:00
martinsumner
82cb49638a
Attempt at performance improvement
...
Try to add some extra jitter in to the process of L0 writes, and also
make L0 writes delayed to help with bufferring
2016-12-09 14:36:03 +00:00
martinsumner
349d194a7c
Increase jitter slightly
2016-12-09 09:52:31 +00:00
martinsumner
2f4013f430
Set Jitter correctly this time
2016-12-08 21:02:39 +00:00
martinsumner
f07ab85a81
Oops
2016-12-08 18:36:57 +00:00
martinsumner
34e7ce170a
Jitter co-ordination issues
...
Experiment to try and reoslve jitter co-orsination issue
2016-12-08 18:35:20 +00:00
martinsumner
ac45f95559
Investigate jitter coordination
2016-12-08 18:30:45 +00:00
martinsumner
e2af60f72b
Add cache size jitter
...
First pan-riak volume requests showed coordination issues
2016-12-08 16:38:44 +00:00
Martin Sumner
d07cad34b7
Why lists reverse?
...
It doesn’t make sense as the Acc may not be a list. Not failing any
tests.
2016-12-02 00:40:00 +00:00
martinsumner
e8c1d39df9
Switch to binary format Riak object
...
Initial change to try and test assuming that leveled received the binary
format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
martinsumner
196c807b5e
Pass through sync_strategy
...
Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
martinsumner
03d025d581
Replace ledger-side gb_trees
...
Try to make minimal change to replace gb_trees with gb_tree API-like
skiplists
2016-11-25 14:50:13 +00:00
martinsumner
51dbad95c0
Change FoldBucketsFun to take just bucket
...
FoldBucketsFun does not take keys should be a 2-arity function (Bucket,
Acc).
2016-11-21 14:12:17 +00:00
martinsumner
acdc702e7d
Check not snap
...
Make sure snapshots cannot trigger deletion
2016-11-21 12:35:20 +00:00
martinsumner
0f7e421371
Add destruction
...
Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
martinsumner
386d40928b
Fast List Buckets
...
Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
martinsumner
f40ecdd529
Pick-up test misses
...
There were some coverage misses in tests, so check in unit test coverage
or remove branches not currently needed.
2016-11-18 21:35:45 +00:00
martinsumner
6684e8e1d3
Refine query to accept fold functions
...
Need to be able to pass external fold functions into different queries,
to work as a Riak backend
2016-11-18 15:53:22 +00:00
martinsumner
ac223ced68
Add FoldKeysFun
...
Add the capability to pass FoldKeysFun into the index_query to allow for
compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
martinsumner
e3004faa6e
Add support for folding keys on a single bucket
2016-11-17 15:55:29 +00:00
martinsumner
8cbe2ef93a
Coverage cheats
...
You juke the stats, and majors become colonels. I've been here before
2016-11-14 20:43:38 +00:00
martinsumner
eb76a6dbcd
Multi-key test
...
Now failing with timeout coverage enabled. 100 keys is unnecessray so
reduce to 20 per loop, but suspect this is another issue
2016-11-14 12:43:45 +00:00
martinsumner
44738f7c75
Deferred Deletion of Journals
...
This allows for deleted journals to be retained for a period (the
waste_retnetion_period). The idea being that a backup strategy can
ensure that all journals are backed up, even ones created and removed
from within a backup period - so that any restore pont is possible.
This is also a pre-cursor to removing some of the PromptDelete
complexity from the Inker Clerk - all compactions can prompt deletion as
deletion is now deferred.
2016-11-14 11:17:14 +00:00
martinsumner
079e713e16
Comment fiddling
2016-11-07 11:07:12 +00:00