umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1551 commits 1 branch 0 tags 25 MiB
Commit graph

1055 commits

Author SHA1 Message Date
martinsumner
364527f3b8 Add potential support for deeper layers 
Unproven attempt to make skiplist code more generic
 2016-11-30 22:35:23 +00:00
martinsumner
743d59c71b Make SkipList Two-Level 
Read and write times were increasing as the siz eof skiplist got larger
(e.g. over 2000).  Tried instead with a smaller skip width and a
two-tier skiplist and this gave much more regular eprformance as the
size of the skiplist went up.
 2016-11-30 15:36:27 +00:00
martinsumner
e3783485de Minor SkipList speed-up 2016-11-29 11:13:08 +00:00
martinsumner
dd6201b34b Try to avoid crashing on invalid key length 
May occurr in corrupted files.
 2016-11-29 00:27:23 +00:00
martinsumner
e8c1d39df9 Switch to binary format Riak object 
Initial change to try and test assuming that leveled received the binary
format of Riak objects (and parses that for metadata).
 2016-11-28 22:26:09 +00:00
martinsumner
b0a515553f Merge remote-tracking branch 'origin/OTP16' into OTP16 
# Conflicts:
#	src/leveled_cdb.erl
 2016-11-27 00:18:20 +00:00
martinsumner
45f7fcb41a Datasync added for riak_sync strategy 2016-11-27 00:16:49 +00:00
Martin Sumner
0cb5fa39d9 Basho Erlang didn't like o_sync 
Or some options, trying sync.
 2016-11-26 23:03:07 +00:00
Martin Sumner
8f83c5226d Add log of write ops 2016-11-26 22:59:33 +00:00
martinsumner
2fdf70ddf6 Add further skiplist test 2016-11-26 12:52:10 +00:00
Martin Sumner
fe080895fd Revert type definition 
Can’t find a type definition support din both OTP 16 and OTP 18, so
reverting to not defining type
 2016-11-25 18:22:35 +00:00
Martin Sumner
ec32a7e3eb OTP16 compliance - array type 2016-11-25 18:20:17 +00:00
martinsumner
196c807b5e Pass through sync_strategy 
Allow to switch for Riak to use o_sync as the sync flag rather than sync
 2016-11-25 17:41:08 +00:00
martinsumner
03d025d581 Replace ledger-side gb_trees 
Try to make minimal change to replace gb_trees with gb_tree API-like
skiplists
 2016-11-25 14:50:13 +00:00
martinsumner
c9afe34fea Missing key tming test 2016-11-24 23:31:13 +00:00
martinsumner
54eedb8282 Add out of range tests for skiplist range 2016-11-24 23:22:19 +00:00
martinsumner
3bbfd8b923 Tighten SkipList range 
Make SkipList range accurate - trim the edges
 2016-11-24 23:09:17 +00:00
martinsumner
2d3b1bbf2c Ongoing work on skip lists 
Initial rough implementation with some timing tests
 2016-11-24 20:16:41 +00:00
martinsumner
96b9e1faa3 Initial Commit for SkipLists 
To speed check on alternate platform
 2016-11-22 23:21:47 +00:00
martinsumner
638fc69e01 Correctly set array type 
Otherwise cannot compile in both OTP 16 and 17
 2016-11-21 22:26:12 +00:00
martinsumner
51dbad95c0 Change FoldBucketsFun to take just bucket 
FoldBucketsFun does not take keys should be a 2-arity function (Bucket,
Acc).
 2016-11-21 14:12:17 +00:00
martinsumner
acdc702e7d Check not snap 
Make sure snapshots cannot trigger deletion
 2016-11-21 12:35:20 +00:00
martinsumner
0f7e421371 Add destruction 
Allow a store to be cleared out and destroyed
 2016-11-21 12:34:40 +00:00
martinsumner
386d40928b Fast List Buckets 
Copied the technique from HanoiDB to speed up list buckets.
 2016-11-20 21:21:31 +00:00
martinsumner
f40ecdd529 Pick-up test misses 
There were some coverage misses in tests, so check in unit test coverage
or remove branches not currently needed.
 2016-11-18 21:35:45 +00:00
martinsumner
6684e8e1d3 Refine query to accept fold functions 
Need to be able to pass external fold functions into different queries,
to work as a Riak backend
 2016-11-18 15:53:22 +00:00
martinsumner
ac223ced68 Add FoldKeysFun 
Add the capability to pass FoldKeysFun into the index_query to allow for
compatability with riak backend requirements.
 2016-11-18 11:53:14 +00:00
martinsumner
e3004faa6e Add support for folding keys on a single bucket 2016-11-17 15:55:29 +00:00
martinsumner
aa355a0aae Coverage cheat pt 2 2016-11-14 20:56:59 +00:00
martinsumner
8cbe2ef93a Coverage cheats 
You juke the stats, and majors become colonels.  I've been here before
 2016-11-14 20:43:38 +00:00
martinsumner
630f802780 Inker Close nastiness 
Try to stop some of the potential deadlocking around Inker close and
prove that snapshots at higher Manifest SQNs can be ignored
 2016-11-14 19:34:11 +00:00
martinsumner
75d6af75c6 Penciller review 
The penciller attempt to close the L0 file if pending was unpredictable
in behaviour.  If a L0 file is still pending it will be lost - but this
is at least a predictable event.
 2016-11-14 17:18:28 +00:00
martinsumner
eb76a6dbcd Multi-key test 
Now failing with timeout coverage enabled.  100 keys is unnecessray so
reduce to 20 per loop, but suspect this is another issue
 2016-11-14 12:43:45 +00:00
martinsumner
c0e1455430 Remove PromptDelete 
Now that deleted journals are stored in waste for the retention period -
don't worry about deleting them
 2016-11-14 11:40:02 +00:00
martinsumner
44738f7c75 Deferred Deletion of Journals 
This allows for deleted journals to be retained for a period (the
waste_retnetion_period).  The idea being that a backup strategy can
ensure that all journals are backed up, even ones created and removed
from within a backup period - so that any restore pont is possible.

This is also a pre-cursor to removing some of the PromptDelete
complexity from the Inker Clerk - all compactions can prompt deletion as
deletion is now deferred.
 2016-11-14 11:17:14 +00:00
martinsumner
3d737dcf3c More sensible test assertion 
Also better for test coverage stats
 2016-11-09 19:38:02 +00:00
martinsumner
2458a90fb3 Remove overlapped case clause 2016-11-09 16:35:13 +00:00
martinsumner
feb3c2a524 Add recovr strategy test 2016-11-09 16:27:47 +00:00
martinsumner
3a77591da9 Removed unnecessary timeout condition from rolling state in CDB 2016-11-09 00:14:57 +00:00
martinsumner
0cf96e7198 Removed daft last function from SFT 2016-11-09 00:13:52 +00:00
martinsumner
0905ea51e2 Further corrupt file unit test for CDB 2016-11-08 23:07:03 +00:00
martinsumner
cc3cbc983b Tidy up closure of CDB Files 2016-11-08 22:43:22 +00:00
martinsumner
8b8652e2d7 Coverage Tests 
Add nonsense tests for nonsense coverage on standard methods.

Look at CDB search_hash_table - looks like it doubled-up on break-outs
so that one would never get hit
 2016-11-08 01:03:09 +00:00
martinsumner
b43ec0de7e Fix the fix 2016-11-08 00:46:01 +00:00
martinsumner
6f0abea401 Stop unncessary handling of pending_delete in test 2016-11-08 00:44:51 +00:00
martinsumner
105d3b66f2 Add hash clash unit tests 
Add unit tests for keys with clashing hashes
 2016-11-07 23:53:14 +00:00
martinsumner
b47f8963d5 Speed of corrupted sft unit test 
Test too slow when running with coverage
 2016-11-07 19:20:46 +00:00
martinsumner
5a36f053d9 End-to-end testing of corrupted bloom 
Add a unit test calling the API of a SFT file clerk with a corrupted
bloom
 2016-11-07 18:26:55 +00:00
martinsumner
05338a1e85 Switch SFT to fsm 
To amke it easier to context switch between the two file formats, make
the SFT file clerk a FSM like the CDB one.
 2016-11-07 17:08:50 +00:00
martinsumner
8035583301 Comment review 2016-11-07 11:17:13 +00:00