umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
669 commits 1 branch 0 tags 25 MiB
Commit graph

86 commits

Author SHA1 Message Date
Martin Sumner
a2c164e1dd Switch to binary vclock 2017-03-20 22:44:19 +00:00
Martin Sumner
994d9758b0 Object now 4 bytes larger 
due to addition of key changes length
 2017-03-20 16:21:49 +00:00
martinsumner
32cdf26ffb Correct to catch new response to snapshot request 2017-03-06 19:37:43 +00:00
martinsumner
41e5686a66 Test coverage - missing 1% 2017-02-26 20:52:40 +00:00
martinsumner
0ddaaf9ac3 Stopped unnecessary seek for last_key 
When rolling we already know the last_key - no need to seek for it on
startup.

The time it takes for this seek needs to be considered with regards to
startup time.  Can we do without knowing lastkey?
 2016-12-22 19:51:39 +00:00
martinsumner
9e28287231 Resolve failing recovery test 
Now passing consistently with a number of different corruptions catered
for (including corruption of the Tag in the Inker Key)
 2016-12-16 23:18:55 +00:00
martinsumner
3417baa3b8 Simple test 
To try and pinpoint any issue with _int index (as seen in Riak
integrtaion testing)
 2016-12-02 17:39:28 +00:00
martinsumner
b62b38caa6 Tidy up test at end 
May cause issues otherwise if there is an attempt to corrupt the journal
running in another bookie
 2016-11-28 23:00:12 +00:00
martinsumner
e8c1d39df9 Switch to binary format Riak object 
Initial change to try and test assuming that leveled received the binary
format of Riak objects (and parses that for metadata).
 2016-11-28 22:26:09 +00:00
martinsumner
45f7fcb41a Datasync added for riak_sync strategy 2016-11-27 00:16:49 +00:00
martinsumner
196c807b5e Pass through sync_strategy 
Allow to switch for Riak to use o_sync as the sync flag rather than sync
 2016-11-25 17:41:08 +00:00
martinsumner
51dbad95c0 Change FoldBucketsFun to take just bucket 
FoldBucketsFun does not take keys should be a 2-arity function (Bucket,
Acc).
 2016-11-21 14:12:17 +00:00
martinsumner
0f7e421371 Add destruction 
Allow a store to be cleared out and destroyed
 2016-11-21 12:34:40 +00:00
martinsumner
386d40928b Fast List Buckets 
Copied the technique from HanoiDB to speed up list buckets.
 2016-11-20 21:21:31 +00:00
martinsumner
427b97873c Change FoldKeysFun 2016-11-18 17:58:43 +00:00
martinsumner
57727e74f9 D'oh 2016-11-18 17:14:01 +00:00
martinsumner
f5944f2466 Change test foler to return (Bucket, Key} as expeceted 2016-11-18 16:52:53 +00:00
martinsumner
ec18f9ab4c Uncomment test 2016-11-18 16:34:16 +00:00
martinsumner
6684e8e1d3 Refine query to accept fold functions 
Need to be able to pass external fold functions into different queries,
to work as a Riak backend
 2016-11-18 15:53:22 +00:00
martinsumner
ac223ced68 Add FoldKeysFun 
Add the capability to pass FoldKeysFun into the index_query to allow for
compatability with riak backend requirements.
 2016-11-18 11:53:14 +00:00
martinsumner
e3004faa6e Add support for folding keys on a single bucket 2016-11-17 15:55:29 +00:00
martinsumner
630f802780 Inker Close nastiness 
Try to stop some of the potential deadlocking around Inker close and
prove that snapshots at higher Manifest SQNs can be ignored
 2016-11-14 19:34:11 +00:00
martinsumner
44738f7c75 Deferred Deletion of Journals 
This allows for deleted journals to be retained for a period (the
waste_retnetion_period).  The idea being that a backup strategy can
ensure that all journals are backed up, even ones created and removed
from within a backup period - so that any restore pont is possible.

This is also a pre-cursor to removing some of the PromptDelete
complexity from the Inker Clerk - all compactions can prompt deletion as
deletion is now deferred.
 2016-11-14 11:17:14 +00:00
martinsumner
dbb840d75e Ooops - test commenting 2016-11-09 22:06:02 +00:00
martinsumner
16c424cd7b Add tombstone to compaction problem 
Minor test refinement
 2016-11-09 20:03:44 +00:00
martinsumner
feb3c2a524 Add recovr strategy test 2016-11-09 16:27:47 +00:00
martinsumner
b9090a3109 Testutil and pause 2016-11-07 12:19:17 +00:00
martinsumner
e2670e6e8c Testutil and Pause 
Part of the testutil code is not recognising pause
 2016-11-07 12:18:00 +00:00
martinsumner
8f068c1280 Fixup API to allow for Tags to be passed 
plus fixes to previous commits moving API functions into testutil
 2016-11-07 10:42:49 +00:00
martinsumner
37c23a5b38 Shift pause out of leveled 
Leveled will now signal the need for a pause due to back-pressure, but
not actually pause itself.  The hope is that in a riak implementation
this pause can be managed by the put_fsm, and so not lock the store.
 2016-11-07 10:27:38 +00:00
martinsumner
4583460328 Clean API of Riak-specific Methods 
Clena the API of Riak specific methods, and also resolve timing issue in
simple_server unit test.  Previously this would end up with missing data
(and a lower sequence number after start) because of the penciller_clerk
timeout being relatively large in the context of this test.  Now the
timeout has bene reduced the L0 slot is cleared by the time of the
close.  To make sure an extra sleep has been added as a precaution to
avoid any intermittent issues.
 2016-11-07 10:11:57 +00:00
martinsumner
a251f3eab0 Speed up query count test 
Less individual querys to make count will speed up this taste, without
changing the nature of it
 2016-11-04 18:20:00 +00:00
martinsumner
479dc3ac80 Registering and releasing of Journal snapshots 
Added a test of journal compaction with a registered snapshot and it
showed that the deleting of files did not correctly check the list of
registerd snapshots.  Corrected.
 2016-11-04 15:56:57 +00:00
martinsumner
171baefc0c SFT Background Failure 
Let it crash approach - stop trying to catch and propgate failure of
write
 2016-11-04 14:31:19 +00:00
martinsumner
eeeee07081 Fold Objects - Check values test 
Test that summed values in fold objects before and after restart
 2016-11-04 14:23:37 +00:00
martinsumner
68b17c71b3 Expand fold objects support 
Fold over bucket and fold over index added
 2016-11-04 11:01:37 +00:00
martinsumner
ee39b48313 Messed up test fixed 2016-11-03 13:34:06 +00:00
martinsumner
2f28ae86e4 Journal compaction test to cover deleted objects 2016-11-03 12:31:00 +00:00
martinsumner
a752074148 Undo commenting 2016-11-03 12:12:56 +00:00
martinsumner
37e78dcdc9 Expanded AAE tests to include busted hashtable 
Busted the hashtable in a Journal file, and demonstrated it can be fixed
by changing the extension name (no need to recover from backup if only
the hashtable is bust)
 2016-11-03 12:11:50 +00:00
martinsumner
e8a7888397 Experiment with new cache size algorithm 
Remove the jitter probability and make it a smooth function heading
towards the max ache size
 2016-11-03 09:19:02 +00:00
martinsumner
0572f43b8a Uncomment test 2016-11-02 15:40:22 +00:00
martinsumner
898f86a08d Add Fold Object by KeyList support 2016-11-02 15:38:51 +00:00
martinsumner
8601e219d5 Revert test commenting 
Commented out some tests - bring back in
 2016-11-02 13:34:34 +00:00
martinsumner
a56ed18ba9 Test timing 
Look to see if test timing related to intermittent failure
 2016-11-02 13:27:16 +00:00
martinsumner
e7506c3c1f Startup work - baffled 
Changes the stratup otpions to a prolist to make it easier to get
environment variables as default.

Tried application:start - and completely baffled as to how to get this
to work.
 2016-11-02 12:58:27 +00:00
martinsumner
ce34235f2f Revert commented out test 
Unintentional commenting
 2016-11-01 01:38:48 +00:00
martinsumner
84a92b5f95 Further testing of compaction 
Check we avoid crashing in challenging compaction scenarios
 2016-11-01 00:46:14 +00:00
martinsumner
7d35ef7126 Lame AAE hashtree query test 
Corruption of the values wihtin the journal doesn't get detected by the
hashtree query
 2016-10-31 22:17:29 +00:00
martinsumner
73004328e1 Recovery Tests 
Some initial entropy tests showing loss of data from a corrupted CDB
file.
 2016-10-31 20:58:19 +00:00