Commit graph

34 commits

Author SHA1 Message Date
Martin Sumner
cb10a00845 Make test reliable
Might corrupt the wrong journal otherwise
2017-11-28 14:57:17 +00:00
Martin Sumner
8f27b3b628 Merge branch 'master' into mas-aae-segementfoldplus 2017-11-07 11:22:56 +00:00
Martin Sumner
f358bd7622 Switch to using passed in compression method for maybe_compress
When the compaction discovers compression is required it will used the passed in method at startup - not the method which had been previously defined.
2017-11-06 21:16:46 +00:00
Martin Sumner
9a0a4ced0d Test LZ4 from uncompressed
Coverage issue
2017-11-06 19:42:35 +00:00
Martin Sumner
1d475235d1 Improve test coverage
Make compress on receipt/compaction configurable
2017-11-06 18:44:08 +00:00
Martin Sumner
c8ad39b33b foldheads_bybucket adds segment list support
Accelerate queries for foldheads_bybucket as well
2017-11-01 22:00:12 +00:00
Martin Sumner
bfaed921e6 Split code for folders - introduce runner actor
Introduce a dedicated module for all the different fold types.  Also simplify the list of folders by deprecating those folds that should eb achieveable by fold_heads/fold_objects type folds but with smarter functions.

Makes sure that the fold functiosn also have better spec coverage, and are dialyzer checked.
2017-10-17 20:39:11 +01:00
Martin Sumner
869e799b41 Fix tests
Obviously got totally messed up and confused when testing previous
commits.

Multiple tests were failing for a change which got merged in as the
tests were not reflecting the required API.
2017-09-15 10:33:16 +01:00
Heinz N. Gies
25389893cf Add compatibility for old and new random / rand functions 2017-08-01 11:24:12 +02:00
martinsumner
f5dd154cee Rename hashtree query
Naming is now confusing now we have TicTac Trees.  This query builds a
list of keys and hashes not a tree - so it was misleading anyaway.  Now
renamed hashlist_query.
2017-06-16 12:38:59 +01:00
Martin Sumner
7642aac2cc Change Riak object hash approach
Change the riak object hash being kept in the metadata, to being a hash
of the vector clock
2017-06-16 10:14:24 +01:00
martinsumner
fbb4879d81 Change fold_heads to do basic Journal presence check
This at least checks the file is present, and the Key exists in the
index of that file.  If the value is corrupt it will be removed by
compation, and then this will fail (unless the file is never compacted).

TODO: resolve issus of files which are corrupt - but never compacted
- a job for backup?
2017-04-21 15:55:03 +01:00
martinsumner
9e28287231 Resolve failing recovery test
Now passing consistently with a number of different corruptions catered
for (including corruption of the Tag in the Inker Key)
2016-12-16 23:18:55 +00:00
martinsumner
b62b38caa6 Tidy up test at end
May cause issues otherwise if there is an attempt to corrupt the journal
running in another bookie
2016-11-28 23:00:12 +00:00
martinsumner
e8c1d39df9 Switch to binary format Riak object
Initial change to try and test assuming that leveled received the binary
format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
martinsumner
196c807b5e Pass through sync_strategy
Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
martinsumner
427b97873c Change FoldKeysFun 2016-11-18 17:58:43 +00:00
martinsumner
6684e8e1d3 Refine query to accept fold functions
Need to be able to pass external fold functions into different queries,
to work as a Riak backend
2016-11-18 15:53:22 +00:00
martinsumner
ac223ced68 Add FoldKeysFun
Add the capability to pass FoldKeysFun into the index_query to allow for
compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
martinsumner
dbb840d75e Ooops - test commenting 2016-11-09 22:06:02 +00:00
martinsumner
16c424cd7b Add tombstone to compaction problem
Minor test refinement
2016-11-09 20:03:44 +00:00
martinsumner
feb3c2a524 Add recovr strategy test 2016-11-09 16:27:47 +00:00
martinsumner
4583460328 Clean API of Riak-specific Methods
Clena the API of Riak specific methods, and also resolve timing issue in
simple_server unit test.  Previously this would end up with missing data
(and a lower sequence number after start) because of the penciller_clerk
timeout being relatively large in the context of this test.  Now the
timeout has bene reduced the L0 slot is cleared by the time of the
close.  To make sure an extra sleep has been added as a precaution to
avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
martinsumner
68b17c71b3 Expand fold objects support
Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
martinsumner
ee39b48313 Messed up test fixed 2016-11-03 13:34:06 +00:00
martinsumner
a752074148 Undo commenting 2016-11-03 12:12:56 +00:00
martinsumner
37e78dcdc9 Expanded AAE tests to include busted hashtable
Busted the hashtable in a Journal file, and demonstrated it can be fixed
by changing the extension name (no need to recover from backup if only
the hashtable is bust)
2016-11-03 12:11:50 +00:00
martinsumner
0572f43b8a Uncomment test 2016-11-02 15:40:22 +00:00
martinsumner
898f86a08d Add Fold Object by KeyList support 2016-11-02 15:38:51 +00:00
martinsumner
e7506c3c1f Startup work - baffled
Changes the stratup otpions to a prolist to make it easier to get
environment variables as default.

Tried application:start - and completely baffled as to how to get this
to work.
2016-11-02 12:58:27 +00:00
martinsumner
ce34235f2f Revert commented out test
Unintentional commenting
2016-11-01 01:38:48 +00:00
martinsumner
84a92b5f95 Further testing of compaction
Check we avoid crashing in challenging compaction scenarios
2016-11-01 00:46:14 +00:00
martinsumner
7d35ef7126 Lame AAE hashtree query test
Corruption of the values wihtin the journal doesn't get detected by the
hashtree query
2016-10-31 22:17:29 +00:00
martinsumner
73004328e1 Recovery Tests
Some initial entropy tests showing loss of data from a corrupted CDB
file.
2016-10-31 20:58:19 +00:00
Renamed from test/end_to_end/restart_SUITE.erl (Browse further)