umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
leveled/test/end_to_end/basic_SUITE.erl

1026 lines
44 KiB
Erlang
Raw Normal View History

Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
-module(basic_SUITE).
-include_lib("common_test/include/ct.hrl").
Fix include target
2016-10-17 14:24:32 +01:00
-include("include/leveled.hrl").
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
-export([all/0]).
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
-export([simple_put_fetch_head_delete/1,
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
many_put_fetch_head/1,
journal_compaction/1,
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
fetchput_snapshot/1,
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
load_and_count/1,
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
load_and_count_withdelete/1,
Add test for is_empty Bucket listing didn't care if keys were active - now does.
2017-05-23 11:59:44 +01:00
space_clear_ondelete/1,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
is_empty_test/1,
Add test of switching from a big journal to a little journal And sending objects over the journal size
2018-09-25 16:56:24 +01:00
many_put_fetch_switchcompression/1,
Add ct test Which passes
2018-09-27 11:32:42 +01:00
bigjournal_littlejournal/1,
Test sst slot configuration change Confirm it results in many more files, if the slot count reduced. Has to handle the fact that Level 0 file has unlimited slots regardless of number of slots configured
2019-01-27 22:03:55 +00:00
bigsst_littlesst/1,
Test to expose startup issue See #194
2018-10-03 18:29:20 +01:00
safereaderror_startup/1,
remove_journal_test/1
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
]).
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
Broken snapshot test The test confirming that deleting sft files wer eheld open whilst snapshots were registered was actually broken. This test has now been fixed, as well as the logic in registring snapshots which had used ledger_sqn mistakenly rather than manifest_sqn.
2016-10-21 11:38:30 +01:00
all() -> [
Uncomment tests!
2019-01-27 23:31:44 +00:00
simple_put_fetch_head_delete,
many_put_fetch_head,
journal_compaction,
fetchput_snapshot,
load_and_count,
load_and_count_withdelete,
space_clear_ondelete,
is_empty_test,
many_put_fetch_switchcompression,
bigjournal_littlejournal,
Test sst slot configuration change Confirm it results in many more files, if the slot count reduced. Has to handle the fact that Level 0 file has unlimited slots regardless of number of slots configured
2019-01-27 22:03:55 +00:00
bigsst_littlesst,
Handle missing journal files without data loss Make sure data added after the journal has been lost will not be lost (even though lost data may stay lost)
2018-10-03 18:38:56 +01:00
safereaderror_startup,
Test to expose startup issue See #194
2018-10-03 18:29:20 +01:00
remove_journal_test
Improve test coverage Some additional tests following previous refactoring for abstraction, primarily to make manifest print safer an dprove co-existence of Riak and non-Riak objects.
2016-10-14 22:58:01 +01:00
].
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
simple_put_fetch_head_delete(_Config) ->
Support for log settings at startup Both log level and forced_logs. Allows for log_level to be changed at startup ad runtime. Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
io:format("simple test with info and no forced logs~n"),
simple_test_withlog(info, []),
io:format("simple test with error and no forced logs~n"),
simple_test_withlog(error, []),
io:format("simple test with error and stats logs~n"),
simple_test_withlog(error, ["B0015", "B0016", "B0017", "B0018",
"P0032", "SST12", "CDB19", "SST13", "I0019"]).
Allow for run-time changes to log_level and forced_logs Will not lead to immediate run time changes in SST or CDB logs. These log settings will only change once the new files are re-written. To completely change the log level - a restart of the store is necessary with new startup options.
2018-12-11 21:59:57 +00:00
Support for log settings at startup Both log level and forced_logs. Allows for log_level to be changed at startup ad runtime. Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
simple_test_withlog(LogLevel, ForcedLogs) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
StartOpts1 = [{root_path, RootPath},
Support for log settings at startup Both log level and forced_logs. Allows for log_level to be changed at startup ad runtime. Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
{sync_strategy, testutil:sync_strategy()},
{log_level, LogLevel},
{forced_logs, ForcedLogs}],
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{TestObject, TestSpec} = testutil:generate_testobject(),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
testutil:check_formissingobject(Bookie1, "Bucket1", "Key2"),
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ok = leveled_bookie:book_close(Bookie1),
Startup work - baffled Changes the stratup otpions to a prolist to make it easier to get environment variables as default. Tried application:start - and completely baffled as to how to get this to work.
2016-11-02 12:58:27 +00:00
StartOpts2 = [{root_path, RootPath},
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{max_journalsize, 3000000},
Support for log settings at startup Both log level and forced_logs. Allows for log_level to be changed at startup ad runtime. Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
{sync_strategy, testutil:sync_strategy()},
{log_level, LogLevel},
{forced_logs, ForcedLogs}],
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
{ok, Bookie2} = leveled_bookie:book_start(StartOpts2),
Allow for run-time changes to log_level and forced_logs Will not lead to immediate run time changes in SST or CDB logs. These log settings will only change once the new files are re-written. To completely change the log level - a restart of the store is necessary with new startup options.
2018-12-11 21:59:57 +00:00
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie2, TestObject),
ObjList1 = testutil:generate_objects(5000, 2),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:riakload(Bookie2, ObjList1),
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ChkList1 = lists:sublist(lists:sort(ObjList1), 100),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, ChkList1),
testutil:check_forobject(Bookie2, TestObject),
testutil:check_formissingobject(Bookie2, "Bucket1", "Key2"),
Improve test coverage Some additional tests following previous refactoring for abstraction, primarily to make manifest print safer an dprove co-existence of Riak and non-Riak objects.
2016-10-14 22:58:01 +01:00
ok = leveled_bookie:book_put(Bookie2, "Bucket1", "Key2", "Value2",
[{add, "Index1", "Term1"}]),
{ok, "Value2"} = leveled_bookie:book_get(Bookie2, "Bucket1", "Key2"),
Isolate better changes needed to support changes to metadata extraction More obvious how to extend the code as it is all in one module. Also add a new field to the standard object metadata tuple that may hold in the future other object metadata base don user-defined functions.
2018-12-06 15:31:11 +00:00
{ok, {62888926, 60, undefined}} = leveled_bookie:book_head(Bookie2,
"Bucket1",
"Key2"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_formissingobject(Bookie2, "Bucket1", "Key2"),
Improve test coverage Some additional tests following previous refactoring for abstraction, primarily to make manifest print safer an dprove co-existence of Riak and non-Riak objects.
2016-10-14 22:58:01 +01:00
ok = leveled_bookie:book_put(Bookie2, "Bucket1", "Key2", <<"Value2">>,
[{remove, "Index1", "Term1"},
{add, "Index1", <<"Term2">>}]),
{ok, <<"Value2">>} = leveled_bookie:book_get(Bookie2, "Bucket1", "Key2"),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
ok = leveled_bookie:book_close(Bookie2),
Improve test coverage Some additional tests following previous refactoring for abstraction, primarily to make manifest print safer an dprove co-existence of Riak and non-Riak objects.
2016-10-14 22:58:01 +01:00
{ok, Bookie3} = leveled_bookie:book_start(StartOpts2),
{ok, <<"Value2">>} = leveled_bookie:book_get(Bookie3, "Bucket1", "Key2"),
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
ok = leveled_bookie:book_delete(Bookie3, "Bucket1", "Key2",
[{remove, "Index1", "Term1"}]),
not_found = leveled_bookie:book_get(Bookie3, "Bucket1", "Key2"),
Quality review Minor test fix-up and quality changes
2016-10-30 22:06:44 +00:00
not_found = leveled_bookie:book_head(Bookie3, "Bucket1", "Key2"),
Improve test coverage Some additional tests following previous refactoring for abstraction, primarily to make manifest print safer an dprove co-existence of Riak and non-Riak objects.
2016-10-14 22:58:01 +01:00
ok = leveled_bookie:book_close(Bookie3),
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
{ok, Bookie4} = leveled_bookie:book_start(StartOpts2),
not_found = leveled_bookie:book_get(Bookie4, "Bucket1", "Key2"),
Add destruction Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
ok = leveled_bookie:book_destroy(Bookie4).
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
many_put_fetch_head(_Config) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
StartOpts1 = [{root_path, RootPath},
{max_pencillercachesize, 16000},
Improve test coverage Make compress on receipt/compaction configurable
2017-11-06 18:44:08 +00:00
{sync_strategy, riak_sync},
{compression_point, on_compact}],
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{TestObject, TestSpec} = testutil:generate_testobject(),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Add access to SQN Use book_sqn/3 or book_sqn/4 to get the SQN of an object in the store.
2019-03-13 16:21:03 +00:00
{ok, 1} = leveled_bookie:book_sqn(Bookie1,
testutil:get_bucket(TestObject),
testutil:get_key(TestObject),
?RIAK_TAG),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
ok = leveled_bookie:book_close(Bookie1),
Startup work - baffled Changes the stratup otpions to a prolist to make it easier to get environment variables as default. Tried application:start - and completely baffled as to how to get this to work.
2016-11-02 12:58:27 +00:00
StartOpts2 = [{root_path, RootPath},
Test logging of samples To prompt the log the journal size needs to be reduced
2017-11-20 15:31:31 +00:00
{max_journalsize, 50000000},
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{max_pencillercachesize, 32000},
Improve test coverage Make compress on receipt/compaction configurable
2017-11-06 18:44:08 +00:00
{sync_strategy, testutil:sync_strategy()},
{compression_point, on_receipt}],
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
{ok, Bookie2} = leveled_bookie:book_start(StartOpts2),
Allow for run-time changes to log_level and forced_logs Will not lead to immediate run time changes in SST or CDB logs. These log settings will only change once the new files are re-written. To completely change the log level - a restart of the store is necessary with new startup options.
2018-12-11 21:59:57 +00:00
ok = leveled_bookie:book_loglevel(Bookie2, error),
ok = leveled_bookie:book_addlogs(Bookie2, ["B0015"]),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie2, TestObject),
Add access to SQN Use book_sqn/3 or book_sqn/4 to get the SQN of an object in the store.
2019-03-13 16:21:03 +00:00
{ok, 1} = leveled_bookie:book_sqn(Bookie2,
testutil:get_bucket(TestObject),
testutil:get_key(TestObject),
?RIAK_TAG),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
GenList = [2, 20002, 40002, 60002, 80002,
100002, 120002, 140002, 160002, 180002],
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
CLs = testutil:load_objects(20000, GenList, Bookie2, TestObject,
fun testutil:generate_smallobjects/2),
Allow for run-time changes to log_level and forced_logs Will not lead to immediate run time changes in SST or CDB logs. These log settings will only change once the new files are re-written. To completely change the log level - a restart of the store is necessary with new startup options.
2018-12-11 21:59:57 +00:00
{error, ["B0015"]} = leveled_bookie:book_logsettings(Bookie2),
ok = leveled_bookie:book_removelogs(Bookie2, ["B0015"]),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
CL1A = lists:nth(1, CLs),
ChkListFixed = lists:nth(length(CLs), CLs),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, CL1A),
Allow for run-time changes to log_level and forced_logs Will not lead to immediate run time changes in SST or CDB logs. These log settings will only change once the new files are re-written. To completely change the log level - a restart of the store is necessary with new startup options.
2018-12-11 21:59:57 +00:00
{error, []} = leveled_bookie:book_logsettings(Bookie2),
ok = leveled_bookie:book_loglevel(Bookie2, info),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ObjList2A = testutil:generate_objects(5000, 2),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:riakload(Bookie2, ObjList2A),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
ChkList2A = lists:sublist(lists:sort(ObjList2A), 1000),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, ChkList2A),
testutil:check_forlist(Bookie2, ChkListFixed),
testutil:check_forobject(Bookie2, TestObject),
testutil:check_forlist(Bookie2, ChkList2A),
testutil:check_forlist(Bookie2, ChkListFixed),
testutil:check_forobject(Bookie2, TestObject),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
ok = leveled_bookie:book_close(Bookie2),
{ok, Bookie3} = leveled_bookie:book_start(StartOpts2),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie3, ChkList2A),
testutil:check_forobject(Bookie3, TestObject),
Add access to SQN Use book_sqn/3 or book_sqn/4 to get the SQN of an object in the store.
2019-03-13 16:21:03 +00:00
{ok, 1} = leveled_bookie:book_sqn(Bookie3,
testutil:get_bucket(TestObject),
testutil:get_key(TestObject),
?RIAK_TAG),
not_found = leveled_bookie:book_sqn(Bookie3,
testutil:get_bucket(TestObject),
testutil:get_key(TestObject),
?STD_TAG),
Test default tag
2019-03-14 00:08:01 +00:00
not_found = leveled_bookie:book_sqn(Bookie3,
testutil:get_bucket(TestObject),
testutil:get_key(TestObject)),
Test coverage - missing 1%
2017-02-26 20:52:40 +00:00
testutil:check_formissingobject(Bookie3, "Bookie1", "MissingKey0123"),
Add destruction Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
ok = leveled_bookie:book_destroy(Bookie3).
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
Add test of switching from a big journal to a little journal And sending objects over the journal size
2018-09-25 16:56:24 +01:00
bigjournal_littlejournal(_Config) ->
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 50000000},
{max_pencillercachesize, 32000},
{sync_strategy, testutil:sync_strategy()},
{compression_point, on_compact}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
ObjL1 =
testutil:generate_objects(100, 1, [],
leveled_rand:rand_bytes(10000),
fun() -> [] end, <<"B">>),
testutil:riakload(Bookie1, ObjL1),
ok = leveled_bookie:book_close(Bookie1),
StartOpts2 = lists:ukeysort(1, [{max_journalsize, 5000}|StartOpts1]),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts2),
ObjL2 =
testutil:generate_objects(10, 1000, [],
leveled_rand:rand_bytes(10000),
fun() -> [] end, <<"B">>),
testutil:riakload(Bookie2, ObjL2),
testutil:check_forlist(Bookie2, ObjL1),
testutil:check_forlist(Bookie2, ObjL2),
ok = leveled_bookie:book_destroy(Bookie2).
Test sst slot configuration change Confirm it results in many more files, if the slot count reduced. Has to handle the fact that Level 0 file has unlimited slots regardless of number of slots configured
2019-01-27 22:03:55 +00:00
bigsst_littlesst(_Config) ->
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 50000000},
Change cache_size in sst tests Makes results more predictable (with coin toss variations)
2019-01-29 13:40:55 +00:00
{cache_size, 500},
Test sst slot configuration change Confirm it results in many more files, if the slot count reduced. Has to handle the fact that Level 0 file has unlimited slots regardless of number of slots configured
2019-01-27 22:03:55 +00:00
{max_pencillercachesize, 16000},
{max_sstslots, 256},
{sync_strategy, testutil:sync_strategy()},
{compression_point, on_compact}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
ObjL1 =
testutil:generate_objects(60000, 1, [],
leveled_rand:rand_bytes(100),
fun() -> [] end, <<"B">>),
testutil:riakload(Bookie1, ObjL1),
testutil:check_forlist(Bookie1, ObjL1),
JFP = RootPath ++ "/ledger/ledger_files/",
{ok, FNS1} = file:list_dir(JFP),
ok = leveled_bookie:book_destroy(Bookie1),
StartOpts2 = lists:ukeysort(1, [{max_sstslots, 24}|StartOpts1]),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts2),
testutil:riakload(Bookie2, ObjL1),
testutil:check_forlist(Bookie2, ObjL1),
{ok, FNS2} = file:list_dir(JFP),
ok = leveled_bookie:book_destroy(Bookie2),
io:format("Big SST ~w files Little SST ~w files~n",
[length(FNS1), length(FNS2)]),
Change cache_size in sst tests Makes results more predictable (with coin toss variations)
2019-01-29 13:40:55 +00:00
true = length(FNS2) >= (2 * length(FNS1)).
Test sst slot configuration change Confirm it results in many more files, if the slot count reduced. Has to handle the fact that Level 0 file has unlimited slots regardless of number of slots configured
2019-01-27 22:03:55 +00:00
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
journal_compaction(_Config) ->
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
journal_compaction_tester(false, 3600),
journal_compaction_tester(false, undefined),
journal_compaction_tester(true, 3600).
journal_compaction_tester(Restart, WRP) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Startup work - baffled Changes the stratup otpions to a prolist to make it easier to get environment variables as default. Tried application:start - and completely baffled as to how to get this to work.
2016-11-02 12:58:27 +00:00
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 10000000},
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{max_run_length, 1},
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
{sync_strategy, testutil:sync_strategy()},
{waste_retention_period, WRP}],
{ok, Bookie0} = leveled_bookie:book_start(StartOpts1),
ok = leveled_bookie:book_compactjournal(Bookie0, 30000),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{TestObject, TestSpec} = testutil:generate_testobject(),
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
ok = testutil:book_riakput(Bookie0, TestObject, TestSpec),
testutil:check_forobject(Bookie0, TestObject),
Journal compaction test to cover deleted objects
2016-11-03 12:31:00 +00:00
ObjList1 = testutil:generate_objects(20000, 2),
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
testutil:riakload(Bookie0, ObjList1),
Make tets less pointless Journla compaction test wouldn't actually cause compaction
2016-10-30 20:14:11 +00:00
ChkList1 = lists:sublist(lists:sort(ObjList1), 10000),
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
testutil:check_forlist(Bookie0, ChkList1),
testutil:check_forobject(Bookie0, TestObject),
Test timeouts So that coverage testing will run.
2017-07-02 22:23:02 +01:00
{B2, K2, V2, Spec2, MD} = {"Bucket2",
"Key2",
"Value2",
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
[],
Test timeouts So that coverage testing will run.
2017-07-02 22:23:02 +01:00
[{"MDK2", "MDV2"}]},
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{TestObject2, TestSpec2} = testutil:generate_testobject(B2, K2,
V2, Spec2, MD),
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
ok = testutil:book_riakput(Bookie0, TestObject2, TestSpec2),
ok = leveled_bookie:book_compactjournal(Bookie0, 30000),
testutil:check_forlist(Bookie0, ChkList1),
testutil:check_forobject(Bookie0, TestObject),
testutil:check_forobject(Bookie0, TestObject2),
testutil:check_forlist(Bookie0, ChkList1),
testutil:check_forobject(Bookie0, TestObject),
testutil:check_forobject(Bookie0, TestObject2),
Journal compaction test to cover deleted objects
2016-11-03 12:31:00 +00:00
%% Delete some of the objects
ObjListD = testutil:generate_objects(10000, 2),
lists:foreach(fun({_R, O, _S}) ->
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
testutil:book_riakdelete(Bookie0,
Push log update through to cdb/sst Using the cdb_options and sst_options records
2018-12-11 20:42:00 +00:00
testutil:get_bucket(O),
testutil:get_key(O),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
[])
Journal compaction test to cover deleted objects
2016-11-03 12:31:00 +00:00
end,
ObjListD),
%% Now replace all the other objects
ObjList2 = testutil:generate_objects(40000, 10002),
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
testutil:riakload(Bookie0, ObjList2),
Bookie1 =
case Restart of
true ->
ok = leveled_bookie:book_close(Bookie0),
{ok, RestartedB} = leveled_bookie:book_start(StartOpts1),
RestartedB;
false ->
Bookie0
end,
Make tets less pointless Journla compaction test wouldn't actually cause compaction
2016-10-30 20:14:11 +00:00
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
Add coverage for CDB delete_pending with snapshot CDB file in delete_pending state, and times out but the Inker has a snapshot open - and so the file is not deleted.
2017-11-09 12:02:03 +00:00
WasteFP = RootPath ++ "/journal/journal_files/waste",
% Start snapshot - should stop deletions
Split code for folders - introduce runner actor Introduce a dedicated module for all the different fold types. Also simplify the list of folders by deprecating those folds that should eb achieveable by fold_heads/fold_objects type folds but with smarter functions. Makes sure that the fold functiosn also have better spec coverage, and are dialyzer checked.
2017-10-17 20:39:11 +01:00
{ok, PclClone, InkClone} =
leveled_bookie:book_snapshot(Bookie1, store, undefined, false),
Add coverage for CDB delete_pending with snapshot CDB file in delete_pending state, and times out but the Inker has a snapshot open - and so the file is not deleted.
2017-11-09 12:02:03 +00:00
ok = leveled_bookie:book_compactjournal(Bookie1, 30000),
testutil:wait_for_compaction(Bookie1),
% Wait to cause delete_pending to be blocked by snapshot
% timeout on switch to delete_pending is 10s
timer:sleep(10100),
case WRP of
undefined ->
ok;
_ ->
% Check nothing got deleted
{ok, CJs} = file:list_dir(WasteFP),
true = length(CJs) == 0
end,
ok = leveled_penciller:pcl_close(PclClone),
ok = leveled_inker:ink_close(InkClone),
% Snapshot released so deletes shoudl occur at next timeout
case WRP of
undefined ->
timer:sleep(10100); % wait for delete_pending timeout
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
% Wait 2 seconds for files to be deleted
Add coverage for CDB delete_pending with snapshot CDB file in delete_pending state, and times out but the Inker has a snapshot open - and so the file is not deleted.
2017-11-09 12:02:03 +00:00
_ ->
FindDeletedFilesFun =
fun(X, Found) ->
case Found of
true ->
Found;
false ->
{ok, Files} = file:list_dir(WasteFP),
if
length(Files) > 0 ->
io:format("Deleted files found~n"),
true;
length(Files) == 0 ->
timer:sleep(X),
false
end
end
end,
lists:foldl(FindDeletedFilesFun,
false,
[2000,2000,2000,2000,2000,2000])
end,
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
{ok, ClearedJournals} = file:list_dir(WasteFP),
io:format("~w ClearedJournals found~n", [length(ClearedJournals)]),
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
case is_integer(WRP) of
true ->
true = length(ClearedJournals) > 0;
false ->
true = length(ClearedJournals) == 0
end,
Make tets less pointless Journla compaction test wouldn't actually cause compaction
2016-10-30 20:14:11 +00:00
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ChkList3 = lists:sublist(lists:sort(ObjList2), 500),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie1, ChkList3),
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ok = leveled_bookie:book_close(Bookie1),
Add coverage for CDB delete_pending with snapshot CDB file in delete_pending state, and times out but the Inker has a snapshot open - and so the file is not deleted.
2017-11-09 12:02:03 +00:00
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
% Restart
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie2, TestObject),
testutil:check_forlist(Bookie2, ChkList3),
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
ok = leveled_bookie:book_close(Bookie2),
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
StartOpts2 = [{root_path, RootPath},
{max_journalsize, 10000000},
{max_run_length, 1},
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{waste_retention_period, 1},
{sync_strategy, testutil:sync_strategy()}],
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
{ok, Bookie3} = leveled_bookie:book_start(StartOpts2),
ok = leveled_bookie:book_compactjournal(Bookie3, 30000),
Make check that compaction not ongoing before accepting new compaction Respond 'busy' if compaction is ongoing
2019-01-24 15:46:17 +00:00
busy = leveled_bookie:book_compactjournal(Bookie3, 30000),
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
testutil:wait_for_compaction(Bookie3),
ok = leveled_bookie:book_close(Bookie3),
{ok, ClearedJournalsPC} = file:list_dir(WasteFP),
io:format("~w ClearedJournals found~n", [length(ClearedJournalsPC)]),
Extend journal compaction test to cover with and without waste retention. Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
case is_integer(WRP) of
true ->
true = length(ClearedJournals) > 0;
false ->
true = length(ClearedJournals) == 0
end,
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
testutil:reset_filestructure(10000).
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
fetchput_snapshot(_Config) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 30000000},
{sync_strategy, none}],
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{TestObject, TestSpec} = testutil:generate_testobject(),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Load up 5000 objects
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ObjList1 = testutil:generate_objects(5000, 2),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:riakload(Bookie1, ObjList1),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Now take a snapshot - check it has the same objects
Startup work - baffled Changes the stratup otpions to a prolist to make it easier to get environment variables as default. Tried application:start - and completely baffled as to how to get this to work.
2016-11-02 12:58:27 +00:00
SnapOpts1 = [{snapshot_bookie, Bookie1}],
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
{ok, SnapBookie1} = leveled_bookie:book_start(SnapOpts1),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
ChkList1 = lists:sublist(lists:sort(ObjList1), 100),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie1, ChkList1),
testutil:check_forlist(SnapBookie1, ChkList1),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Close the snapshot, check the original store still has the objects
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
ok = leveled_bookie:book_close(SnapBookie1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie1, ChkList1),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
ok = leveled_bookie:book_close(Bookie1),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Closed initial bookies~n"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% all now closed
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Open a new store (to start with the previously loaded data)
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
Startup work - baffled Changes the stratup otpions to a prolist to make it easier to get environment variables as default. Tried application:start - and completely baffled as to how to get this to work.
2016-11-02 12:58:27 +00:00
SnapOpts2 = [{snapshot_bookie, Bookie2}],
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% And take a snapshot of that store
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
{ok, SnapBookie2} = leveled_bookie:book_start(SnapOpts2),
io:format("Bookies restarted~n"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Check both the newly opened store and its snapshot have the data
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, ChkList1),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Check active bookie still contains original data~n"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(SnapBookie2, ChkList1),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Check snapshot still contains original data~n"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Generate some replacement objects, load them up - check the master
% store has the replacement objects, but the snapshot still has the old
% objects
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ObjList2 = testutil:generate_objects(5000, 2),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:riakload(Bookie2, ObjList2),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Replacement objects put~n"),
ChkList2 = lists:sublist(lists:sort(ObjList2), 100),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, ChkList2),
testutil:check_forlist(SnapBookie2, ChkList1),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Checked for replacement objects in active bookie" ++
", old objects in snapshot~n"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Check out how many ledger files we now have (should just be 1)
Revert ominshambles performance refactoring To try and improve performance index entries had been removed from the Ledger Cache, and a shadow list of the LedgerCache (in SQN order) was kept to avoid gb_trees:to_list on push_mem. This did not go well. The issue was that ets does not deal with duplicate keys in the list when inserting (it will only insert one, but it is not clear which one). This has been reverted back out. The ETS parameters have been changed to [set, private]. It is not used as an iterator, and is no longer passed out of the process (the memtable_copy is sent instead). This also avoids the tab2list function being called.
2016-10-19 00:10:48 +01:00
ok = filelib:ensure_dir(RootPath ++ "/ledger/ledger_files"),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
{ok, FNsA} = file:list_dir(RootPath ++ "/ledger/ledger_files"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% generate some new objects and load them up. Check that the master store
% has the new objects, and the snapshot doesn't
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ObjList3 = testutil:generate_objects(15000, 5002),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:riakload(Bookie2, ObjList3),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
ChkList3 = lists:sublist(lists:sort(ObjList3), 100),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, ChkList3),
testutil:check_formissinglist(SnapBookie2, ChkList3),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Now loads lots of new objects
GenList = [20002, 40002, 60002, 80002, 100002],
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
CLs2 = testutil:load_objects(20000, GenList, Bookie2, TestObject,
fun testutil:generate_smallobjects/2),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Loaded significant numbers of new objects~n"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, lists:nth(length(CLs2), CLs2)),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Checked active bookie has new objects~n"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Start a second snapshot, which should have the new objects, whilst the
% previous snapshot still doesn't
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
{ok, SnapBookie3} = leveled_bookie:book_start(SnapOpts2),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(SnapBookie3, lists:nth(length(CLs2), CLs2)),
testutil:check_formissinglist(SnapBookie2, ChkList3),
testutil:check_formissinglist(SnapBookie2, lists:nth(length(CLs2), CLs2)),
Revert ominshambles performance refactoring To try and improve performance index entries had been removed from the Ledger Cache, and a shadow list of the LedgerCache (in SQN order) was kept to avoid gb_trees:to_list on push_mem. This did not go well. The issue was that ets does not deal with duplicate keys in the list when inserting (it will only insert one, but it is not clear which one). This has been reverted back out. The ETS parameters have been changed to [set, private]. It is not used as an iterator, and is no longer passed out of the process (the memtable_copy is sent instead). This also avoids the tab2list function being called.
2016-10-19 00:10:48 +01:00
testutil:check_forlist(Bookie2, ChkList2),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(SnapBookie3, ChkList2),
testutil:check_forlist(SnapBookie2, ChkList1),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
io:format("Started new snapshot and check for new objects~n"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Load yet more objects, these are replacement objects for the last load
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
CLs3 = testutil:load_objects(20000, GenList, Bookie2, TestObject,
fun testutil:generate_smallobjects/2),
testutil:check_forlist(Bookie2, lists:nth(length(CLs3), CLs3)),
testutil:check_forlist(Bookie2, lists:nth(1, CLs3)),
Broken snapshot test The test confirming that deleting sft files wer eheld open whilst snapshots were registered was actually broken. This test has now been fixed, as well as the logic in registring snapshots which had used ledger_sqn mistakenly rather than manifest_sqn.
2016-10-21 11:38:30 +01:00
io:format("Starting 15s sleep in which snap2 should block deletion~n"),
timer:sleep(15000),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% There should be lots of ledger files, as we have replaced the objects
% which has created new files, but the old files are still in demand from
% the snapshot
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
{ok, FNsB} = file:list_dir(RootPath ++ "/ledger/ledger_files"),
ok = leveled_bookie:book_close(SnapBookie2),
Broken snapshot test The test confirming that deleting sft files wer eheld open whilst snapshots were registered was actually broken. This test has now been fixed, as well as the logic in registring snapshots which had used ledger_sqn mistakenly rather than manifest_sqn.
2016-10-21 11:38:30 +01:00
io:format("Starting 15s sleep as snap2 close should unblock deletion~n"),
timer:sleep(15000),
io:format("Pause for deletion has ended~n"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% So the pause here is to allow for delete pendings to take effect after the
% closing of the snapshot
% Now check that any deletions haven't impacted the availability of data
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, lists:nth(length(CLs3), CLs3)),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
% Close the other snapshot, and pause - after the pause there should be a
% reduction in the number of ledger files due to the deletes
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
ok = leveled_bookie:book_close(SnapBookie3),
Broken snapshot test The test confirming that deleting sft files wer eheld open whilst snapshots were registered was actually broken. This test has now been fixed, as well as the logic in registring snapshots which had used ledger_sqn mistakenly rather than manifest_sqn.
2016-10-21 11:38:30 +01:00
io:format("Starting 15s sleep as snap3 close should unblock deletion~n"),
timer:sleep(15000),
io:format("Pause for deletion has ended~n"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forlist(Bookie2, lists:nth(length(CLs3), CLs3)),
testutil:check_forlist(Bookie2, lists:nth(1, CLs3)),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
{ok, FNsC} = file:list_dir(RootPath ++ "/ledger/ledger_files"),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
io:format("FNsA ~w FNsB ~w FNsC ~w~n",
[length(FNsA), length(FNsB), length(FNsC)]),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
true = length(FNsB) > length(FNsA),
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
true = length(FNsB) > length(FNsC),
% smaller due to replacements and files deleting
% This is dependent on the sleep though (yuk)
Iterator support Add iterator support, used initially only for retrieving bucket statistics. The iterator is supported by exporting a function, and when the function is claled it will take a snapshot of the ledger, run the iterator and hten close the snapshot. This required a numbe rof underlying changes, in particular to get key comparison to work as "expected". The code had previously misunderstood how comparison worked between Erlang terms, and in particular did not account for tuple length being compared first by size of the tuple (and not just by each element in order).
2016-10-12 17:12:49 +01:00
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{B1Size, B1Count} = testutil:check_bucket_stats(Bookie2, "Bucket1"),
Iterator support Add iterator support, used initially only for retrieving bucket statistics. The iterator is supported by exporting a function, and when the function is claled it will take a snapshot of the ledger, run the iterator and hten close the snapshot. This required a numbe rof underlying changes, in particular to get key comparison to work as "expected". The code had previously misunderstood how comparison worked between Erlang terms, and in particular did not account for tuple length being compared first by size of the tuple (and not just by each element in order).
2016-10-12 17:12:49 +01:00
true = B1Size > 0,
true = B1Count == 1,
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{B1Size, B1Count} = testutil:check_bucket_stats(Bookie2, "Bucket1"),
{BSize, BCount} = testutil:check_bucket_stats(Bookie2, "Bucket"),
Iterator support Add iterator support, used initially only for retrieving bucket statistics. The iterator is supported by exporting a function, and when the function is claled it will take a snapshot of the ledger, run the iterator and hten close the snapshot. This required a numbe rof underlying changes, in particular to get key comparison to work as "expected". The code had previously misunderstood how comparison worked between Erlang terms, and in particular did not account for tuple length being compared first by size of the tuple (and not just by each element in order).
2016-10-12 17:12:49 +01:00
true = BSize > 0,
Add comments to test Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
true = BCount == 120000,
Iterator support Add iterator support, used initially only for retrieving bucket statistics. The iterator is supported by exporting a function, and when the function is claled it will take a snapshot of the ledger, run the iterator and hten close the snapshot. This required a numbe rof underlying changes, in particular to get key comparison to work as "expected". The code had previously misunderstood how comparison worked between Erlang terms, and in particular did not account for tuple length being compared first by size of the tuple (and not just by each element in order).
2016-10-12 17:12:49 +01:00
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
ok = leveled_bookie:book_close(Bookie2),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:reset_filestructure().
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
load_and_count(_Config) ->
% Use artificially small files, and the load keys, counting they're all
% present
Handle L0 cache being full A test thta will cause leveled to crash due to a low cache size being set - but protect against this (as well as the general scenario of the cache being full). There could be a potential case where a L0 file present (post pending) without work backlog being set. In this case we want to roll the level zero to memory, but don't accept the cache update if the L0 cache is already full.
2019-01-14 12:27:51 +00:00
load_and_count(50000000, 2500, 28000),
load_and_count(200000000, 100, 300000).
load_and_count(JournalSize, BookiesMemSize, PencillerMemSize) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
StartOpts1 = [{root_path, RootPath},
Handle L0 cache being full A test thta will cause leveled to crash due to a low cache size being set - but protect against this (as well as the general scenario of the cache being full). There could be a potential case where a L0 file present (post pending) without work backlog being set. In this case we want to roll the level zero to memory, but don't accept the cache update if the L0 cache is already full.
2019-01-14 12:27:51 +00:00
{max_journalsize, JournalSize},
{cache_size, BookiesMemSize},
{max_pencillercachesize, PencillerMemSize},
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{sync_strategy, testutil:sync_strategy()}],
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{TestObject, TestSpec} = testutil:generate_testobject(),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
io:format("Loading initial small objects~n"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
G1 = fun testutil:generate_smallobjects/2,
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
lists:foldl(fun(_X, Acc) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:load_objects(5000,
[Acc + 2],
Bookie1,
TestObject,
G1),
Search range failure Discovered a bug with search ranges in leveled_tree - this was uncovered by an intermittently fialing 19.3 test. Test case added and bug fixed. It was due to a fialure to use end_key passed causing issues with particular manifests and full bucket ranges.
2017-10-24 13:19:30 +01:00
{_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
if
Acc + 5000 == Count ->
ok
end,
Acc + 5000 end,
0,
lists:seq(1, 20)),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
io:format("Loading larger compressible objects~n"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
G2 = fun testutil:generate_compressibleobjects/2,
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
lists:foldl(fun(_X, Acc) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:load_objects(5000,
[Acc + 2],
Bookie1,
TestObject,
G2),
Search range failure Discovered a bug with search ranges in leveled_tree - this was uncovered by an intermittently fialing 19.3 test. Test case added and bug fixed. It was due to a fialure to use end_key passed causing issues with particular manifests and full bucket ranges.
2017-10-24 13:19:30 +01:00
{_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
if
Acc + 5000 == Count ->
ok
end,
Acc + 5000 end,
100000,
lists:seq(1, 20)),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
io:format("Replacing small objects~n"),
lists:foldl(fun(_X, Acc) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:load_objects(5000,
[Acc + 2],
Bookie1,
TestObject,
G1),
Search range failure Discovered a bug with search ranges in leveled_tree - this was uncovered by an intermittently fialing 19.3 test. Test case added and bug fixed. It was due to a fialure to use end_key passed causing issues with particular manifests and full bucket ranges.
2017-10-24 13:19:30 +01:00
{_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
if
Count == 200000 ->
ok
end,
Acc + 5000 end,
0,
lists:seq(1, 20)),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
io:format("Loading more small objects~n"),
lists:foldl(fun(_X, Acc) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:load_objects(5000,
[Acc + 2],
Bookie1,
TestObject,
G2),
Search range failure Discovered a bug with search ranges in leveled_tree - this was uncovered by an intermittently fialing 19.3 test. Test case added and bug fixed. It was due to a fialure to use end_key passed causing issues with particular manifests and full bucket ranges.
2017-10-24 13:19:30 +01:00
{_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
if
Acc + 5000 == Count ->
ok
end,
Acc + 5000 end,
200000,
lists:seq(1, 20)),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
ok = leveled_bookie:book_close(Bookie1),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{_, 300000} = testutil:check_bucket_stats(Bookie2, "Bucket"),
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
ok = leveled_bookie:book_close(Bookie2),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:reset_filestructure().
Load and Count test This test exposed two bugs: - Yet another set of off-by-one errors (really stupidly scanning the Manifest from Level 1 not Level 0) - The return of an old issue related to scanning the journal on load whereby we fail to go back to the previous file before the current SQN
2016-10-13 17:51:47 +01:00
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
load_and_count_withdelete(_Config) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 50000000},
{sync_strategy, testutil:sync_strategy()}],
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{TestObject, TestSpec} = testutil:generate_testobject(),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
io:format("Loading initial small objects~n"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
G1 = fun testutil:generate_smallobjects/2,
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
lists:foldl(fun(_X, Acc) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:load_objects(5000,
[Acc + 2],
Bookie1,
TestObject,
G1),
{_S, Count} = testutil:check_bucket_stats(Bookie1,
"Bucket"),
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
if
Acc + 5000 == Count ->
ok
end,
Acc + 5000 end,
0,
lists:seq(1, 20)),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
Push log update through to cdb/sst Using the cdb_options and sst_options records
2018-12-11 20:42:00 +00:00
{BucketD, KeyD} =
{testutil:get_bucket(TestObject), testutil:get_key(TestObject)},
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{_, 1} = testutil:check_bucket_stats(Bookie1, BucketD),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakdelete(Bookie1, BucketD, KeyD, []),
not_found = testutil:book_riakget(Bookie1, BucketD, KeyD),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{_, 0} = testutil:check_bucket_stats(Bookie1, BucketD),
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
io:format("Loading larger compressible objects~n"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
G2 = fun testutil:generate_compressibleobjects/2,
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
lists:foldl(fun(_X, Acc) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:load_objects(5000,
[Acc + 2],
Bookie1,
no_check,
G2),
{_S, Count} = testutil:check_bucket_stats(Bookie1,
"Bucket"),
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
if
Acc + 5000 == Count ->
ok
end,
Acc + 5000 end,
100000,
lists:seq(1, 20)),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
not_found = testutil:book_riakget(Bookie1, BucketD, KeyD),
Delete Add functionality to delete keys. No tombstone reaping yet.
2016-10-16 15:41:09 +01:00
ok = leveled_bookie:book_close(Bookie1),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_formissingobject(Bookie2, BucketD, KeyD),
Test coverage - missing 1%
2017-02-26 20:52:40 +00:00
testutil:check_formissingobject(Bookie2, "Bookie1", "MissingKey0123"),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{_BSize, 0} = testutil:check_bucket_stats(Bookie2, BucketD),
ok = leveled_bookie:book_close(Bookie2),
testutil:reset_filestructure().
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
Recovery strategy testing Test added for the "retain" recovery strategy. This strategy makes sure a full history of index changes is made so that if the Ledger is wiped out, the Ledger cna be fully rebuilt from the Journal. This exposed two journal compaction problems - The BestRun selected did not have the source files correctly sorted in order before compaction - The compaction process incorrectly dealt with the KeyDelta object left after a compaction - i.e. compacting twice the same key caused that key history to be lost. These issues have now been corrected.
2016-10-27 00:57:19 +01:00
space_clear_ondelete(_Config) ->
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
RootPath = testutil:reset_filestructure(),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
StartOpts1 = [{root_path, RootPath},
Change default compaction settings Need to allow specific settings to be passed into unit tests. Also, too much journal compaction may lead to intermittent failures on the basic_SUITE space_clear_on_delete test. think this is because there are less “deletes” to reload in on startup to trigger the cascade down and clear up?
2017-06-02 08:37:57 +01:00
{max_journalsize, 10000000},
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{sync_strategy, testutil:sync_strategy()}],
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
{ok, Book1} = leveled_bookie:book_start(StartOpts1),
G2 = fun testutil:generate_compressibleobjects/2,
testutil:load_objects(20000,
[uuid, uuid, uuid, uuid],
Book1,
no_check,
G2),
Fix bad accumulator in test Test takes a long time due to sleep (still need to work on that), but also FoldKeysFun uses ++ rather than [{B, K}|Acc] to extend the list. Order of magnitude speed-up for these queries by changing the way this accumulates
2018-05-01 22:30:28 +01:00
FoldKeysFun = fun(B, K, Acc) -> [{B, K}|Acc] end,
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, F1} = leveled_bookie:book_keylist(Book1, o_rkv, {FoldKeysFun, []}),
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
SW1 = os:timestamp(),
KL1 = F1(),
ok = case length(KL1) of
80000 ->
io:format("Key list took ~w microseconds for 80K keys~n",
[timer:now_diff(os:timestamp(), SW1)]),
ok
end,
timer:sleep(10000), % Allow for any L0 file to be rolled
{ok, FNsA_L} = file:list_dir(RootPath ++ "/ledger/ledger_files"),
{ok, FNsA_J} = file:list_dir(RootPath ++ "/journal/journal_files"),
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
io:format("FNsA - Bookie created ~w journal files and ~w ledger files~n",
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
[length(FNsA_J), length(FNsA_L)]),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
Registering and releasing of Journal snapshots Added a test of journal compaction with a registered snapshot and it showed that the deleting of files did not correctly check the list of registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
% Get an iterator to lock the inker during compaction
Switch to binary format Riak object Initial change to try and test assuming that leveled received the binary format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
FoldObjectsFun = fun(B, K, ObjBin, Acc) ->
[{B, K, erlang:phash2(ObjBin)}|Acc] end,
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, HTreeF1} = leveled_bookie:book_objectfold(Book1,
?RIAK_TAG,
{FoldObjectsFun, []},
false),
Fix bad accumulator in test Test takes a long time due to sleep (still need to work on that), but also FoldKeysFun uses ++ rather than [{B, K}|Acc] to extend the list. Order of magnitude speed-up for these queries by changing the way this accumulates
2018-05-01 22:30:28 +01:00
% This query does not Snap PreFold - and so will not prevent
% pending deletes from prompting actual deletes
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, KF1} = leveled_bookie:book_keylist(Book1, o_rkv, {FoldKeysFun, []}),
Fix bad accumulator in test Test takes a long time due to sleep (still need to work on that), but also FoldKeysFun uses ++ rather than [{B, K}|Acc] to extend the list. Order of magnitude speed-up for these queries by changing the way this accumulates
2018-05-01 22:30:28 +01:00
% This query does Snap PreFold, and so will prevent deletes from
% the ledger
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
Registering and releasing of Journal snapshots Added a test of journal compaction with a registered snapshot and it showed that the deleting of files did not correctly check the list of registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
% Delete the keys
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
SW2 = os:timestamp(),
lists:foreach(fun({Bucket, Key}) ->
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:book_riakdelete(Book1,
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
Bucket,
Key,
[])
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
end,
KL1),
io:format("Deletion took ~w microseconds for 80K keys~n",
[timer:now_diff(os:timestamp(), SW2)]),
Registering and releasing of Journal snapshots Added a test of journal compaction with a registered snapshot and it showed that the deleting of files did not correctly check the list of registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
ok = leveled_bookie:book_compactjournal(Book1, 30000),
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
F = fun leveled_bookie:book_islastcompactionpending/1,
lists:foldl(fun(X, Pending) ->
case Pending of
false ->
false;
true ->
io:format("Loop ~w waiting for journal "
++ "compaction to complete~n", [X]),
timer:sleep(20000),
F(Book1)
end end,
true,
lists:seq(1, 15)),
Registering and releasing of Journal snapshots Added a test of journal compaction with a registered snapshot and it showed that the deleting of files did not correctly check the list of registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
io:format("Waiting for journal deletes - blocked~n"),
timer:sleep(20000),
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
Fix bad accumulator in test Test takes a long time due to sleep (still need to work on that), but also FoldKeysFun uses ++ rather than [{B, K}|Acc] to extend the list. Order of magnitude speed-up for these queries by changing the way this accumulates
2018-05-01 22:30:28 +01:00
io:format("Sleep over - Fold Objects query ~n"),
% for this query snapshot is made at fold time, and so the results are
% empty
Split code for folders - introduce runner actor Introduce a dedicated module for all the different fold types. Also simplify the list of folders by deprecating those folds that should eb achieveable by fold_heads/fold_objects type folds but with smarter functions. Makes sure that the fold functiosn also have better spec coverage, and are dialyzer checked.
2017-10-17 20:39:11 +01:00
true = length(HTreeF1()) == 0,
Fix bad accumulator in test Test takes a long time due to sleep (still need to work on that), but also FoldKeysFun uses ++ rather than [{B, K}|Acc] to extend the list. Order of magnitude speed-up for these queries by changing the way this accumulates
2018-05-01 22:30:28 +01:00
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
% This query uses a genuine async fold on a snasphot made at request time
Fix bad accumulator in test Test takes a long time due to sleep (still need to work on that), but also FoldKeysFun uses ++ rather than [{B, K}|Acc] to extend the list. Order of magnitude speed-up for these queries by changing the way this accumulates
2018-05-01 22:30:28 +01:00
% and so the results should be non-empty
io:format("Now Query 2 - Fold Keys query~n"),
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
true = length(KF1()) == 80000,
Registering and releasing of Journal snapshots Added a test of journal compaction with a registered snapshot and it showed that the deleting of files did not correctly check the list of registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
io:format("Waiting for journal deletes - unblocked~n"),
timer:sleep(20000),
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
{ok, FNsB_L} = file:list_dir(RootPath ++ "/ledger/ledger_files"),
{ok, FNsB_J} = file:list_dir(RootPath ++ "/journal/journal_files"),
Registering and releasing of Journal snapshots Added a test of journal compaction with a registered snapshot and it showed that the deleting of files did not correctly check the list of registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
{ok, FNsB_PC} = file:list_dir(RootPath
++ "/journal/journal_files/post_compact"),
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
PointB_Journals = length(FNsB_J) + length(FNsB_PC),
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
io:format("FNsB - Bookie has ~w journal files and ~w ledger files " ++
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
"after deletes~n",
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
[PointB_Journals, length(FNsB_L)]),
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, F2} = leveled_bookie:book_keylist(Book1, o_rkv, {FoldKeysFun, []}),
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
SW3 = os:timestamp(),
KL2 = F2(),
ok = case length(KL2) of
0 ->
io:format("Key list took ~w microseconds for no keys~n",
[timer:now_diff(os:timestamp(), SW3)]),
ok
end,
ok = leveled_bookie:book_close(Book1),
{ok, Book2} = leveled_bookie:book_start(StartOpts1),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, F3} = leveled_bookie:book_keylist(Book2, o_rkv, {FoldKeysFun, []}),
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
SW4 = os:timestamp(),
KL3 = F3(),
ok = case length(KL3) of
0 ->
io:format("Key list took ~w microseconds for no keys~n",
[timer:now_diff(os:timestamp(), SW4)]),
ok
end,
ok = leveled_bookie:book_close(Book2),
{ok, FNsC_L} = file:list_dir(RootPath ++ "/ledger/ledger_files"),
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
io:format("FNsC - Bookie has ~w ledger files " ++
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
"after close~n", [length(FNsC_L)]),
{ok, Book3} = leveled_bookie:book_start(StartOpts1),
io:format("This should cause a final ledger merge event~n"),
io:format("Will require the penciller to resolve the issue of creating" ++
" an empty file as all keys compact on merge~n"),
Roll in fix for intermittently failing test As descibed in https://github.com/martinsumner/leveled/issues/92 Only the first fix was made. Just to eb safe - archiving means renaming to another file with a different extension. Assumption is that renamed files cna be manually reaped if necessary.
2017-09-27 23:52:49 +01:00
CheckFun =
fun(X, FileCount) ->
case FileCount of
0 ->
0;
_ ->
timer:sleep(X),
{ok, NewFC} =
file:list_dir(RootPath ++ "/ledger/ledger_files"),
io:format("Looping with ledger file count ~w~n",
[length(NewFC)]),
length(strip_nonsst(NewFC))
end
end,
FC = lists:foldl(CheckFun, infinity, [2000, 3000, 5000, 8000]),
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
ok = leveled_bookie:book_close(Book3),
Roll in fix for intermittently failing test As descibed in https://github.com/martinsumner/leveled/issues/92 Only the first fix was made. Just to eb safe - archiving means renaming to another file with a different extension. Assumption is that renamed files cna be manually reaped if necessary.
2017-09-27 23:52:49 +01:00
case FC of
0 ->
ok;
_ ->
{ok, Book4} = leveled_bookie:book_start(StartOpts1),
lists:foldl(CheckFun, infinity, [2000, 3000, 5000, 8000]),
leveled_bookie:book_close(Book4)
end,
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
{ok, FNsD_L} = file:list_dir(RootPath ++ "/ledger/ledger_files"),
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
io:format("FNsD - Bookie has ~w ledger files " ++
Roll in fix for intermittently failing test As descibed in https://github.com/martinsumner/leveled/issues/92 Only the first fix was made. Just to eb safe - archiving means renaming to another file with a different extension. Assumption is that renamed files cna be manually reaped if necessary.
2017-09-27 23:52:49 +01:00
"after second close~n", [length(strip_nonsst(FNsD_L))]),
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
lists:foreach(fun(FN) ->
Change default compaction settings Need to allow specific settings to be passed into unit tests. Also, too much journal compaction may lead to intermittent failures on the basic_SUITE space_clear_on_delete test. think this is because there are less “deletes” to reload in on startup to trigger the cascade down and clear up?
2017-06-02 08:37:57 +01:00
io:format("FNsD - Ledger file is ~s~n", [FN])
Correct async fold fold objects which snaps in the fold was implemented incorrectly - it took information from the LedgeCache at the point of the request, not at the point of the fold. So the LedgerCache SQN may have been surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
end,
FNsD_L),
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
true = PointB_Journals < length(FNsA_J),
Roll in fix for intermittently failing test As descibed in https://github.com/martinsumner/leveled/issues/92 Only the first fix was made. Just to eb safe - archiving means renaming to another file with a different extension. Assumption is that renamed files cna be manually reaped if necessary.
2017-09-27 23:52:49 +01:00
true = length(strip_nonsst(FNsD_L)) < length(strip_nonsst(FNsA_L)),
true = length(strip_nonsst(FNsD_L)) < length(strip_nonsst(FNsB_L)),
true = length(strip_nonsst(FNsD_L)) < length(strip_nonsst(FNsC_L)),
true = length(strip_nonsst(FNsD_L)) == 0.
Add test for is_empty Bucket listing didn't care if keys were active - now does.
2017-05-23 11:59:44 +01:00
Roll in fix for intermittently failing test As descibed in https://github.com/martinsumner/leveled/issues/92 Only the first fix was made. Just to eb safe - archiving means renaming to another file with a different extension. Assumption is that renamed files cna be manually reaped if necessary.
2017-09-27 23:52:49 +01:00
strip_nonsst(FileList) ->
SSTOnlyFun =
fun(FN, Acc) ->
case filename:extension(FN) of
".sst" ->
[FN|Acc];
_ ->
Acc
end
end,
lists:foldl(SSTOnlyFun, [], FileList).
Add test for is_empty Bucket listing didn't care if keys were active - now does.
2017-05-23 11:59:44 +01:00
is_empty_test(_Config) ->
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{sync_strategy, testutil:sync_strategy()}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{B1, K1, V1, Spec, MD} = {term_to_binary("Bucket1"),
term_to_binary("Key1"),
"Value1",
[],
[{"MDK1", "MDV1"}]},
{TestObject1, TestSpec1} =
testutil:generate_testobject(B1, K1, V1, Spec, MD),
{B1, K2, V2, Spec, MD} = {term_to_binary("Bucket1"),
term_to_binary("Key2"),
"Value2",
[],
[{"MDK1", "MDV1"}]},
{TestObject2, TestSpec2} =
testutil:generate_testobject(B1, K2, V2, Spec, MD),
{B2, K3, V3, Spec, MD} = {term_to_binary("Bucket2"),
term_to_binary("Key3"),
"Value3",
[],
[{"MDK1", "MDV1"}]},
{TestObject3, TestSpec3} =
testutil:generate_testobject(B2, K3, V3, Spec, MD),
ok = testutil:book_riakput(Bookie1, TestObject1, TestSpec1),
ok = testutil:book_riakput(Bookie1, TestObject2, TestSpec2),
ok = testutil:book_riakput(Bookie1, TestObject3, TestSpec3),
FoldBucketsFun = fun(B, Acc) -> sets:add_element(B, Acc) end,
Switch from binary_bucketlist Allow for bucket listing of non-binary buckets (integer buckets, buckets with ascii strings)
2018-09-01 10:39:23 +01:00
BucketListQuery = {bucket_list,
Add test for is_empty Bucket listing didn't care if keys were active - now does.
2017-05-23 11:59:44 +01:00
?RIAK_TAG,
{FoldBucketsFun, sets:new()}},
{async, BL} = leveled_bookie:book_returnfolder(Bookie1, BucketListQuery),
true = sets:size(BL()) == 2,
ok = leveled_bookie:book_put(Bookie1, B2, K3, delete, [], ?RIAK_TAG),
{async, BLpd1} = leveled_bookie:book_returnfolder(Bookie1, BucketListQuery),
true = sets:size(BLpd1()) == 1,
ok = leveled_bookie:book_put(Bookie1, B1, K2, delete, [], ?RIAK_TAG),
{async, BLpd2} = leveled_bookie:book_returnfolder(Bookie1, BucketListQuery),
true = sets:size(BLpd2()) == 1,
ok = leveled_bookie:book_put(Bookie1, B1, K1, delete, [], ?RIAK_TAG),
{async, BLpd3} = leveled_bookie:book_returnfolder(Bookie1, BucketListQuery),
true = sets:size(BLpd3()) == 0,
ok = leveled_bookie:book_close(Bookie1).
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
Test to expose startup issue See #194
2018-10-03 18:29:20 +01:00
remove_journal_test(_Config) ->
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{max_pencillercachesize, 16000},
{sync_strategy, testutil:sync_strategy()},
{compression_point, on_compact}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
GenList = [1, 20001, 40001, 60001],
CLs = testutil:load_objects(20000, GenList, Bookie1, no_check,
fun testutil:generate_smallobjects/2),
CheckList1 = lists:sublist(lists:nth(1, CLs), 100, 1000),
CheckList2 = lists:sublist(lists:nth(2, CLs), 100, 1000),
CheckList3 = lists:sublist(lists:nth(3, CLs), 100, 1000),
CheckList4 = lists:sublist(lists:nth(4, CLs), 100, 1000),
testutil:check_forlist(Bookie1, CheckList1),
testutil:check_forlist(Bookie1, CheckList2),
testutil:check_forlist(Bookie1, CheckList3),
testutil:check_forlist(Bookie1, CheckList4),
ok = leveled_bookie:book_close(Bookie1),
leveled_inker:clean_testdir(RootPath ++ "/journal"),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
% If we're not careful here new data will be added, and we
% won't be able to read it
[NewCheckList] =
testutil:load_objects(1000, [80001], Bookie2, no_check,
fun testutil:generate_smallobjects/2),
ok = leveled_bookie:book_close(Bookie2),
{ok, Bookie3} = leveled_bookie:book_start(StartOpts1),
testutil:check_forlist(Bookie3, NewCheckList),
ok = leveled_bookie:book_destroy(Bookie3).
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
many_put_fetch_switchcompression(_Config) ->
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{max_pencillercachesize, 16000},
{sync_strategy, riak_sync},
{compression_method, native}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{TestObject, TestSpec} = testutil:generate_testobject(),
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
testutil:check_forobject(Bookie1, TestObject),
ok = leveled_bookie:book_close(Bookie1),
StartOpts2 = [{root_path, RootPath},
{max_journalsize, 500000000},
{max_pencillercachesize, 32000},
{sync_strategy, testutil:sync_strategy()},
{compression_method, lz4}],
%% Change compression method
{ok, Bookie2} = leveled_bookie:book_start(StartOpts2),
testutil:check_forobject(Bookie2, TestObject),
GenList = [2, 40002, 80002, 120002],
CLs = testutil:load_objects(40000, GenList, Bookie2, TestObject,
fun testutil:generate_smallobjects/2),
CL1A = lists:nth(1, CLs),
ChkListFixed = lists:nth(length(CLs), CLs),
testutil:check_forlist(Bookie2, CL1A),
ObjList2A = testutil:generate_objects(5000, 2),
testutil:riakload(Bookie2, ObjList2A),
ChkList2A = lists:sublist(lists:sort(ObjList2A), 1000),
testutil:check_forlist(Bookie2, ChkList2A),
testutil:check_forlist(Bookie2, ChkListFixed),
testutil:check_forobject(Bookie2, TestObject),
testutil:check_forlist(Bookie2, ChkList2A),
testutil:check_forlist(Bookie2, ChkListFixed),
testutil:check_forobject(Bookie2, TestObject),
ok = leveled_bookie:book_close(Bookie2),
%% Change method back again
{ok, Bookie3} = leveled_bookie:book_start(StartOpts1),
testutil:check_forlist(Bookie3, ChkList2A),
testutil:check_forlist(Bookie3, ChkListFixed),
testutil:check_forobject(Bookie3, TestObject),
testutil:check_formissingobject(Bookie3, "Bookie1", "MissingKey0123"),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
ok = leveled_bookie:book_destroy(Bookie3).
Add ct test Which passes
2018-09-27 11:32:42 +01:00
safereaderror_startup(_Config) ->
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{compression_point, on_compact},
{max_journalsize, 1000}, {cache_size, 2060}],
{ok, Bookie1} = leveled_bookie:book_plainstart(StartOpts1),
B1 = <<98, 117, 99, 107, 101, 116, 51>>,
K1 =
<<38, 50, 201, 47, 167, 125, 57, 232, 84, 38, 14, 114, 24, 62,
12, 74>>,
Obj1 =
<<87, 150, 217, 230, 4, 81, 170, 68, 181, 224, 60, 232, 4, 74,
159, 12, 156, 56, 194, 181, 18, 158, 195, 207, 106, 191, 80,
111, 100, 81, 252, 248>>,
Obj2 =
<<86, 201, 253, 149, 213, 10, 32, 166, 33, 136, 42, 79, 103, 250,
139, 95, 42, 143, 161, 3, 185, 74, 149, 226, 232, 214, 183, 64,
69, 56, 167, 78>>,
ok = leveled_bookie:book_put(Bookie1, B1, K1, Obj1, []),
ok = leveled_bookie:book_put(Bookie1, B1, K1, Obj2, []),
exit(Bookie1, kill),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
{ok, ReadBack} = leveled_bookie:book_get(Bookie2, B1, K1),
io:format("Read back ~w", [ReadBack]),
true = ReadBack == Obj2,
ok = leveled_bookie:book_close(Bookie2).
Reference in a new issue Copy permalink