umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
leveled/test/end_to_end/tictac_SUITE.erl

886 lines
34 KiB
Erlang
Raw Normal View History

Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
-module(tictac_SUITE).
Switch to logger (#442) * Switch to logger Use logger rather than io:format when logging. The ct tests have besn switched to log to file, testutil/init_per_suite/1 may offer useful guidance on configuring logger with leveled. As all logs are produced by the leveled_log module, the MFA metadata is uninteresting for log outputs, but can be used for explicit filter controls for leveled logs. * iolist_to_binary not unicode_binary() logger filters will be error and be removed if the format line is a binary(). Must be either a charlist() or a unicode_binary() - so iolist_to_binary() can't be used * Add metadata for filter * Update test/end_to_end/tictac_SUITE.erl Co-authored-by: Thomas Arts <thomas.arts@quviq.com> --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-09-06 11:18:24 +01:00
-include("leveled.hrl").
-export([all/0, init_per_suite/1, end_per_suite/1]).
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
-export([
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
many_put_compare/1,
Support for recent AAE index With basic ct test. Doesn't currently prove expiry of index. Doesn't prove ability to find segments. Assumes that either "all" buckets or a special list of buckets require indexing this way. Will lead to unexpected results if the same bucket name is used across different Tags. The format of the index has been chosen so that hopeully standard index features can be used (e.g. return_terms).
2017-06-30 16:31:22 +01:00
index_compare/1,
Check segment is as expected with tuplebuckets In head_only mode
2018-11-05 10:31:15 +00:00
basic_headonly/1,
tuplebuckets_headonly/1
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
]).
all() -> [
Re-introduce commented tests
2017-06-30 16:31:48 +01:00
many_put_compare,
index_compare,
Check segment is as expected with tuplebuckets In head_only mode
2018-11-05 10:31:15 +00:00
basic_headonly,
tuplebuckets_headonly
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
].
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
-define(V1_VERS, 1).
-define(MAGIC, 53). % riak_kv -> riak_object
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
Switch to logger (#442) * Switch to logger Use logger rather than io:format when logging. The ct tests have besn switched to log to file, testutil/init_per_suite/1 may offer useful guidance on configuring logger with leveled. As all logs are produced by the leveled_log module, the MFA metadata is uninteresting for log outputs, but can be used for explicit filter controls for leveled logs. * iolist_to_binary not unicode_binary() logger filters will be error and be removed if the format line is a binary(). Must be either a charlist() or a unicode_binary() - so iolist_to_binary() can't be used * Add metadata for filter * Update test/end_to_end/tictac_SUITE.erl Co-authored-by: Thomas Arts <thomas.arts@quviq.com> --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-09-06 11:18:24 +01:00
init_per_suite(Config) ->
testutil:init_per_suite([{suite, "tictac"}|Config]),
Config.
end_per_suite(Config) ->
testutil:end_per_suite(Config).
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
many_put_compare(_Config) ->
Switch to logger (#442) * Switch to logger Use logger rather than io:format when logging. The ct tests have besn switched to log to file, testutil/init_per_suite/1 may offer useful guidance on configuring logger with leveled. As all logs are produced by the leveled_log module, the MFA metadata is uninteresting for log outputs, but can be used for explicit filter controls for leveled logs. * iolist_to_binary not unicode_binary() logger filters will be error and be removed if the format line is a binary(). Must be either a charlist() or a unicode_binary() - so iolist_to_binary() can't be used * Add metadata for filter * Update test/end_to_end/tictac_SUITE.erl Co-authored-by: Thomas Arts <thomas.arts@quviq.com> --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-09-06 11:18:24 +01:00
Flexible TicTacTree sizes Allow tictac tree sizes to be flexible. Tested lots of different sizes. Having both level 1 and level 2 the same size seemed to be consistently quicker than trying to make either of the levels relatively wider. There's an 8% performance improvement if the SegmentCount is reduced by a quarter.
2017-06-20 10:58:13 +01:00
TreeSize = small,
SegmentCount = 256 * 256,
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% Test requires multiple different databases, so want to mount them all
% on individual file paths
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
RootPathA = testutil:reset_filestructure("testA"),
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
RootPathB = testutil:reset_filestructure("testB"),
RootPathC = testutil:reset_filestructure("testC"),
RootPathD = testutil:reset_filestructure("testD"),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% Start the first database, load a test object, close it, start it again
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
StartOpts1 = [{root_path, RootPathA},
{max_pencillercachesize, 16000},
{sync_strategy, riak_sync}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
corrected test differing object was in wrong bucket
2017-06-19 13:11:43 +01:00
{B1, K1, V1, S1, MD} = {"Bucket",
"Key1.1.4567.4321",
"Value1",
[],
[{"MDK1", "MDV1"}]},
{TestObject, TestSpec} = testutil:generate_testobject(B1, K1, V1, S1, MD),
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
testutil:check_forobject(Bookie1, TestObject),
ok = leveled_bookie:book_close(Bookie1),
StartOpts2 = [{root_path, RootPathA},
{max_journalsize, 500000000},
{max_pencillercachesize, 32000},
{sync_strategy, testutil:sync_strategy()}],
{ok, Bookie2} = leveled_bookie:book_start(StartOpts2),
testutil:check_forobject(Bookie2, TestObject),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% Generate 200K objects to be sued within the test, and load them into
% the first store (outputting the generated objects as a list of lists)
% to be used elsewhere
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
GenList = [2, 20002, 40002, 60002, 80002,
100002, 120002, 140002, 160002, 180002],
CLs = testutil:load_objects(20000,
GenList,
Bookie2,
TestObject,
fun testutil:generate_smallobjects/2,
20000),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% Start a new store, and load the same objects (except fot the original
% test object) into this store
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
StartOpts3 = [{root_path, RootPathB},
{max_journalsize, 200000000},
{max_pencillercachesize, 16000},
{sync_strategy, testutil:sync_strategy()}],
{ok, Bookie3} = leveled_bookie:book_start(StartOpts3),
lists:foreach(fun(ObjL) -> testutil:riakload(Bookie3, ObjL) end, CLs),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
% Now run a tictac query against both stores to see the extent to which
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% state between stores is consistent
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
TicTacQ = {tictactree_obj,
Improve test coverage
2017-10-17 22:06:30 +01:00
{o_rkv, "Bucket", null, null, true},
Flexible TicTacTree sizes Allow tictac tree sizes to be flexible. Tested lots of different sizes. Having both level 1 and level 2 the same size seemed to be consistently quicker than trying to make either of the levels relatively wider. There's an 8% performance improvement if the SegmentCount is reduced by a quarter.
2017-06-20 10:58:13 +01:00
TreeSize,
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
fun(_B, _K) -> accumulate end},
{async, TreeAFolder} = leveled_bookie:book_returnfolder(Bookie2, TicTacQ),
{async, TreeBFolder} = leveled_bookie:book_returnfolder(Bookie3, TicTacQ),
SWA0 = os:timestamp(),
TreeA = TreeAFolder(),
io:format("Build tictac tree with 200K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWA0)]),
SWB0 = os:timestamp(),
TreeB = TreeBFolder(),
io:format("Build tictac tree with 200K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWB0)]),
SWC0 = os:timestamp(),
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
SegList0 = leveled_tictac:find_dirtyleaves(TreeA, TreeB),
Initial code with busted ct test Initiat comparison made betwene trees externally - but ct test is bust.
2017-06-19 11:36:57 +01:00
io:format("Compare tictac trees with 200K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWC0)]),
io:format("Tree comparison shows ~w different leaves~n",
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
[length(SegList0)]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
AltList =
leveled_tictac:find_dirtyleaves(TreeA,
leveled_tictac:new_tree(0, TreeSize)),
corrected test differing object was in wrong bucket
2017-06-19 13:11:43 +01:00
io:format("Tree comparison shows ~w altered leaves~n",
[length(AltList)]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
true = length(SegList0) == 1,
corrected test differing object was in wrong bucket
2017-06-19 13:11:43 +01:00
% only the test object should be different
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
true = length(AltList) > 10000,
Finding keys Progresses from a segment list to scanning for the keys in that segment
2017-06-19 18:38:55 +01:00
% check there are a significant number of differences from empty
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Improve test coverage
2017-10-17 22:06:30 +01:00
WrongPartitionTicTacQ = {tictactree_obj,
{o_rkv, "Bucket", null, null, false},
TreeSize,
fun(_B, _K) -> pass end},
{async, TreeAFolder_WP} =
leveled_bookie:book_returnfolder(Bookie2, WrongPartitionTicTacQ),
TreeAWP = TreeAFolder_WP(),
DoubleEmpty =
leveled_tictac:find_dirtyleaves(TreeAWP,
leveled_tictac:new_tree(0, TreeSize)),
true = length(DoubleEmpty) == 0,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
% Now run the same query by putting the tree-building responsibility onto
% the fold_objects_fun
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
Add exportable option to tictac Idea being that sometimes you may wish to compare a tictac tree between leveled and something that doesn't understand erlang:phash or term_to_binary. So allow the magic_hash to be used instead - and perhaps an extract function that does base64 encoding or something similar.
2017-09-26 22:49:40 +01:00
ExtractClockFun =
fun(Key, Value) ->
{proxy_object, HeadBin, _Size, _FetchFun} = binary_to_term(Value),
<<?MAGIC:8/integer, ?V1_VERS:8/integer, VclockLen:32/integer,
VclockBin:VclockLen/binary, _Rest/binary>> = HeadBin,
Attempt to standardise on segment hashes To allow for the segment has that accelerates queries to be re-used in tictac tree related queries.
2017-10-30 13:57:41 +00:00
case is_binary(Key) of
true ->
{Key,
lists:sort(binary_to_term(VclockBin))};
false ->
{term_to_binary(Key),
lists:sort(binary_to_term(VclockBin))}
end
Add exportable option to tictac Idea being that sometimes you may wish to compare a tictac tree between leveled and something that doesn't understand erlang:phash or term_to_binary. So allow the magic_hash to be used instead - and perhaps an extract function that does base64 encoding or something similar.
2017-09-26 22:49:40 +01:00
end,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
FoldObjectsFun =
fun(_Bucket, Key, Value, Acc) ->
Attempt to standardise on segment hashes To allow for the segment has that accelerates queries to be re-used in tictac tree related queries.
2017-10-30 13:57:41 +00:00
leveled_tictac:add_kv(Acc, Key, Value, ExtractClockFun)
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
end,
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
FoldAccT = {FoldObjectsFun, leveled_tictac:new_tree(0, TreeSize)},
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
{async, TreeAObjFolder0} =
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
leveled_bookie:book_headfold(Bookie2,
o_rkv,
{range, "Bucket", all},
FoldAccT,
false,
true,
false),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
SWB0Obj = os:timestamp(),
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
TreeAObj0 = TreeAObjFolder0(),
io:format("Build tictac tree via object fold with no "++
"presence check and 200K objects in ~w~n",
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
[timer:now_diff(os:timestamp(), SWB0Obj)]),
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
true = length(leveled_tictac:find_dirtyleaves(TreeA, TreeAObj0)) == 0,
Fix issues with deprecated folders They were deprecated for a reason
2018-10-31 11:04:23 +00:00
InitAccTree = leveled_tictac:new_tree(0, TreeSize),
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
{async, TreeAObjFolder1} =
Fix issues with deprecated folders They were deprecated for a reason
2018-10-31 11:04:23 +00:00
leveled_bookie:book_headfold(Bookie2,
?RIAK_TAG,
{range, "Bucket", all},
{FoldObjectsFun,
InitAccTree},
true, true, false),
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
SWB1Obj = os:timestamp(),
TreeAObj1 = TreeAObjFolder1(),
io:format("Build tictac tree via object fold with "++
"presence check and 200K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWB1Obj)]),
true = length(leveled_tictac:find_dirtyleaves(TreeA, TreeAObj1)) == 0,
Add exportable option to tictac Idea being that sometimes you may wish to compare a tictac tree between leveled and something that doesn't understand erlang:phash or term_to_binary. So allow the magic_hash to be used instead - and perhaps an extract function that does base64 encoding or something similar.
2017-09-26 22:49:40 +01:00
% For an exportable comparison, want hash to be based on something not
% coupled to erlang language - so use exportable query
AltExtractFun =
fun(K, V) ->
{proxy_object, HeadBin, _Size, _FetchFun} = binary_to_term(V),
<<?MAGIC:8/integer, ?V1_VERS:8/integer, VclockLen:32/integer,
VclockBin:VclockLen/binary, _Rest/binary>> = HeadBin,
{term_to_binary(K), VclockBin}
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
end,
AltFoldObjectsFun =
fun(_Bucket, Key, Value, Acc) ->
Attempt to standardise on segment hashes To allow for the segment has that accelerates queries to be re-used in tictac tree related queries.
2017-10-30 13:57:41 +00:00
leveled_tictac:add_kv(Acc, Key, Value, AltExtractFun)
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
end,
Fix issues with deprecated folders They were deprecated for a reason
2018-10-31 11:04:23 +00:00
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
{async, TreeAAltObjFolder0} =
Fix issues with deprecated folders They were deprecated for a reason
2018-10-31 11:04:23 +00:00
leveled_bookie:book_headfold(Bookie2,
?RIAK_TAG,
{range, "Bucket", all},
{AltFoldObjectsFun,
InitAccTree},
false, true, false),
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
SWB2Obj = os:timestamp(),
TreeAAltObj = TreeAAltObjFolder0(),
io:format("Build tictac tree via object fold with no "++
"presence check and 200K objects and alt hash in ~w~n",
[timer:now_diff(os:timestamp(), SWB2Obj)]),
{async, TreeBAltObjFolder0} =
Fix issues with deprecated folders They were deprecated for a reason
2018-10-31 11:04:23 +00:00
leveled_bookie:book_headfold(Bookie3,
?RIAK_TAG,
{range, "Bucket", all},
{AltFoldObjectsFun,
InitAccTree},
false, true, false),
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
SWB3Obj = os:timestamp(),
TreeBAltObj = TreeBAltObjFolder0(),
io:format("Build tictac tree via object fold with no "++
"presence check and 200K objects and alt hash in ~w~n",
[timer:now_diff(os:timestamp(), SWB3Obj)]),
Add exportable option to tictac Idea being that sometimes you may wish to compare a tictac tree between leveled and something that doesn't understand erlang:phash or term_to_binary. So allow the magic_hash to be used instead - and perhaps an extract function that does base64 encoding or something similar.
2017-09-26 22:49:40 +01:00
DL_ExportFold =
length(leveled_tictac:find_dirtyleaves(TreeBAltObj, TreeAAltObj)),
io:format("Found dirty leaves with exportable comparison of ~w~n",
[DL_ExportFold]),
true = DL_ExportFold == 1,
Add tests using fold_heads Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads. The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU. However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends. Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
2017-08-07 10:45:41 +01:00
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
%% Finding differing keys
Finding keys Progresses from a segment list to scanning for the keys in that segment
2017-06-19 18:38:55 +01:00
FoldKeysFun =
fun(SegListToFind) ->
Revert passing Bucket Bad edit. Reverted
2017-07-03 19:06:41 +01:00
fun(_B, K, Acc) ->
Attempt to standardise on segment hashes To allow for the segment has that accelerates queries to be re-used in tictac tree related queries.
2017-10-30 13:57:41 +00:00
Seg = get_segment(K, SegmentCount),
Finding keys Progresses from a segment list to scanning for the keys in that segment
2017-06-19 18:38:55 +01:00
case lists:member(Seg, SegListToFind) of
true ->
Revert passing Bucket Bad edit. Reverted
2017-07-03 19:06:41 +01:00
[K|Acc];
Finding keys Progresses from a segment list to scanning for the keys in that segment
2017-06-19 18:38:55 +01:00
false ->
Acc
end
end
end,
SegQuery = {keylist, o_rkv, "Bucket", {FoldKeysFun(SegList0), []}},
{async, SegKeyFinder} =
leveled_bookie:book_returnfolder(Bookie2, SegQuery),
SWSKL0 = os:timestamp(),
SegKeyList = SegKeyFinder(),
io:format("Finding ~w keys in ~w dirty segments in ~w~n",
[length(SegKeyList),
length(SegList0),
timer:now_diff(os:timestamp(), SWSKL0)]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Finding keys Progresses from a segment list to scanning for the keys in that segment
2017-06-19 18:38:55 +01:00
true = length(SegKeyList) >= 1,
true = length(SegKeyList) < 10,
true = lists:member("Key1.1.4567.4321", SegKeyList),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% Now remove the object which represents the difference between these
% stores and confirm that the tictac trees will now match
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
corrected test differing object was in wrong bucket
2017-06-19 13:11:43 +01:00
testutil:book_riakdelete(Bookie2, B1, K1, []),
{async, TreeAFolder0} = leveled_bookie:book_returnfolder(Bookie2, TicTacQ),
SWA1 = os:timestamp(),
TreeA0 = TreeAFolder0(),
io:format("Build tictac tree with 200K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWA1)]),
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
SegList1 = leveled_tictac:find_dirtyleaves(TreeA0, TreeB),
io:format("Tree comparison following delete shows ~w different leaves~n",
[length(SegList1)]),
true = length(SegList1) == 0,
corrected test differing object was in wrong bucket
2017-06-19 13:11:43 +01:00
% Removed test object so tictac trees should match
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
ok = testutil:book_riakput(Bookie3, TestObject, TestSpec),
{async, TreeBFolder0} = leveled_bookie:book_returnfolder(Bookie3, TicTacQ),
SWB1 = os:timestamp(),
TreeB0 = TreeBFolder0(),
io:format("Build tictac tree with 200K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWB1)]),
SegList2 = leveled_tictac:find_dirtyleaves(TreeA0, TreeB0),
true = SegList2 == SegList0,
% There is an identical difference now the difference is on Bookie3 not
% Bookie 2 (compared to it being in Bookie2 not Bookie3)
corrected test differing object was in wrong bucket
2017-06-19 13:11:43 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
ok = leveled_bookie:book_close(Bookie3),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% Replace Bookie 3 with two stores Bookie 4 and Bookie 5 where the ojects
% have been randomly split between the stores
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
StartOpts4 = [{root_path, RootPathC},
{max_journalsize, 200000000},
{max_pencillercachesize, 24000},
{sync_strategy, testutil:sync_strategy()}],
{ok, Bookie4} = leveled_bookie:book_start(StartOpts4),
StartOpts5 = [{root_path, RootPathD},
{max_journalsize, 200000000},
{max_pencillercachesize, 24000},
{sync_strategy, testutil:sync_strategy()}],
{ok, Bookie5} = leveled_bookie:book_start(StartOpts5),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
SplitFun =
fun(Obj) ->
case erlang:phash2(Obj) rem 2 of
0 ->
true;
1 ->
false
end
end,
lists:foreach(fun(ObjL) ->
{ObjLA, ObjLB} = lists:partition(SplitFun, ObjL),
testutil:riakload(Bookie4, ObjLA),
testutil:riakload(Bookie5, ObjLB)
end,
CLs),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
% query both the stores, then merge the trees - the result should be the
% same as the result from the tree created aginst the store with both
% partitions
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
{async, TreeC0Folder} = leveled_bookie:book_returnfolder(Bookie4, TicTacQ),
{async, TreeC1Folder} = leveled_bookie:book_returnfolder(Bookie5, TicTacQ),
SWD0 = os:timestamp(),
TreeC0 = TreeC0Folder(),
io:format("Build tictac tree with 100K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWD0)]),
SWD1 = os:timestamp(),
TreeC1 = TreeC1Folder(),
io:format("Build tictac tree with 100K objects in ~w~n",
[timer:now_diff(os:timestamp(), SWD1)]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
TreeC2 = leveled_tictac:merge_trees(TreeC0, TreeC1),
SegList3 = leveled_tictac:find_dirtyleaves(TreeC2, TreeB),
io:format("Tree comparison following delete shows ~w different leaves~n",
[length(SegList3)]),
true = length(SegList3) == 0,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
corrected test differing object was in wrong bucket
2017-06-19 13:11:43 +01:00
ok = leveled_bookie:book_close(Bookie2),
Expanded test ct testing of tictac trees now compares between differently partitioned stores.
2017-06-19 15:43:19 +01:00
ok = leveled_bookie:book_close(Bookie4),
ok = leveled_bookie:book_close(Bookie5).
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
index_compare(_Config) ->
Add smaller trees The "small" tree will serialise to 1.5MB - which seems large. Much smaller trees seem to be more suitable for things like recently modified aae indexes.
2017-09-25 13:07:08 +01:00
TreeSize = xxsmall,
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
LS = 2000,
JS = 50000000,
SS = testutil:sync_strategy(),
Add smaller trees The "small" tree will serialise to 1.5MB - which seems large. Much smaller trees seem to be more suitable for things like recently modified aae indexes.
2017-09-25 13:07:08 +01:00
SegmentCount = 64 * 64,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Test requires multiple different databases, so want to mount them all
% on individual file paths
RootPathA = testutil:reset_filestructure("testA"),
RootPathB = testutil:reset_filestructure("testB"),
RootPathC = testutil:reset_filestructure("testC"),
RootPathD = testutil:reset_filestructure("testD"),
% Book1A to get all objects
{ok, Book1A} = leveled_bookie:book_start(RootPathA, LS, JS, SS),
% Book1B/C/D will have objects partitioned across it
{ok, Book1B} = leveled_bookie:book_start(RootPathB, LS, JS, SS),
{ok, Book1C} = leveled_bookie:book_start(RootPathC, LS, JS, SS),
{ok, Book1D} = leveled_bookie:book_start(RootPathD, LS, JS, SS),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Generate nine lists of objects
BucketBin = list_to_binary("Bucket"),
GenMapFun =
fun(_X) ->
V = testutil:get_compressiblevalue(),
Indexes = testutil:get_randomindexes_generator(8),
testutil:generate_objects(10000, binary_uuid, [], V, Indexes)
end,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
ObjLists = lists:map(GenMapFun, lists:seq(1, 9)),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Load all nine lists into Book1A
lists:foreach(fun(ObjL) -> testutil:riakload(Book1A, ObjL) end,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
ObjLists),
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Split nine lists across Book1B to Book1D, three object lists in each
lists:foreach(fun(ObjL) -> testutil:riakload(Book1B, ObjL) end,
lists:sublist(ObjLists, 1, 3)),
lists:foreach(fun(ObjL) -> testutil:riakload(Book1C, ObjL) end,
lists:sublist(ObjLists, 4, 3)),
lists:foreach(fun(ObjL) -> testutil:riakload(Book1D, ObjL) end,
lists:sublist(ObjLists, 7, 3)),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
GetTicTacTreeFun =
fun(X, Bookie) ->
SW = os:timestamp(),
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
ST = <<"!">>,
ET = <<"|">>,
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
Q = {tictactree_idx,
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
{BucketBin,
list_to_binary("idx" ++ integer_to_list(X) ++ "_bin"),
ST,
ET},
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
TreeSize,
fun(_B, _K) -> accumulate end},
{async, Folder} = leveled_bookie:book_returnfolder(Bookie, Q),
R = Folder(),
io:format("TicTac Tree for index ~w took " ++
"~w microseconds~n",
[X, timer:now_diff(os:timestamp(), SW)]),
R
end,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Get a TicTac tree representing one of the indexes in Bucket A
TicTacTree1_Full = GetTicTacTreeFun(1, Book1A),
TicTacTree1_P1 = GetTicTacTreeFun(1, Book1B),
TicTacTree1_P2 = GetTicTacTreeFun(1, Book1C),
TicTacTree1_P3 = GetTicTacTreeFun(1, Book1D),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Merge the tree across the partitions
TicTacTree1_Joined = lists:foldl(fun leveled_tictac:merge_trees/2,
TicTacTree1_P1,
[TicTacTree1_P2, TicTacTree1_P3]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
% Go compare! Also check we're not comparing empty trees
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
DL1_0 = leveled_tictac:find_dirtyleaves(TicTacTree1_Full,
TicTacTree1_Joined),
EmptyTree = leveled_tictac:new_tree(empty, TreeSize),
DL1_1 = leveled_tictac:find_dirtyleaves(TicTacTree1_Full, EmptyTree),
true = DL1_0 == [],
true = length(DL1_1) > 100,
ok = leveled_bookie:book_close(Book1A),
ok = leveled_bookie:book_close(Book1B),
ok = leveled_bookie:book_close(Book1C),
ok = leveled_bookie:book_close(Book1D),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Double chekc all is well still after a restart
% Book1A to get all objects
{ok, Book2A} = leveled_bookie:book_start(RootPathA, LS, JS, SS),
% Book1B/C/D will have objects partitioned across it
{ok, Book2B} = leveled_bookie:book_start(RootPathB, LS, JS, SS),
{ok, Book2C} = leveled_bookie:book_start(RootPathC, LS, JS, SS),
{ok, Book2D} = leveled_bookie:book_start(RootPathD, LS, JS, SS),
% Get a TicTac tree representing one of the indexes in Bucket A
TicTacTree2_Full = GetTicTacTreeFun(2, Book2A),
TicTacTree2_P1 = GetTicTacTreeFun(2, Book2B),
TicTacTree2_P2 = GetTicTacTreeFun(2, Book2C),
TicTacTree2_P3 = GetTicTacTreeFun(2, Book2D),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
% Merge the tree across the partitions
TicTacTree2_Joined = lists:foldl(fun leveled_tictac:merge_trees/2,
TicTacTree2_P1,
[TicTacTree2_P2, TicTacTree2_P3]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
% Go compare! Also check we're not comparing empty trees
Add test of index comparison Compare two indexes for consistency
2017-06-22 13:54:51 +01:00
DL2_0 = leveled_tictac:find_dirtyleaves(TicTacTree2_Full,
TicTacTree2_Joined),
EmptyTree = leveled_tictac:new_tree(empty, TreeSize),
DL2_1 = leveled_tictac:find_dirtyleaves(TicTacTree2_Full, EmptyTree),
true = DL2_0 == [],
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
true = length(DL2_1) > 100,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
IdxSpc = {add, <<"idx2_bin">>, <<"zz999">>},
{TestObj, TestSpc} =
testutil:generate_testobject(
BucketBin,
term_to_binary("K9.Z"),
"Value1",
[IdxSpc],
[{"MDK1", "MDV1"}]),
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
ok = testutil:book_riakput(Book2C, TestObj, TestSpc),
testutil:check_forobject(Book2C, TestObj),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
TicTacTree3_Full = GetTicTacTreeFun(2, Book2A),
TicTacTree3_P1 = GetTicTacTreeFun(2, Book2B),
TicTacTree3_P2 = GetTicTacTreeFun(2, Book2C),
TicTacTree3_P3 = GetTicTacTreeFun(2, Book2D),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
% Merge the tree across the partitions
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
TicTacTree3_Joined =
lists:foldl(
fun leveled_tictac:merge_trees/2,
TicTacTree3_P1,
[TicTacTree3_P2, TicTacTree3_P3]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
% Find all keys index, and then just the last key
IdxQ1 = {index_query,
BucketBin,
{fun testutil:foldkeysfun/3, []},
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
{<<"idx2_bin">>, <<"zz">>, <<"zz|">>},
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
{true, undefined}},
{async, IdxFolder1} = leveled_bookie:book_returnfolder(Book2C, IdxQ1),
true = IdxFolder1() >= 1,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
DL_3to2B =
leveled_tictac:find_dirtyleaves(
TicTacTree2_P1, TicTacTree3_P1),
DL_3to2C =
leveled_tictac:find_dirtyleaves(
TicTacTree2_P2, TicTacTree3_P2),
DL_3to2D =
leveled_tictac:find_dirtyleaves(
TicTacTree2_P3, TicTacTree3_P3),
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
io:format("Individual tree comparison found dirty leaves of ~w ~w ~w~n",
[DL_3to2B, DL_3to2C, DL_3to2D]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
true = length(DL_3to2B) == 0,
true = length(DL_3to2C) == 1,
true = length(DL_3to2D) == 0,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Fix merge Fix typo in merge, and extra validation step to unit tests to prevent it returning.
2017-06-23 12:32:37 +01:00
% Go compare! Should find a difference in one leaf
DL3_0 = leveled_tictac:find_dirtyleaves(TicTacTree3_Full,
TicTacTree3_Joined),
io:format("Different leaves count ~w~n", [length(DL3_0)]),
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
true = length(DL3_0) == 1,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
% Now we want to find for the {Term, Key} pairs that make up the segment
% diferrence (there should only be one)
%
% We want the database to filter on segment - so this doesn't have the
% overheads of key listing
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
FoldKeysIndexQFun =
fun(_Bucket, {Term, Key}, Acc) ->
Attempt to standardise on segment hashes To allow for the segment has that accelerates queries to be re-used in tictac tree related queries.
2017-10-30 13:57:41 +00:00
Seg = get_segment(Key, SegmentCount),
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
case lists:member(Seg, DL3_0) of
true ->
[{Term, Key}|Acc];
false ->
Acc
end
end,
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
MismatchQ = {index_query,
BucketBin,
{FoldKeysIndexQFun, []},
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
{<<"idx2_bin">>, <<"!">>, <<"|">>},
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
{true, undefined}},
{async, MMFldr_2A} = leveled_bookie:book_returnfolder(Book2A, MismatchQ),
{async, MMFldr_2B} = leveled_bookie:book_returnfolder(Book2B, MismatchQ),
{async, MMFldr_2C} = leveled_bookie:book_returnfolder(Book2C, MismatchQ),
{async, MMFldr_2D} = leveled_bookie:book_returnfolder(Book2D, MismatchQ),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
SWSS = os:timestamp(),
SL_Joined = MMFldr_2B() ++ MMFldr_2C() ++ MMFldr_2D(),
SL_Full = MMFldr_2A(),
Broken test log
2017-06-23 15:20:24 +01:00
io:format("Segment search across both clusters took ~w~n",
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
[timer:now_diff(os:timestamp(), SWSS)]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
io:format("Joined SegList ~w~n", [SL_Joined]),
io:format("Full SegList ~w~n", [SL_Full]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
Diffs = lists:subtract(SL_Full, SL_Joined)
++ lists:subtract(SL_Joined, SL_Full),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
io:format("Differences between lists ~w~n", [Diffs]),
Add test with fold_heads Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak. In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend. The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
2017-08-05 16:43:03 +01:00
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
% The actual difference is discovered
Extend perf_SUITE (#434) * Extend perf_SUITE This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features. * Load chunk in spawned processes Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24. * Correctly account for pause alos try and improve test stability by increasing pause * Add microstate accounting to profile * Add memory tracking during test phases Identify and log out memory usage by test phase * Use macros instead (#437) * Don't print memory to screen in standard ct test --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
2024-07-15 20:49:21 +01:00
true = lists:member({<<"zz999">>, term_to_binary("K9.Z")}, Diffs),
Add test to find {term, key} that represents difference Not just detect existence of difference, but clarify what that difference that is.
2017-06-23 14:55:49 +01:00
% Without discovering too many others
Add close to test
2017-06-23 16:51:28 +01:00
true = length(Diffs) < 20,
ok = leveled_bookie:book_close(Book2A),
ok = leveled_bookie:book_close(Book2B),
ok = leveled_bookie:book_close(Book2C),
ok = leveled_bookie:book_close(Book2D).
Support for recent AAE index With basic ct test. Doesn't currently prove expiry of index. Doesn't prove ability to find segments. Assumes that either "all" buckets or a special list of buckets require indexing this way. Will lead to unexpected results if the same bucket name is used across different Tags. The format of the index has been chosen so that hopeully standard index features can be used (e.g. return_terms).
2017-06-30 16:31:22 +01:00
Check segment is as expected with tuplebuckets In head_only mode
2018-11-05 10:31:15 +00:00
tuplebuckets_headonly(_Config) ->
ObjectCount = 60000,
RootPathHO = testutil:reset_filestructure("testTBHO"),
StartOpts1 = [{root_path, RootPathHO},
{max_pencillercachesize, 16000},
{sync_strategy, none},
{head_only, with_lookup},
{max_journalsize, 500000}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
ObjectSpecFun =
fun(Op) ->
fun(N) ->
Bucket = {<<"BucketType">>, <<"B", 0:4/integer, N:4/integer>>},
Key = <<"K", N:32/integer>>,
<<Hash:32/integer, _RestBN/bitstring>> =
crypto:hash(md5, <<N:32/integer>>),
{Op, Bucket, Key, null, Hash}
end
end,
ObjectSpecL = lists:map(ObjectSpecFun(add), lists:seq(1, ObjectCount)),
SW0 = os:timestamp(),
ok = load_objectspecs(ObjectSpecL, 32, Bookie1),
io:format("Loaded an object count of ~w in ~w ms~n",
[ObjectCount, timer:now_diff(os:timestamp(), SW0)/1000]),
CheckHeadFun =
fun({add, B, K, null, H}) ->
{ok, H} =
leveled_bookie:book_headonly(Bookie1, B, K, null)
end,
lists:foreach(CheckHeadFun, ObjectSpecL),
BucketList =
lists:map(fun(I) ->
{<<"BucketType">>, <<"B", 0:4/integer, I:4/integer>>}
end,
lists:seq(0, 15)),
FoldHeadFun =
fun(B, {K, null}, V, Acc) ->
[{add, B, K, null, V}|Acc]
end,
SW1 = os:timestamp(),
{async, HeadRunner1} =
leveled_bookie:book_headfold(Bookie1,
?HEAD_TAG,
{bucket_list, BucketList},
{FoldHeadFun, []},
false, false,
false),
ReturnedObjSpecL1 = lists:reverse(HeadRunner1()),
[FirstItem|_Rest] = ReturnedObjSpecL1,
LastItem = lists:last(ReturnedObjSpecL1),
io:format("Returned ~w objects with first ~w and last ~w in ~w ms~n",
[length(ReturnedObjSpecL1),
FirstItem, LastItem,
timer:now_diff(os:timestamp(), SW1)/1000]),
true = ReturnedObjSpecL1 == lists:sort(ObjectSpecL),
{add, {TB, B1}, K1, null, _H1} = FirstItem,
{add, {TB, BL}, KL, null, _HL} = LastItem,
SegList = [testutil:get_aae_segment({TB, B1}, K1),
testutil:get_aae_segment({TB, BL}, KL)],
SW2 = os:timestamp(),
{async, HeadRunner2} =
leveled_bookie:book_headfold(Bookie1,
?HEAD_TAG,
{bucket_list, BucketList},
{FoldHeadFun, []},
false, false,
SegList),
ReturnedObjSpecL2 = lists:reverse(HeadRunner2()),
io:format("Returned ~w objects using seglist in ~w ms~n",
[length(ReturnedObjSpecL2),
timer:now_diff(os:timestamp(), SW2)/1000]),
true = length(ReturnedObjSpecL2) < (ObjectCount/1000 + 2),
% Not too many false positives
true = lists:member(FirstItem, ReturnedObjSpecL2),
true = lists:member(LastItem, ReturnedObjSpecL2),
leveled_bookie:book_destroy(Bookie1).
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
basic_headonly(_Config) ->
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
ObjectCount = 200000,
Coverage issues Not making proxy object so get_size not required. Extend tests to improve coverage
2018-02-16 20:27:49 +00:00
RemoveCount = 100,
basic_headonly_test(ObjectCount, RemoveCount, with_lookup),
basic_headonly_test(ObjectCount, RemoveCount, no_lookup).
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
Coverage issues Not making proxy object so get_size not required. Extend tests to improve coverage
2018-02-16 20:27:49 +00:00
basic_headonly_test(ObjectCount, RemoveCount, HeadOnly) ->
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
% Load some AAE type objects into Leveled using the read_only mode. This
% should allow for the items to be added in batches. Confirm that the
% journal is garbage collected as expected, and that it is possible to
% perform a fold_heads style query
RootPathHO = testutil:reset_filestructure("testHO"),
StartOpts1 = [{root_path, RootPathHO},
{max_pencillercachesize, 16000},
{sync_strategy, sync},
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
{head_only, HeadOnly},
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
{max_journalsize, 500000}],
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{B1, K1, V1, S1, MD} = {"Bucket",
"Key1.1.4567.4321",
"Value1",
[],
[{"MDK1", "MDV1"}]},
{TestObject, TestSpec} = testutil:generate_testobject(B1, K1, V1, S1, MD),
{unsupported_message, put} =
testutil:book_riakput(Bookie1, TestObject, TestSpec),
ObjectSpecFun =
fun(Op) ->
fun(N) ->
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
Bucket = <<"B", N:8/integer>>,
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
Key = <<"K", N:32/integer>>,
<<SegmentID:20/integer, _RestBS/bitstring>> =
crypto:hash(md5, term_to_binary({Bucket, Key})),
<<Hash:32/integer, _RestBN/bitstring>> =
crypto:hash(md5, <<N:32/integer>>),
{Op, <<SegmentID:32/integer>>, Bucket, Key, Hash}
end
end,
ObjectSpecL = lists:map(ObjectSpecFun(add), lists:seq(1, ObjectCount)),
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
SW0 = os:timestamp(),
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
ok = load_objectspecs(ObjectSpecL, 32, Bookie1),
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
io:format("Loaded an object count of ~w in ~w microseconds with ~w~n",
[ObjectCount, timer:now_diff(os:timestamp(), SW0), HeadOnly]),
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
FoldFun =
fun(_B, _K, V, {HashAcc, CountAcc}) ->
{HashAcc bxor V, CountAcc + 1}
end,
InitAcc = {0, 0},
RunnerDefinition =
Allow for all keys head folds to used modifed range This helps with kv_index_tictcatree with the leveled_so backend. Now this cna do folds over ranges of keys with modified filters (as folds over ranges of keys must go over lal keys if the backend is segment_ordered)
2018-11-01 17:30:18 +00:00
{foldheads_allkeys, h, {FoldFun, InitAcc},
false, false, false, false, false},
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
{async, Runner1} =
leveled_bookie:book_returnfolder(Bookie1, RunnerDefinition),
SW1 = os:timestamp(),
{AccH1, AccC1} = Runner1(),
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
io:format("AccH and AccC of ~w ~w in ~w microseconds~n",
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
[AccH1, AccC1, timer:now_diff(os:timestamp(), SW1)]),
true = AccC1 == ObjectCount,
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
JFP = RootPathHO ++ "/journal/journal_files",
{ok, FNs} = file:list_dir(JFP),
ok = leveled_bookie:book_trimjournal(Bookie1),
WaitForTrimFun =
fun(N, _Acc) ->
{ok, PollFNs} = file:list_dir(JFP),
case length(PollFNs) < length(FNs) of
true ->
true;
false ->
timer:sleep(N * 1000),
false
end
end,
true = lists:foldl(WaitForTrimFun, false, [1, 2, 3, 5, 8, 13]),
{ok, FinalFNs} = file:list_dir(JFP),
Change to cast in inker/iclerk interaction This allows for leveled_iclerk:clerk_stop to be a sync call, so that files will only be closed once the iclerk has stopped. This is designed ot prevent iclerk crashes during shutdowns when files it is depnding on are closed mid shutdown.
2019-01-24 21:32:54 +00:00
ok = leveled_bookie:book_trimjournal(Bookie1),
% CCheck a second trim is still OK
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
[{add, SegmentID0, Bucket0, Key0, Hash0}|_Rest] = ObjectSpecL,
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
case HeadOnly of
with_lookup ->
% If we allow HEAD_TAG to be suubject to a lookup, then test this
% here
{ok, Hash0} =
Add new book_headonly/4 API To address special situation of performing a head requets in head_only mode - where a sub-key is a required input.
2018-09-20 12:08:33 +01:00
leveled_bookie:book_headonly(Bookie1,
SegmentID0,
Bucket0,
Key0),
Add support for with_lookup head_only head_only mode cna be run with_lookup - but there is no L0 index created in this case. So the L0 index wasn't returning a potition list and the L0 cache wasn't being checked. Code now checks every position in the L0 cache, when a lookup is attempted in head_only mode.
2018-06-23 15:15:49 +01:00
CheckHeadFun =
Mas d31 i413 (#415) * Allow snapshots to be reused in queries Allow for a full bookie snapshot to be re-used for multiple queries, not just KV fetches. * Reduce log noise The internal dummy tag is expected so should not prompt a log on reload * Snapshot should have same status of active db wrt head_only and head_lookup * Allow logging to specified on snapshots * Shutdown snapshot bookie is primary goes down Inker and Penciller already will shut down based on `erlang:monitor/2` * Review feedback Formatting and code readability fixes
2023-11-08 09:18:01 +00:00
fun(DB) ->
fun({add, SegID, B, K, H}) ->
{ok, H} =
leveled_bookie:book_headonly(DB, SegID, B, K)
end
Add support for with_lookup head_only head_only mode cna be run with_lookup - but there is no L0 index created in this case. So the L0 index wasn't returning a potition list and the L0 cache wasn't being checked. Code now checks every position in the L0 cache, when a lookup is attempted in head_only mode.
2018-06-23 15:15:49 +01:00
end,
Mas d31 i413 (#415) * Allow snapshots to be reused in queries Allow for a full bookie snapshot to be re-used for multiple queries, not just KV fetches. * Reduce log noise The internal dummy tag is expected so should not prompt a log on reload * Snapshot should have same status of active db wrt head_only and head_lookup * Allow logging to specified on snapshots * Shutdown snapshot bookie is primary goes down Inker and Penciller already will shut down based on `erlang:monitor/2` * Review feedback Formatting and code readability fixes
2023-11-08 09:18:01 +00:00
lists:foreach(CheckHeadFun(Bookie1), ObjectSpecL),
{ok, Snapshot} =
leveled_bookie:book_start([{snapshot_bookie, Bookie1}]),
ok = leveled_bookie:book_loglevel(Snapshot, warn),
ok =
leveled_bookie:book_addlogs(
Snapshot, [b0001, b0002, b0003, i0027, p0007]
),
ok =
leveled_bookie:book_removelogs(
Snapshot, [b0019]
),
io:format(
"Checking for ~w objects against Snapshot ~w~n",
[length(ObjectSpecL), Snapshot]),
lists:foreach(CheckHeadFun(Snapshot), ObjectSpecL),
io:format("Closing snapshot ~w~n", [Snapshot]),
ok = leveled_bookie:book_close(Snapshot),
{ok, AltSnapshot} =
leveled_bookie:book_start([{snapshot_bookie, Bookie1}]),
ok =
leveled_bookie:book_addlogs(
AltSnapshot, [b0001, b0002, b0003, b0004, i0027, p0007]
),
true = is_process_alive(AltSnapshot),
io:format(
"Closing actual store ~w with snapshot ~w open~n",
[Bookie1, AltSnapshot]
),
ok = leveled_bookie:book_close(Bookie1),
% Sleep a beat so as not to race with the 'DOWN' message
timer:sleep(10),
false = is_process_alive(AltSnapshot);
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
no_lookup ->
{unsupported_message, head} =
leveled_bookie:book_head(Bookie1,
SegmentID0,
{Bucket0, Key0},
Add new book_headonly/4 API To address special situation of performing a head requets in head_only mode - where a sub-key is a required input.
2018-09-20 12:08:33 +01:00
h),
{unsupported_message, head} =
leveled_bookie:book_headonly(Bookie1,
SegmentID0,
Bucket0,
Mas d31 i413 (#415) * Allow snapshots to be reused in queries Allow for a full bookie snapshot to be re-used for multiple queries, not just KV fetches. * Reduce log noise The internal dummy tag is expected so should not prompt a log on reload * Snapshot should have same status of active db wrt head_only and head_lookup * Allow logging to specified on snapshots * Shutdown snapshot bookie is primary goes down Inker and Penciller already will shut down based on `erlang:monitor/2` * Review feedback Formatting and code readability fixes
2023-11-08 09:18:01 +00:00
Key0),
io:format("Closing actual store ~w~n", [Bookie1]),
ok = leveled_bookie:book_close(Bookie1)
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
end,
Mas d31 i413 (#415) * Allow snapshots to be reused in queries Allow for a full bookie snapshot to be re-used for multiple queries, not just KV fetches. * Reduce log noise The internal dummy tag is expected so should not prompt a log on reload * Snapshot should have same status of active db wrt head_only and head_lookup * Allow logging to specified on snapshots * Shutdown snapshot bookie is primary goes down Inker and Penciller already will shut down based on `erlang:monitor/2` * Review feedback Formatting and code readability fixes
2023-11-08 09:18:01 +00:00
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
{ok, FinalJournals} = file:list_dir(JFP),
io:format("Trim has reduced journal count from " ++
"~w to ~w and ~w after restart~n",
[length(FNs), length(FinalFNs), length(FinalJournals)]),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
{async, Runner2} =
leveled_bookie:book_returnfolder(Bookie2, RunnerDefinition),
Coverage issues Not making proxy object so get_size not required. Extend tests to improve coverage
2018-02-16 20:27:49 +00:00
{AccH2, AccC2} = Runner2(),
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
true = AccC2 == ObjectCount,
Optional lookup in head_only mode Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
2018-02-16 17:06:30 +00:00
case HeadOnly of
with_lookup ->
% If we allow HEAD_TAG to be suubject to a lookup, then test this
% here
{ok, Hash0} =
leveled_bookie:book_head(Bookie2,
SegmentID0,
{Bucket0, Key0},
h);
no_lookup ->
{unsupported_message, head} =
leveled_bookie:book_head(Bookie2,
SegmentID0,
{Bucket0, Key0},
h)
end,
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
Coverage issues Not making proxy object so get_size not required. Extend tests to improve coverage
2018-02-16 20:27:49 +00:00
RemoveSpecL0 = lists:sublist(ObjectSpecL, RemoveCount),
RemoveSpecL1 =
lists:map(fun(Spec) -> setelement(1, Spec, remove) end, RemoveSpecL0),
ok = load_objectspecs(RemoveSpecL1, 32, Bookie2),
{async, Runner3} =
leveled_bookie:book_returnfolder(Bookie2, RunnerDefinition),
{AccH3, AccC3} = Runner3(),
true = AccC3 == (ObjectCount - RemoveCount),
false = AccH3 == AccH2,
Add lookup support in head_only mode Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook. Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting). So reverted to re-adding support for HEAD requests with these keys.
2018-02-16 14:16:28 +00:00
ok = leveled_bookie:book_close(Bookie2).
Initial head_only features Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger). The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
2018-02-15 16:14:46 +00:00
load_objectspecs([], _SliceSize, _Bookie) ->
ok;
load_objectspecs(ObjectSpecL, SliceSize, Bookie)
when length(ObjectSpecL) < SliceSize ->
load_objectspecs(ObjectSpecL, length(ObjectSpecL), Bookie);
load_objectspecs(ObjectSpecL, SliceSize, Bookie) ->
{Head, Tail} = lists:split(SliceSize, ObjectSpecL),
case leveled_bookie:book_mput(Bookie, Head) of
ok ->
load_objectspecs(Tail, SliceSize, Bookie);
pause ->
timer:sleep(10),
load_objectspecs(Tail, SliceSize, Bookie)
end.
Attempt to standardise on segment hashes To allow for the segment has that accelerates queries to be re-used in tictac tree related queries.
2017-10-30 13:57:41 +00:00
get_segment(K, SegmentCount) ->
BinKey =
case is_binary(K) of
true ->
K;
false ->
term_to_binary(K)
end,
{SegmentID, ExtraHash} = leveled_codec:segment_hash(BinKey),
SegHash = (ExtraHash band 65535) bsl 16 + SegmentID,
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
leveled_tictac:get_segment(SegHash, SegmentCount).
Reference in a new issue Copy permalink