umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
leveled/test/end_to_end/iterator_SUITE.erl

961 lines
41 KiB
Erlang
Raw Normal View History

Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
-module(iterator_SUITE).
-include_lib("common_test/include/ct.hrl").
-include("include/leveled.hrl").
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
-define(KEY_ONLY, {false, undefined}).
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
-export([all/0]).
Add test of index expiry To show how this works, and prove that it does work thta way. Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
-export([expiring_indexes/1,
breaking_folds/1,
Add capability to exit a head or object fold with a throw This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected. This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception). This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
2018-11-23 16:00:11 +00:00
single_object_with2i/1,
Simple test To try and pinpoint any issue with _int index (as seen in Riak integrtaion testing)
2016-12-02 17:39:11 +00:00
small_load_with2i/1,
Experiment with new cache size algorithm Remove the jitter probability and make it a smooth function heading towards the max ache size
2016-11-03 09:19:02 +00:00
query_count/1,
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
multibucket_fold/1,
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
foldobjects_bybucket_range/1,
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
rotating_objects/1]).
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
Recovery strategy testing Test added for the "retain" recovery strategy. This strategy makes sure a full history of index changes is made so that if the Ledger is wiped out, the Ledger cna be fully rebuilt from the Journal. This exposed two journal compaction problems - The BestRun selected did not have the source files correctly sorted in order before compaction - The compaction process incorrectly dealt with the KeyDelta object left after a compaction - i.e. compacting twice the same key caused that key history to be lost. These issues have now been corrected.
2016-10-27 00:57:19 +01:00
all() -> [
Add test of index expiry To show how this works, and prove that it does work thta way. Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
expiring_indexes,
Add capability to exit a head or object fold with a throw This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected. This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception). This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
2018-11-23 16:00:11 +00:00
breaking_folds,
Simple test To try and pinpoint any issue with _int index (as seen in Riak integrtaion testing)
2016-12-02 17:39:11 +00:00
single_object_with2i,
Uncomment test
2016-11-18 16:34:16 +00:00
small_load_with2i,
query_count,
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
multibucket_fold,
Actually run the new test DERP!
2018-09-06 16:38:49 +01:00
rotating_objects,
foldobjects_bybucket_range
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
].
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
Add test of index expiry To show how this works, and prove that it does work thta way. Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
expiring_indexes(_Config) ->
% Add objects to ths tore with index entries, where the objects (and hence
% the indexes have an expiry time. Confirm that the indexes and the
% objects are no longer present after the expiry time (and are present
% before). Confirm that replacing an object has the expected outcome, if
% the IndexSpecs are updated as part of the request.
KeyCount = 50000,
Future = 120,
% 2 minutes - if running tests on a slow machine, may need to increase
% this value
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 100000000},
{sync_strategy, testutil:sync_strategy()}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
SW1 = os:timestamp(),
IBKL1 = testutil:stdload_expiring(Bookie1, KeyCount, Future),
LoadTime = timer:now_diff(os:timestamp(), SW1)/1000000,
io:format("Load of ~w std objects in ~w seconds~n", [KeyCount, LoadTime]),
FilterFun = fun({I, _B, _K}) -> lists:member(I, [5, 6, 7, 8]) end,
LoadedEntriesInRange = lists:sort(lists:filter(FilterFun, IBKL1)),
true = LoadTime < (Future - 30),
% need 30 seconds spare to run query
% and add replacements
Add check that index disappears Check I0 count goes down when that index is removed
2018-12-05 15:42:21 +00:00
{I0, B0, K0} = hd(IBKL1),
Add test of index expiry To show how this works, and prove that it does work thta way. Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
false = FilterFun(hd(IBKL1)),
% The head entry should not have index between 5 and 8
Add check that index disappears Check I0 count goes down when that index is removed
2018-12-05 15:42:21 +00:00
CountI0Fold =
fun() ->
leveled_bookie:book_indexfold(Bookie1,
B0,
{fun(_BF, _KT, Acc) -> Acc + 1 end,
0},
{<<"temp_int">>, I0, I0},
{true, undefined})
end,
{async, I0Counter1} = CountI0Fold(),
I0Count1 = I0Counter1(),
Add test of index expiry To show how this works, and prove that it does work thta way. Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
FoldFun = fun(BF, {IdxV, KeyF}, Acc) -> [{IdxV, BF, KeyF}|Acc] end,
InitAcc = [],
IndexFold =
fun() ->
leveled_bookie:book_indexfold(Bookie1,
B0,
{FoldFun, InitAcc},
{<<"temp_int">>, 5, 8},
{true, undefined})
end,
{async, Folder1} = IndexFold(),
QR1 = Folder1(),
true = lists:sort(QR1) == LoadedEntriesInRange,
% Replace object with one with an index value of 6
testutil:stdload_object(Bookie1, B0, K0, 6, <<"value">>,
leveled_util:integer_now() + 600),
Add check that index disappears Check I0 count goes down when that index is removed
2018-12-05 15:42:21 +00:00
% Confirm that this has reduced the index entries in I0 by 1
{async, I0Counter2} = CountI0Fold(),
I0Count2 = I0Counter2(),
io:format("Count with index value ~w changed from ~w to ~w~n",
[I0, I0Count1, I0Count2]),
true = I0Count2 == (I0Count1 - 1),
Add test of index expiry To show how this works, and prove that it does work thta way. Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
% Now replace again, shortening the timeout to 15s,
% this time index value of 6
testutil:stdload_object(Bookie1, B0, K0, 5, <<"value">>,
leveled_util:integer_now() + 15),
{async, Folder2} = IndexFold(),
leveled_bookie:book_indexfold(Bookie1,
B0,
{FoldFun, InitAcc},
{<<"temp_int">>, 5, 8},
{true, undefined}),
QR2 = Folder2(),
io:format("Query with additional entry length ~w~n", [length(QR2)]),
true = lists:sort(QR2) == lists:sort([{5, B0, K0}|LoadedEntriesInRange]),
% Wait for a 15s timeout + lus a second to be sure
timer:sleep(15000 + 1000),
{async, Folder3} = IndexFold(),
QR3 = Folder3(),
% Now the entry should be missing (and the 600s TTL entry should not have
% resurfaced)
io:format("Query results length ~w following sleep~n", [length(QR3)]),
true = lists:sort(QR3) == LoadedEntriesInRange,
FoldTime = timer:now_diff(os:timestamp(), SW1)/1000000 - LoadTime,
io:format("Query returned ~w entries in ~w seconds - 3 queries + 15s wait~n",
[length(QR1), FoldTime]),
true = (LoadTime + FoldTime) < Future,
SleepTime = round((Future - (LoadTime + FoldTime)) * 1000 + 1000), % add a second
io:format("Sleeping ~w s for all to expire~n", [SleepTime/1000]),
timer:sleep(SleepTime),
% Index entries should now have expired
{async, Folder4} = IndexFold(),
QR4 = Folder4(),
true = QR4 == [],
ok = leveled_bookie:book_close(Bookie1),
testutil:reset_filestructure().
Add capability to exit a head or object fold with a throw This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected. This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception). This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
2018-11-23 16:00:11 +00:00
breaking_folds(_Config) ->
% Run various iterators and show that they can be broken by throwing an
% exception from within the fold
KeyCount = 10000,
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 10000000},
{sync_strategy, testutil:sync_strategy()}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = testutil:get_randomindexes_generator(8),
ObjL1 = testutil:generate_objects(KeyCount,
binary_uuid,
[],
ObjectGen,
IndexGen),
testutil:riakload(Bookie1, ObjL1),
% Find all keys index, and then same again but stop at a midpoint using a
% throw
{async, IdxFolder} =
leveled_bookie:book_indexfold(Bookie1,
list_to_binary("Bucket"),
{fun testutil:foldkeysfun/3, []},
{"idx1_bin", "#", "|"},
{true, undefined}),
KeyList1 = lists:reverse(IdxFolder()),
io:format("Index fold with result size ~w~n", [length(KeyList1)]),
true = KeyCount == length(KeyList1),
{MidTerm, MidKey} = lists:nth(KeyCount div 2, KeyList1),
FoldKeyThrowFun =
fun(_B, {Term, Key}, Acc) ->
case {Term, Key} > {MidTerm, MidKey} of
true ->
throw({stop_fold, Acc});
false ->
[{Term, Key}|Acc]
end
end,
{async, IdxFolderToMidK} =
leveled_bookie:book_indexfold(Bookie1,
list_to_binary("Bucket"),
{FoldKeyThrowFun, []},
{"idx1_bin", "#", "|"},
{true, undefined}),
CatchingFold =
fun(AsyncFolder) ->
try
AsyncFolder()
catch
throw:{stop_fold, Acc} ->
Acc
end
end,
KeyList2 = lists:reverse(CatchingFold(IdxFolderToMidK)),
io:format("Index fold with result size ~w~n", [length(KeyList2)]),
true = KeyCount div 2 == length(KeyList2),
HeadFoldFun =
fun(_B, K, PO, Acc) ->
{proxy_object, _MDBin, Size, _FF} = binary_to_term(PO),
[{K, Size}|Acc]
end,
{async, HeadFolder} =
leveled_bookie:book_headfold(Bookie1,
?RIAK_TAG,
{HeadFoldFun, []},
true, true, false),
KeySizeList1 = lists:reverse(HeadFolder()),
io:format("Head fold with result size ~w~n", [length(KeySizeList1)]),
true = KeyCount == length(KeySizeList1),
{MidHeadKey, _MidSize} = lists:nth(KeyCount div 2, KeySizeList1),
FoldThrowFun =
fun(FoldFun) ->
fun(B, K, PO, Acc) ->
case K > MidHeadKey of
true ->
throw({stop_fold, Acc});
false ->
FoldFun(B, K, PO, Acc)
end
end
end,
{async, HeadFolderToMidK} =
leveled_bookie:book_headfold(Bookie1,
?RIAK_TAG,
{FoldThrowFun(HeadFoldFun), []},
true, true, false),
KeySizeList2 = lists:reverse(CatchingFold(HeadFolderToMidK)),
io:format("Head fold with result size ~w~n", [length(KeySizeList2)]),
true = KeyCount div 2 == length(KeySizeList2),
ObjFoldFun =
fun(_B, K, V, Acc) ->
[{K,byte_size(V)}|Acc]
end,
{async, ObjectFolderKO} =
leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
{ObjFoldFun, []},
false,
key_order),
ObjSizeList1 = lists:reverse(ObjectFolderKO()),
io:format("Obj fold with result size ~w~n", [length(ObjSizeList1)]),
true = KeyCount == length(ObjSizeList1),
{async, ObjFolderToMidK} =
leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
{FoldThrowFun(ObjFoldFun), []},
false,
key_order),
ObjSizeList2 = lists:reverse(CatchingFold(ObjFolderToMidK)),
io:format("Object fold with result size ~w~n", [length(ObjSizeList2)]),
true = KeyCount div 2 == length(ObjSizeList2),
% Object folds which are SQN order use a different path through the code,
% so testing that here. Note that it would not make sense to have a fold
% that was terminated by reaching a point in the key range .. as results
% will not be passed to the fold function in key order
{async, ObjectFolderSO} =
leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
{ObjFoldFun, []},
false,
sqn_order),
ObjSizeList1_SO = lists:reverse(ObjectFolderSO()),
io:format("Obj fold with result size ~w~n", [length(ObjSizeList1_SO)]),
true = KeyCount == length(ObjSizeList1_SO),
% Exit fold when we've reached a thousand accumulated obects
FoldThrowThousandFun =
fun(FoldFun) ->
fun(B, K, PO, Acc) ->
case length(Acc) == 1000 of
true ->
throw({stop_fold, Acc});
false ->
FoldFun(B, K, PO, Acc)
end
end
end,
{async, ObjFolderTo1K} =
leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
{FoldThrowThousandFun(ObjFoldFun), []},
false,
sqn_order),
ObjSizeList2_SO = lists:reverse(CatchingFold(ObjFolderTo1K)),
io:format("Object fold with result size ~w~n", [length(ObjSizeList2_SO)]),
Bucket list update the docs to advertise throw capability. Test it for bucket list (and fix ordering of bucket lists)
2018-11-23 18:56:30 +00:00
true = 1000 == length(ObjSizeList2_SO),
ObjL2 = testutil:generate_objects(10,
binary_uuid,
[],
ObjectGen,
IndexGen,
"B2"),
ObjL3 = testutil:generate_objects(10,
binary_uuid,
[],
ObjectGen,
IndexGen,
"B3"),
ObjL4 = testutil:generate_objects(10,
binary_uuid,
[],
ObjectGen,
IndexGen,
"B4"),
testutil:riakload(Bookie1, ObjL2),
testutil:riakload(Bookie1, ObjL3),
testutil:riakload(Bookie1, ObjL4),
FBAccT = {fun(B, Acc) -> [B|Acc] end, []},
{async, BucketFolder} =
leveled_bookie:book_bucketlist(Bookie1, ?RIAK_TAG, FBAccT, all),
BucketList1 = lists:reverse(BucketFolder()),
io:format("bucket list with result size ~w~n", [length(BucketList1)]),
true = 4 == length(BucketList1),
StopAt3Fun =
fun(B, Acc) ->
Acc0 = [B|Acc],
case B of
<<"B3">> ->
throw({stop_fold, Acc0});
_ ->
Acc0
end
end,
{async, StopAt3BucketFolder} =
leveled_bookie:book_bucketlist(Bookie1,
?RIAK_TAG,
{StopAt3Fun, []},
all),
BucketListSA3 = lists:reverse(CatchingFold(StopAt3BucketFolder)),
io:format("bucket list with result ~w~n", [BucketListSA3]),
true = [<<"B2">>, <<"B3">>] == BucketListSA3,
Add capability to exit a head or object fold with a throw This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected. This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception). This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
2018-11-23 16:00:11 +00:00
ok = leveled_bookie:book_close(Bookie1),
testutil:reset_filestructure().
Simple test To try and pinpoint any issue with _int index (as seen in Riak integrtaion testing)
2016-12-02 17:39:11 +00:00
single_object_with2i(_Config) ->
% Load a single object with an integer and a binary
% index and query for it
RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath},
{max_journalsize, 5000000},
{sync_strategy, testutil:sync_strategy()}],
% low journal size to make sure > 1 created
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{TestObject, _TestSpec} = testutil:generate_testobject(),
TestSpec = [{add, list_to_binary("integer_int"), 100},
{add, list_to_binary("binary_bin"), <<100:32/integer>>}],
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
%% @TODO replace all index queries with new Top-Level API if tests
%% pass
{async, IdxFolder1} = leveled_bookie:book_indexfold(Bookie1,
"Bucket1",
{fun testutil:foldkeysfun/3, []},
{list_to_binary("binary_bin"),
<<99:32/integer>>, <<101:32/integer>>},
{true, undefined}),
Simple test To try and pinpoint any issue with _int index (as seen in Riak integrtaion testing)
2016-12-02 17:39:11 +00:00
R1 = IdxFolder1(),
io:format("R1 of ~w~n", [R1]),
true = [{<<100:32/integer>>,"Key1"}] == R1,
IdxQ2 = {index_query,
"Bucket1",
{fun testutil:foldkeysfun/3, []},
{list_to_binary("integer_int"),
99, 101},
{true, undefined}},
{async, IdxFolder2} = leveled_bookie:book_returnfolder(Bookie1, IdxQ2),
R2 = IdxFolder2(),
io:format("R2 of ~w~n", [R2]),
true = [{100,"Key1"}] == R2,
IdxQ3 = {index_query,
{"Bucket1", "Key1"},
{fun testutil:foldkeysfun/3, []},
{list_to_binary("integer_int"),
99, 101},
{true, undefined}},
{async, IdxFolder3} = leveled_bookie:book_returnfolder(Bookie1, IdxQ3),
R3 = IdxFolder3(),
io:format("R2 of ~w~n", [R3]),
true = [{100,"Key1"}] == R3,
ok = leveled_bookie:book_close(Bookie1),
testutil:reset_filestructure().
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
small_load_with2i(_Config) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Startup work - baffled Changes the stratup otpions to a prolist to make it easier to get environment variables as default. Tried application:start - and completely baffled as to how to get this to work.
2016-11-02 12:58:27 +00:00
StartOpts1 = [{root_path, RootPath},
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{max_journalsize, 5000000},
{sync_strategy, testutil:sync_strategy()}],
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
% low journal size to make sure > 1 created
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{TestObject, TestSpec} = testutil:generate_testobject(),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Bookie1, TestObject),
testutil:check_formissingobject(Bookie1, "Bucket1", "Key2"),
testutil:check_forobject(Bookie1, TestObject),
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = testutil:get_randomindexes_generator(8),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ObjL1 = testutil:generate_objects(10000,
uuid,
[],
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
ObjectGen,
IndexGen),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:riakload(Bookie1, ObjL1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ChkList1 = lists:sublist(lists:sort(ObjL1), 100),
testutil:check_forlist(Bookie1, ChkList1),
testutil:check_forobject(Bookie1, TestObject),
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
Refine query to accept fold functions Need to be able to pass external fold functions into different queries, to work as a Riak backend
2016-11-18 15:53:22 +00:00
% Find all keys index, and then just the last key
IdxQ1 = {index_query,
"Bucket",
{fun testutil:foldkeysfun/3, []},
Change tests - binary keys the new code requires bucket listing to be on binary keys not just binary buckets. As this is only intended for use within Riak (where all keys are buckets are binaries), this constraint seems OK. A test needed changing to ensure it had a binary key in the bucket.
2017-05-23 15:54:11 +01:00
{"idx1_bin", "#", "|"},
Refine query to accept fold functions Need to be able to pass external fold functions into different queries, to work as a Riak backend
2016-11-18 15:53:22 +00:00
{true, undefined}},
{async, IdxFolder} = leveled_bookie:book_returnfolder(Bookie1, IdxQ1),
KeyList1 = lists:usort(IdxFolder()),
true = 10000 == length(KeyList1),
{LastTerm, LastKey} = lists:last(KeyList1),
IdxQ2 = {index_query,
{"Bucket", LastKey},
{fun testutil:foldkeysfun/3, []},
Change tests - binary keys the new code requires bucket listing to be on binary keys not just binary buckets. As this is only intended for use within Riak (where all keys are buckets are binaries), this constraint seems OK. A test needed changing to ensure it had a binary key in the bucket.
2017-05-23 15:54:11 +01:00
{"idx1_bin", LastTerm, "|"},
Refine query to accept fold functions Need to be able to pass external fold functions into different queries, to work as a Riak backend
2016-11-18 15:53:22 +00:00
{false, undefined}},
{async, IdxFolderLK} = leveled_bookie:book_returnfolder(Bookie1, IdxQ2),
KeyList2 = lists:usort(IdxFolderLK()),
io:format("List should be last key ~w ~w~n", [LastKey, KeyList2]),
true = 1 == length(KeyList2),
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
%% Delete the objects from the ChkList removing the indexes
lists:foreach(fun({_RN, Obj, Spc}) ->
DSpc = lists:map(fun({add, F, T}) -> {remove, F, T}
end,
Spc),
Switch to binary format Riak object Initial change to try and test assuming that leveled received the binary format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
{B, K} = {Obj#r_object.bucket, Obj#r_object.key},
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
testutil:book_riakdelete(Bookie1, B, K, DSpc)
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
end,
ChkList1),
%% Get the Buckets Keys and Hashes for the whole bucket
Switch to binary format Riak object Initial change to try and test assuming that leveled received the binary format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
FoldObjectsFun = fun(B, K, V, Acc) -> [{B, K, erlang:phash2(V)}|Acc]
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
end,
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, HTreeF1} = leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
{FoldObjectsFun, []},
false),
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
KeyHashList1 = HTreeF1(),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, HTreeF2} = leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
"Bucket",
all,
{FoldObjectsFun, []},
false),
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
KeyHashList2 = HTreeF2(),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, HTreeF3} = leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
"Bucket",
{"idx1_bin", "#", "|"},
{FoldObjectsFun, []},
false),
Expand fold objects support Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
KeyHashList3 = HTreeF3(),
true = 9901 == length(KeyHashList1), % also includes the test object
true = 9900 == length(KeyHashList2),
true = 9900 == length(KeyHashList3),
Switch to binary format Riak object Initial change to try and test assuming that leveled received the binary format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
SumIntFun = fun(_B, _K, Obj, Acc) ->
{I, _Bin} = testutil:get_value(Obj),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
Acc + I
end,
Split code for folders - introduce runner actor Introduce a dedicated module for all the different fold types. Also simplify the list of folders by deprecating those folds that should eb achieveable by fold_heads/fold_objects type folds but with smarter functions. Makes sure that the fold functiosn also have better spec coverage, and are dialyzer checked.
2017-10-17 20:39:11 +01:00
BucketObjQ =
{foldobjects_bybucket, ?RIAK_TAG, "Bucket", all, {SumIntFun, 0}, true},
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
{async, Sum1} = leveled_bookie:book_returnfolder(Bookie1, BucketObjQ),
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
Total1 = Sum1(),
Test mistake If random integer was low, total could be below threshold - so calculate total correctly. Should make value re-generate random uniform, but test is still valid without this
2018-09-25 18:32:48 +01:00
io:format("Total from summing all I is ~w~n", [Total1]),
SumFromObjLFun =
fun(Obj, Acc) ->
{I, _Bin} = testutil:get_value_from_objectlistitem(Obj),
Acc + I
end,
ObjL1Total =
lists:foldl(SumFromObjLFun, 0, ObjL1),
ChkList1Total =
lists:foldl(SumFromObjLFun, 0, ChkList1),
io:format("Total in original object list ~w and from removed list ~w~n",
[ObjL1Total, ChkList1Total]),
Total1 = ObjL1Total - ChkList1Total,
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ok = leveled_bookie:book_close(Bookie1),
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
{async, Sum2} = leveled_bookie:book_returnfolder(Bookie2, BucketObjQ),
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
Total2 = Sum2(),
true = Total2 == Total1,
Change FoldBucketsFun to take just bucket FoldBucketsFun does not take keys should be a 2-arity function (Bucket, Acc).
2016-11-21 14:12:17 +00:00
FoldBucketsFun = fun(B, Acc) -> sets:add_element(B, Acc) end,
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
Switch from binary_bucketlist Allow for bucket listing of non-binary buckets (integer buckets, buckets with ascii strings)
2018-09-01 10:39:23 +01:00
% this should find Bucket and Bucket1 - as we can now find string-based
% buckets using bucket_list - i.e. it isn't just binary buckets now
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, BL} = leveled_bookie:book_bucketlist(Bookie2, ?RIAK_TAG, {FoldBucketsFun, sets:new()}, all),
Switch from binary_bucketlist Allow for bucket listing of non-binary buckets (integer buckets, buckets with ascii strings)
2018-09-01 10:39:23 +01:00
true = sets:size(BL()) == 2,
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
Fold Objects - Check values test Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
ok = leveled_bookie:book_close(Bookie2),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:reset_filestructure().
Experiment with new cache size algorithm Remove the jitter probability and make it a smooth function heading towards the max ache size
2016-11-03 09:19:02 +00:00
query_count(_Config) ->
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
RootPath = testutil:reset_filestructure(),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{ok, Book1} = leveled_bookie:book_start(RootPath,
2000,
50000000,
testutil:sync_strategy()),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin = list_to_binary("Bucket"),
{TestObject, TestSpec} = testutil:generate_testobject(BucketBin,
Change tests - binary keys the new code requires bucket listing to be on binary keys not just binary buckets. As this is only intended for use within Riak (where all keys are buckets are binaries), this constraint seems OK. A test needed changing to ensure it had a binary key in the bucket.
2017-05-23 15:54:11 +01:00
term_to_binary("Key1"),
Redundant clauses Remove some redundant clauses, and fix up some logging
2016-10-19 20:51:30 +01:00
"Value1",
[],
Switch to binary format Riak object Initial change to try and test assuming that leveled received the binary format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
[{"MDK1", "MDV1"}]),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Book1, TestObject, TestSpec),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:check_forobject(Book1, TestObject),
testutil:check_formissingobject(Book1, "Bucket1", "Key2"),
testutil:check_forobject(Book1, TestObject),
lists:foreach(fun(_X) ->
V = testutil:get_compressiblevalue(),
Indexes = testutil:get_randomindexes_generator(8),
SW = os:timestamp(),
ObjL1 = testutil:generate_objects(10000,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
binary_uuid,
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
[],
V,
Indexes),
Shift pause out of leveled Leveled will now signal the need for a pause due to back-pressure, but not actually pause itself. The hope is that in a riak implementation this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
testutil:riakload(Book1, ObjL1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
io:format("Put of 10000 objects with 8 index entries "
++
"each completed in ~w microseconds~n",
[timer:now_diff(os:timestamp(), SW)])
end,
lists:seq(1, 8)),
testutil:check_forobject(Book1, TestObject),
Total = lists:foldl(fun(X, Acc) ->
IdxF = "idx" ++ integer_to_list(X) ++ "_bin",
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
T = count_termsonindex(BucketBin,
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
IdxF,
Book1,
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
?KEY_ONLY),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
io:format("~w terms found on index ~s~n",
[T, IdxF]),
Acc + T
end,
0,
lists:seq(1, 8)),
ok = case Total of
640000 ->
ok
end,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
Index1Count = count_termsonindex(BucketBin,
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
"idx1_bin",
Book1,
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
?KEY_ONLY),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ok = leveled_bookie:book_close(Book1),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{ok, Book2} = leveled_bookie:book_start(RootPath,
1000,
50000000,
testutil:sync_strategy()),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
Index1Count = count_termsonindex(BucketBin,
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
"idx1_bin",
Book2,
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
?KEY_ONLY),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
NameList = testutil:name_list(),
TotalNameByName = lists:foldl(fun({_X, Name}, Acc) ->
{ok, Regex} = re:compile("[0-9]+" ++
Name),
SW = os:timestamp(),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
T = count_termsonindex(BucketBin,
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
"idx1_bin",
Book2,
{false,
Regex}),
TD = timer:now_diff(os:timestamp(),
SW),
io:format("~w terms found on " ++
"index idx1 with a " ++
"regex in ~w " ++
"microseconds~n",
[T, TD]),
Acc + T
end,
0,
NameList),
ok = case TotalNameByName of
Index1Count ->
ok
end,
2i query test The 2i query test added in the previous commit didn't correctly test regex queries. This has now been improved.
2016-10-18 09:42:33 +01:00
{ok, RegMia} = re:compile("[0-9]+Mia"),
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Query1 = {index_query,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{fun testutil:foldkeysfun/3, []},
Change tests - binary keys the new code requires bucket listing to be on binary keys not just binary buckets. As this is only intended for use within Riak (where all keys are buckets are binaries), this constraint seems OK. A test needed changing to ensure it had a binary key in the bucket.
2017-05-23 15:54:11 +01:00
{"idx2_bin", "2000", "2000|"},
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{false, RegMia}},
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{async,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Mia2KFolder1} = leveled_bookie:book_returnfolder(Book2, Query1),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
Mia2000Count1 = length(Mia2KFolder1()),
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Query2 = {index_query,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{fun testutil:foldkeysfun/3, []},
{"idx2_bin", "2000", "2001"},
{true, undefined}},
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{async,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Mia2KFolder2} = leveled_bookie:book_returnfolder(Book2, Query2),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
Mia2000Count2 = lists:foldl(fun({Term, _Key}, Acc) ->
2i query test The 2i query test added in the previous commit didn't correctly test regex queries. This has now been improved.
2016-10-18 09:42:33 +01:00
case re:run(Term, RegMia) of
nomatch ->
Acc;
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
_ ->
2i query test The 2i query test added in the previous commit didn't correctly test regex queries. This has now been improved.
2016-10-18 09:42:33 +01:00
Acc + 1
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
end end,
0,
Mia2KFolder2()),
ok = case Mia2000Count2 of
2i query test The 2i query test added in the previous commit didn't correctly test regex queries. This has now been improved.
2016-10-18 09:42:33 +01:00
Mia2000Count1 when Mia2000Count1 > 0 ->
io:format("Mia2000 counts match at ~w~n",
[Mia2000Count1]),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ok
end,
2i query test The 2i query test added in the previous commit didn't correctly test regex queries. This has now been improved.
2016-10-18 09:42:33 +01:00
{ok, RxMia2K} = re:compile("^2000[0-9]+Mia"),
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Query3 = {index_query,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{fun testutil:foldkeysfun/3, []},
{"idx2_bin", "1980", "2100"},
{false, RxMia2K}},
2i query test The 2i query test added in the previous commit didn't correctly test regex queries. This has now been improved.
2016-10-18 09:42:33 +01:00
{async,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Mia2KFolder3} = leveled_bookie:book_returnfolder(Book2, Query3),
2i query test The 2i query test added in the previous commit didn't correctly test regex queries. This has now been improved.
2016-10-18 09:42:33 +01:00
Mia2000Count1 = length(Mia2KFolder3()),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
V9 = testutil:get_compressiblevalue(),
Indexes9 = testutil:get_randomindexes_generator(8),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
[{_RN, Obj9, Spc9}] = testutil:generate_objects(1,
binary_uuid,
[],
V9,
Indexes9),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Book2, Obj9, Spc9),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
R9 = lists:map(fun({add, IdxF, IdxT}) ->
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Q = {index_query,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{fun testutil:foldkeysfun/3, []},
{IdxF, IdxT, IdxT},
?KEY_ONLY},
R = leveled_bookie:book_returnfolder(Book2, Q),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
{async, Fldr} = R,
case length(Fldr()) of
X when X > 0 ->
{IdxF, IdxT, X}
end
end,
Spc9),
Spc9Del = lists:map(fun({add, IdxF, IdxT}) -> {remove, IdxF, IdxT} end,
Spc9),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Book2, Obj9, Spc9Del),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
lists:foreach(fun({IdxF, IdxT, X}) ->
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Q = {index_query,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{fun testutil:foldkeysfun/3, []},
{IdxF, IdxT, IdxT},
?KEY_ONLY},
R = leveled_bookie:book_returnfolder(Book2, Q),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
{async, Fldr} = R,
case length(Fldr()) of
Y ->
Y = X - 1
end
end,
R9),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
ok = leveled_bookie:book_close(Book2),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{ok, Book3} = leveled_bookie:book_start(RootPath,
2000,
50000000,
testutil:sync_strategy()),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
lists:foreach(fun({IdxF, IdxT, X}) ->
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Q = {index_query,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{fun testutil:foldkeysfun/3, []},
{IdxF, IdxT, IdxT},
?KEY_ONLY},
R = leveled_bookie:book_returnfolder(Book3, Q),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
{async, Fldr} = R,
case length(Fldr()) of
Y ->
Y = X - 1
end
end,
R9),
Clean API of Riak-specific Methods Clena the API of Riak specific methods, and also resolve timing issue in simple_server unit test. Previously this would end up with missing data (and a lower sequence number after start) because of the penciller_clerk timeout being relatively large in the context of this test. Now the timeout has bene reduced the L0 slot is cleared by the time of the close. To make sure an extra sleep has been added as a precaution to avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
ok = testutil:book_riakput(Book3, Obj9, Spc9),
Test fix-up There was a test that failed to close down a bookie and that caused some issues. The issues are double-reoslved, the close down was tidied as well as the forgotten close being added back in. There is some generla tidy around in anticipation of TTL support.
2016-10-21 21:26:28 +01:00
ok = leveled_bookie:book_close(Book3),
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{ok, Book4} = leveled_bookie:book_start(RootPath,
2000,
50000000,
testutil:sync_strategy()),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
lists:foreach(fun({IdxF, IdxT, X}) ->
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Q = {index_query,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
BucketBin,
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
{fun testutil:foldkeysfun/3, []},
{IdxF, IdxT, IdxT},
?KEY_ONLY},
R = leveled_bookie:book_returnfolder(Book4, Q),
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
{async, Fldr} = R,
case length(Fldr()) of
X ->
ok
end
end,
R9),
Redundant clauses Remove some redundant clauses, and fix up some logging
2016-10-19 20:51:30 +01:00
testutil:check_forobject(Book4, TestObject),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
Change FoldBucketsFun to take just bucket FoldBucketsFun does not take keys should be a 2-arity function (Bucket, Acc).
2016-11-21 14:12:17 +00:00
FoldBucketsFun = fun(B, Acc) -> sets:add_element(B, Acc) end,
Switch from binary_bucketlist Allow for bucket listing of non-binary buckets (integer buckets, buckets with ascii strings)
2018-09-01 10:39:23 +01:00
BucketListQuery = {bucket_list,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
?RIAK_TAG,
{FoldBucketsFun, sets:new()}},
{async, BLF1} = leveled_bookie:book_returnfolder(Book4, BucketListQuery),
SW_QA = os:timestamp(),
BucketSet1 = BLF1(),
io:format("Bucket set returned in ~w microseconds",
[timer:now_diff(os:timestamp(), SW_QA)]),
Change tests - binary keys the new code requires bucket listing to be on binary keys not just binary buckets. As this is only intended for use within Riak (where all keys are buckets are binaries), this constraint seems OK. A test needed changing to ensure it had a binary key in the bucket.
2017-05-23 15:54:11 +01:00
true = sets:size(BucketSet1) == 1,
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
ObjList10A = testutil:generate_objects(5000,
binary_uuid,
[],
V9,
Indexes9,
"BucketA"),
ObjList10B = testutil:generate_objects(5000,
binary_uuid,
[],
V9,
Indexes9,
"BucketB"),
ObjList10C = testutil:generate_objects(5000,
binary_uuid,
[],
V9,
Indexes9,
"BucketC"),
testutil:riakload(Book4, ObjList10A),
testutil:riakload(Book4, ObjList10B),
testutil:riakload(Book4, ObjList10C),
{async, BLF2} = leveled_bookie:book_returnfolder(Book4, BucketListQuery),
SW_QB = os:timestamp(),
BucketSet2 = BLF2(),
io:format("Bucket set returned in ~w microseconds",
[timer:now_diff(os:timestamp(), SW_QB)]),
true = sets:size(BucketSet2) == 4,
Complete 2i work - some refactoring The 2i work now has tests for removals as well as regex etc. Some initial refactoring work has also been tried - to try and take some tasks of the critical path of push_mem. The primary change has been to avoid putting index keys into the gb_tree, and building the KeyChanges list in parallel to the gb_tree (now known as ObjectTree) within the Ledger Cache. Some initial experiments done as to changing the ETS table in the Penciller now that it will now be used for iterating - but that has been reverted for now.
2016-10-18 19:41:33 +01:00
ok = leveled_bookie:book_close(Book4),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
Pass through sync_strategy Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
{ok, Book5} = leveled_bookie:book_start(RootPath,
2000,
50000000,
testutil:sync_strategy()),
Fast List Buckets Copied the technique from HanoiDB to speed up list buckets.
2016-11-20 21:21:31 +00:00
{async, BLF3} = leveled_bookie:book_returnfolder(Book5, BucketListQuery),
SW_QC = os:timestamp(),
BucketSet3 = BLF3(),
io:format("Bucket set returned in ~w microseconds",
[timer:now_diff(os:timestamp(), SW_QC)]),
true = sets:size(BucketSet3) == 4,
ok = leveled_bookie:book_close(Book5),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
testutil:reset_filestructure().
count_termsonindex(Bucket, IdxField, Book, QType) ->
lists:foldl(fun(X, Acc) ->
SW = os:timestamp(),
ST = integer_to_list(X),
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
ET = ST ++ "|",
Add FoldKeysFun Add the capability to pass FoldKeysFun into the index_query to allow for compatability with riak backend requirements.
2016-11-18 11:53:14 +00:00
Q = {index_query,
Bucket,
{fun testutil:foldkeysfun/3, []},
{IdxField, ST, ET},
QType},
R = leveled_bookie:book_returnfolder(Book, Q),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
{async, Folder} = R,
Items = length(Folder()),
io:format("2i query from term ~s on index ~s took " ++
"~w microseconds~n",
[ST,
IdxField,
timer:now_diff(os:timestamp(), SW)]),
Acc + Items
end,
0,
Speed up query count test Less individual querys to make count will speed up this taste, without changing the nature of it
2016-11-04 18:20:00 +00:00
lists:seq(190, 221)).
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
multibucket_fold(_Config) ->
RootPath = testutil:reset_filestructure(),
{ok, Bookie1} = leveled_bookie:book_start(RootPath,
2000,
50000000,
testutil:sync_strategy()),
ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = fun() -> [] end,
ObjL1 = testutil:generate_objects(13000,
uuid,
[],
ObjectGen,
IndexGen,
Add support for buckets that are tuples Only {binary(), binary()} tuples
2018-09-27 09:34:40 +01:00
{<<"Type1">>, <<"Bucket1">>}),
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
testutil:riakload(Bookie1, ObjL1),
ObjL2 = testutil:generate_objects(17000,
uuid,
[],
ObjectGen,
IndexGen,
<<"Bucket2">>),
testutil:riakload(Bookie1, ObjL2),
ObjL3 = testutil:generate_objects(7000,
uuid,
[],
ObjectGen,
IndexGen,
<<"Bucket3">>),
testutil:riakload(Bookie1, ObjL3),
ObjL4 = testutil:generate_objects(23000,
uuid,
[],
ObjectGen,
IndexGen,
Add support for buckets that are tuples Only {binary(), binary()} tuples
2018-09-27 09:34:40 +01:00
{<<"Type2">>, <<"Bucket4">>}),
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
testutil:riakload(Bookie1, ObjL4),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
FF = fun(B, K, _PO, Acc) ->
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
[{B, K}|Acc]
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
end,
FoldAccT = {FF, []},
Fix issues with deprecated folders They were deprecated for a reason
2018-10-31 11:04:23 +00:00
{async, R1} =
leveled_bookie:book_headfold(Bookie1,
?RIAK_TAG,
{bucket_list,
[{<<"Type1">>, <<"Bucket1">>},
{<<"Type2">>, <<"Bucket4">>}]},
FoldAccT,
false,
true,
false),
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
O1 = length(R1()),
io:format("Result R1 of length ~w~n", [O1]),
Fix issues with deprecated folders They were deprecated for a reason
2018-10-31 11:04:23 +00:00
{async, R2} =
leveled_bookie:book_headfold(Bookie1,
?RIAK_TAG,
{bucket_list,
[<<"Bucket2">>,
<<"Bucket3">>]},
{fun(_B, _K, _PO, Acc) ->
Acc +1
end,
0},
false, true, false),
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
O2 = R2(),
io:format("Result R2 of ~w~n", [O2]),
true = 36000 == O1,
true = 24000 == O2,
Add support for buckets that are tuples Only {binary(), binary()} tuples
2018-09-27 09:34:40 +01:00
FoldBucketsFun = fun(B, Acc) -> [B|Acc] end,
{async, Folder} =
leveled_bookie:book_bucketlist(Bookie1,
?RIAK_TAG,
{FoldBucketsFun, []},
all),
Reverse necessary on bucket list The function should see the buckets in order, so it accumulates in such a way to reverse the order - it makes sense that the outcome should be in reverse.
2018-11-23 19:03:24 +00:00
BucketList = lists:reverse(Folder()),
Add support for buckets that are tuples Only {binary(), binary()} tuples
2018-09-27 09:34:40 +01:00
ExpectedBucketList =
[{<<"Type1">>, <<"Bucket1">>}, {<<"Type2">>, <<"Bucket4">>},
<<"Bucket2">>, <<"Bucket3">>],
io:format("BucketList ~w", [BucketList]),
true = ExpectedBucketList == BucketList,
Support multi-query fold Allow a single snapshot to run query over multiple ranges. Used initially to fold over multiple buckets.
2018-03-01 23:19:52 +00:00
ok = leveled_bookie:book_close(Bookie1),
testutil:reset_filestructure().
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
rotating_objects(_Config) ->
RootPath = testutil:reset_filestructure(),
Recovery strategy testing Test added for the "retain" recovery strategy. This strategy makes sure a full history of index changes is made so that if the Ledger is wiped out, the Ledger cna be fully rebuilt from the Journal. This exposed two journal compaction problems - The BestRun selected did not have the source files correctly sorted in order before compaction - The compaction process incorrectly dealt with the KeyDelta object left after a compaction - i.e. compacting twice the same key caused that key history to be lost. These issues have now been corrected.
2016-10-27 00:57:19 +01:00
ok = testutil:rotating_object_check(RootPath, "Bucket1", 10),
ok = testutil:rotating_object_check(RootPath, "Bucket2", 200),
ok = testutil:rotating_object_check(RootPath, "Bucket3", 800),
ok = testutil:rotating_object_check(RootPath, "Bucket4", 1600),
ok = testutil:rotating_object_check(RootPath, "Bucket5", 3200),
ok = testutil:rotating_object_check(RootPath, "Bucket6", 9600),
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
testutil:reset_filestructure().
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
foldobjects_bybucket_range(_Config) ->
RootPath = testutil:reset_filestructure(),
{ok, Bookie1} = leveled_bookie:book_start(RootPath,
2000,
50000000,
testutil:sync_strategy()),
ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = fun() -> [] end,
ObjL1 = testutil:generate_objects(1300,
{fixed_binary, 1},
[],
ObjectGen,
IndexGen,
<<"Bucket1">>),
testutil:riakload(Bookie1, ObjL1),
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
Actually run the new test DERP!
2018-09-06 16:38:49 +01:00
FoldKeysFun = fun(_B, K,_V, Acc) ->
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
[ K |Acc]
end,
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
StartKey = testutil:fixed_bin_key(123),
EndKey = testutil:fixed_bin_key(779),
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, Folder} = leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
<<"Bucket1">>,
{StartKey, EndKey}, {FoldKeysFun, []},
true
),
ResLen = length(Folder()),
io:format("Length of Result of folder ~w~n", [ResLen]),
true = 657 == ResLen,
Rotating object tests Recent fixes have been made to problems associated with rapidly changing objexts especially on re-opening of the bookie. Test of rotating objects from both an index query and a fetch perspective added to better detect such issues in the future.
2016-10-20 12:16:17 +01:00
Provide a top level API for folds As the fold functions have been added to get_runner in an ad hoc way, naturally, given the ongoing development of levelEd to support Riak, it was difficult for a new user (in this case Quviq) to see what folds are supported, and with what arguments, and expectations. This PR is for discussion. It is one of many ways to group, spec, and document the fold functions. A test is also added for coverage of range queries.
2018-08-23 10:27:43 +01:00
{async, AllFolder} = leveled_bookie:book_objectfold(Bookie1,
?RIAK_TAG,
<<"Bucket1">>,
all,
{FoldKeysFun, []},
true
),
AllResLen = length(AllFolder()),
io:format("Length of Result of all keys folder ~w~n", [AllResLen]),
true = 1300 == AllResLen,
ok = leveled_bookie:book_close(Bookie1),
testutil:reset_filestructure().
Reference in a new issue Copy permalink