umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions

Mas d34 i453 eqwalizer (#454)

Browse source 
* Add eqwalizer and clear for codec & sst

The eqwalizer errors highlighted the need in several places for type clarification.

Within tests there are some issue where a type is assumed, and so ignore has been used to handle this rather than write more complex code to be explicit about the assumption.

The handling of arrays isn't great by eqwalizer - to be specific about the content of array causes issues when initialising an array.  Perhaps a type (map maybe) where one can be more explicit about types might be a better option (even if there is a minimal performance impact).

The use of a ?TOMB_COUNT defined option complicated the code much more with eqwalizer.  So for now, there is no developer option to disable ?TOMB_COUNT.

Test fixes required where strings have been used for buckets/keys not binaries.

The leveled_sst statem needs a different state record for starting when compared to other modes.  The state record has been divided up to reflect this, to make type management easier.  The impact on performance needs to be tested.

* Update ct tests to support binary keys/buckets only

* Eqwalizer for leveled_cdb and leveled_tictac

As array is used in leveled_tictac - there is the same issue as with leveled_sst

* Remove redundant indirection of leveled_rand

A legacy of pre-20 OTP

* Morde modules eqwalized

ebloom/log/util/monitor

* Eqwalize further modules

elp eqwalize leveled_codec; elp eqwalize leveled_sst; elp eqwalize leveled_cdb; elp eqwalize leveled_tictac; elp eqwalize leveled_log; elp eqwalize leveled_monitor; elp eqwalize leveled_head; elp eqwalize leveled_ebloom; elp eqwalize leveled_iclerk

All concurrently OK

* Refactor unit tests to use binary() no string() in key

Previously string() was allowed just to avoid having to change all these tests.  Go through the pain now, as part of eqwalizing.

* Add fixes for penciller, inker

Add a new ?IS_DEF macro to replace =/= undefined.

Now more explicit about primary, object and query keys

* Further fixes

Need to clarify functions used by runner - where keys , query keys and object keys are used

* Further eqwalisation

* Eqwalize leveled_pmanifest

Also make implementation independent of choice of dict - i.e. one can save a manifest using dict for blooms/pending_deletions and then open a manifest with code that uses a different type.  Allow for slow dict to be replaced with map.

Would not be backwards compatible though, without further thought - i.e. if you upgrade then downgrade.

Redundant code created by leveled_sst refactoring removed.

* Fix backwards compatibility issues

* Manifest Entry to belong to leveled_pmanifest

There are two manifests - leveled_pmanifest and leveled_imanifest.  Both have manifest_entry() type objects, but these types are different.  To avoid confusion don't include the pmanifest manifest_entry() within the global include file - be specific that it belongs to the leveled_pmanifest module

* Ignore elp file - large binary

* Update src/leveled_pmem.erl

Remove unnecessary empty list from type definition

Co-authored-by: Thomas Arts <thomas.arts@quviq.com>

---------

Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
This commit is contained in:
Martin Sumner 2024-11-13 13:37:13 +00:00 • committed by GitHub
parent 1be55fcd15
commit aaeac7ba36
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
30 changed files with 4778 additions and 3334 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -9,3 +9,4 @@ cover
cover_* cover_*
.eqc-info .eqc-info
leveled_data/* leveled_data/*
elp

21
include/leveled.hrl
View file

@ -70,6 +70,15 @@
%% Inker key type used for tombstones %% Inker key type used for tombstones
%%%============================================================================ %%%============================================================================
%%%============================================================================
%%% Helper Function
%%%============================================================================
-define(IS_DEF(Attribute), Attribute =/= undefined).
-if(?OTP_RELEASE < 26).
-type dynamic() :: any().
-endif.
%%%============================================================================ %%%============================================================================
%%% Shared records %%% Shared records
@ -79,13 +88,6 @@
is_basement = false :: boolean(), is_basement = false :: boolean(),
timestamp :: integer()}). timestamp :: integer()}).
-record(manifest_entry,
{start_key :: tuple() | undefined,
end_key :: tuple() | undefined,
owner :: pid()|list(),
filename :: string() | undefined,
bloom = none :: leveled_ebloom:bloom() | none}).
-record(cdb_options, -record(cdb_options,
{max_size :: pos_integer() | undefined, {max_size :: pos_integer() | undefined,
max_count :: pos_integer() | undefined, max_count :: pos_integer() | undefined,
@ -129,14 +131,15 @@
singlefile_compactionperc :: float()|undefined, singlefile_compactionperc :: float()|undefined,
maxrunlength_compactionperc :: float()|undefined, maxrunlength_compactionperc :: float()|undefined,
score_onein = 1 :: pos_integer(), score_onein = 1 :: pos_integer(),
snaptimeout_long :: pos_integer() | undefined, snaptimeout_long = 60 :: pos_integer(),
monitor = {no_monitor, 0} monitor = {no_monitor, 0}
:: leveled_monitor:monitor()}). :: leveled_monitor:monitor()}).
-record(penciller_options, -record(penciller_options,
{root_path :: string() | undefined, {root_path :: string() | undefined,
sst_options = #sst_options{} :: #sst_options{}, sst_options = #sst_options{} :: #sst_options{},
max_inmemory_tablesize :: integer() | undefined, max_inmemory_tablesize = ?MIN_PCL_CACHE_SIZE
:: pos_integer(),
start_snapshot = false :: boolean(), start_snapshot = false :: boolean(),
snapshot_query, snapshot_query,
bookies_pid :: pid() | undefined, bookies_pid :: pid() | undefined,

11
rebar.config
View file

@ -12,6 +12,14 @@
{eunit_opts, [verbose]}. {eunit_opts, [verbose]}.
{project_plugins, [
{eqwalizer_rebar3,
{git_subdir,
"https://github.com/whatsapp/eqwalizer.git",
{branch, "main"},
"eqwalizer_rebar3"}}
]}.
{profiles, {profiles,
[{eqc, [{deps, [meck, fqc]}, [{eqc, [{deps, [meck, fqc]},
{erl_opts, [debug_info, {d, 'EQC'}]}, {erl_opts, [debug_info, {d, 'EQC'}]},
@ -28,7 +36,8 @@
{deps, [ {deps, [
{lz4, ".*", {git, "https://github.com/nhs-riak/erlang-lz4", {branch, "nhse-develop-3.4"}}}, {lz4, ".*", {git, "https://github.com/nhs-riak/erlang-lz4", {branch, "nhse-develop-3.4"}}},
{zstd, ".*", {git, "https://github.com/nhs-riak/zstd-erlang", {branch, "nhse-develop"}}} {zstd, ".*", {git, "https://github.com/nhs-riak/zstd-erlang", {branch, "nhse-develop"}}},
{eqwalizer_support, {git_subdir, "https://github.com/whatsapp/eqwalizer.git", {branch, "main"}, "eqwalizer_support"}}
]}. ]}.
{ct_opts, [{dir, ["test/end_to_end"]}]}. {ct_opts, [{dir, ["test/end_to_end"]}]}.

1176
src/leveled_bookie.erl
View file

File diff suppressed because it is too large Load diff

417
src/leveled_cdb.erl
View file

@ -94,15 +94,11 @@
hashtable_calc/2]). hashtable_calc/2]).
-define(DWORD_SIZE, 8). -define(DWORD_SIZE, 8).
-define(WORD_SIZE, 4).
-define(MAX_FILE_SIZE, 3221225472). -define(MAX_FILE_SIZE, 3221225472).
-define(BINARY_MODE, false). -define(BINARY_MODE, false).
-define(BASE_POSITION, 2048). -define(BASE_POSITION, 2048).
-define(WRITE_OPS, [binary, raw, read, write]). -define(WRITE_OPS, [binary, raw, read, write]).
-define(PENDING_ROLL_WAIT, 30).
-define(DELETE_TIMEOUT, 10000). -define(DELETE_TIMEOUT, 10000).
-define(TIMING_SAMPLECOUNTDOWN, 5000).
-define(TIMING_SAMPLESIZE, 100).
-define(GETPOS_FACTOR, 8). -define(GETPOS_FACTOR, 8).
-define(MAX_OBJECT_SIZE, 1000000000). -define(MAX_OBJECT_SIZE, 1000000000).
% 1GB but really should be much smaller than this % 1GB but really should be much smaller than this
@ -111,18 +107,24 @@
-record(state, {hashtree, -record(state, {hashtree,
last_position :: integer() | undefined, last_position :: integer() | undefined,
% defined when writing, not required once rolled
last_key = empty, last_key = empty,
current_count = 0 :: non_neg_integer(), current_count = 0 :: non_neg_integer(),
hash_index = {} :: tuple(), hash_index = {} :: tuple(),
filename :: string() | undefined, filename :: string() | undefined,
handle :: file:fd() | undefined, % defined when starting
max_size :: pos_integer() | undefined, handle :: file:io_device() | undefined,
max_count :: pos_integer() | undefined, % defined when starting
max_size :: pos_integer(),
max_count :: pos_integer(),
binary_mode = false :: boolean(), binary_mode = false :: boolean(),
delete_point = 0 :: integer(), delete_point = 0 :: integer(),
inker :: pid() | undefined, inker :: pid() | undefined,
% undefined until delete_pending
deferred_delete = false :: boolean(), deferred_delete = false :: boolean(),
waste_path :: string() | undefined, waste_path :: string()|undefined,
% undefined has functional meaning
% - no sending to waste on delete
sync_strategy = none, sync_strategy = none,
log_options = leveled_log:get_opts() log_options = leveled_log:get_opts()
:: leveled_log:log_options(), :: leveled_log:log_options(),
@ -133,8 +135,11 @@
-type hashtable_index() :: tuple(). -type hashtable_index() :: tuple().
-type file_location() :: integer()|eof. -type file_location() :: integer()|eof.
-type filter_fun() :: -type filter_fun() ::
fun((any(), binary(), integer(), any(), fun((binary()) -> any())) -> fun((any(),
{stop|loop, any()}). binary(),
integer(),
term()|{term(), term()},
fun((binary()) -> any())) -> {stop|loop, any()}).
-export_type([filter_fun/0]). -export_type([filter_fun/0]).
@ -265,11 +270,11 @@ cdb_getpositions(Pid, SampleSize) ->
cdb_getpositions_fromidx(Pid, FC, Index, Acc) cdb_getpositions_fromidx(Pid, FC, Index, Acc)
end end
end, end,
RandFun = fun(X) -> {leveled_rand:uniform(), X} end, RandFun = fun(X) -> {rand:uniform(), X} end,
SeededL = lists:map(RandFun, lists:seq(0, 255)), SeededL = lists:map(RandFun, lists:seq(0, 255)),
SortedL = lists:keysort(1, SeededL), SortedL = lists:keysort(1, SeededL),
PosList0 = lists:foldl(FoldFun, [], SortedL), PosList0 = lists:foldl(FoldFun, [], SortedL),
P1 = leveled_rand:uniform(max(1, length(PosList0) - S0)), P1 = rand:uniform(max(1, length(PosList0) - S0)),
lists:sublist(lists:sort(PosList0), P1, S0) lists:sublist(lists:sort(PosList0), P1, S0)
end. end.
@ -367,7 +372,7 @@ cdb_scan(Pid, FilterFun, InitAcc, StartPosition) ->
{cdb_scan, FilterFun, InitAcc, StartPosition}, {cdb_scan, FilterFun, InitAcc, StartPosition},
infinity). infinity).
-spec cdb_lastkey(pid()) -> any(). -spec cdb_lastkey(pid()) -> leveled_codec:journal_key()|empty.
%% @doc %% @doc
%% Get the last key to be added to the file (which will have the highest %% Get the last key to be added to the file (which will have the highest
%% sequence number) %% sequence number)
@ -487,38 +492,49 @@ starting({call, From}, {open_reader, Filename, LastKey}, State) ->
{next_state, reader, State0, [{reply, From, ok}, hibernate]}. {next_state, reader, State0, [{reply, From, ok}, hibernate]}.
writer({call, From}, {get_kv, Key}, State) -> writer(
{call, From}, {get_kv, Key}, State = #state{handle =IO})
when ?IS_DEF(IO) ->
{keep_state_and_data, {keep_state_and_data,
[{reply, [{reply,
From, From,
get_mem( get_mem(
Key, Key,
State#state.handle, IO,
State#state.hashtree, State#state.hashtree,
State#state.binary_mode)}]}; State#state.binary_mode)}]};
writer({call, From}, {key_check, Key}, State) -> writer(
{call, From}, {key_check, Key}, State = #state{handle =IO})
when ?IS_DEF(IO) ->
{keep_state_and_data, {keep_state_and_data,
[{reply, [{reply,
From, From,
get_mem( get_mem(
Key, Key,
State#state.handle, IO,
State#state.hashtree, State#state.hashtree,
State#state.binary_mode, State#state.binary_mode,
loose_presence)}]}; loose_presence)}]};
writer({call, From}, {put_kv, Key, Value, Sync}, State) -> writer(
{call, From},
{put_kv, Key, Value, Sync},
State = #state{last_position = LP, handle = IO})
when ?IS_DEF(last_position), ?IS_DEF(IO) ->
NewCount = State#state.current_count + 1, NewCount = State#state.current_count + 1,
case NewCount >= State#state.max_count of case NewCount >= State#state.max_count of
true -> true ->
{keep_state_and_data, [{reply, From, roll}]}; {keep_state_and_data, [{reply, From, roll}]};
false -> false ->
Result = put(State#state.handle, Result =
put(
IO,
Key, Key,
Value, Value,
{State#state.last_position, State#state.hashtree}, {LP, State#state.hashtree},
State#state.binary_mode, State#state.binary_mode,
State#state.max_size, State#state.max_size,
State#state.last_key == empty), State#state.last_key == empty
),
case Result of case Result of
roll -> roll ->
%% Key and value could not be written %% Key and value could not be written
@ -545,7 +561,11 @@ writer({call, From}, {put_kv, Key, Value, Sync}, State) ->
end; end;
writer({call, From}, {mput_kv, []}, _State) -> writer({call, From}, {mput_kv, []}, _State) ->
{keep_state_and_data, [{reply, From, ok}]}; {keep_state_and_data, [{reply, From, ok}]};
writer({call, From}, {mput_kv, KVList}, State) -> writer(
{call, From},
{mput_kv, KVList},
State = #state{last_position = LP, handle = IO})
when ?IS_DEF(last_position), ?IS_DEF(IO) ->
NewCount = State#state.current_count + length(KVList), NewCount = State#state.current_count + length(KVList),
TooMany = NewCount >= State#state.max_count, TooMany = NewCount >= State#state.max_count,
NotEmpty = State#state.current_count > 0, NotEmpty = State#state.current_count > 0,
@ -553,11 +573,14 @@ writer({call, From}, {mput_kv, KVList}, State) ->
true -> true ->
{keep_state_and_data, [{reply, From, roll}]}; {keep_state_and_data, [{reply, From, roll}]};
false -> false ->
Result = mput(State#state.handle, Result =
mput(
IO,
KVList, KVList,
{State#state.last_position, State#state.hashtree}, {LP, State#state.hashtree},
State#state.binary_mode, State#state.binary_mode,
State#state.max_size), State#state.max_size
),
case Result of case Result of
roll -> roll ->
%% Keys and values could not be written %% Keys and values could not be written
@ -573,38 +596,46 @@ writer({call, From}, {mput_kv, KVList}, State) ->
[{reply, From, ok}]} [{reply, From, ok}]}
end end
end; end;
writer({call, From}, cdb_complete, State) -> writer(
NewName = determine_new_filename(State#state.filename), {call, From}, cdb_complete, State = #state{filename = FN})
when ?IS_DEF(FN) ->
NewName = determine_new_filename(FN),
ok = close_file(State#state.handle, ok = close_file(State#state.handle,
State#state.hashtree, State#state.hashtree,
State#state.last_position), State#state.last_position),
ok = rename_for_read(State#state.filename, NewName), ok = rename_for_read(FN, NewName),
{stop_and_reply, normal, [{reply, From, {ok, NewName}}]}; {stop_and_reply, normal, [{reply, From, {ok, NewName}}]};
writer({call, From}, Event, State) -> writer({call, From}, Event, State) ->
handle_sync_event(Event, From, State); handle_sync_event(Event, From, State);
writer(cast, cdb_roll, State) -> writer(
cast, cdb_roll, State = #state{last_position = LP})
when ?IS_DEF(LP) ->
ok = ok =
leveled_iclerk:clerk_hashtablecalc( leveled_iclerk:clerk_hashtablecalc(
State#state.hashtree, State#state.last_position, self()), State#state.hashtree, LP, self()),
{next_state, rolling, State}. {next_state, rolling, State}.
rolling({call, From}, {get_kv, Key}, State) -> rolling(
{call, From}, {get_kv, Key}, State = #state{handle = IO})
when ?IS_DEF(IO) ->
{keep_state_and_data, {keep_state_and_data,
[{reply, [{reply,
From, From,
get_mem( get_mem(
Key, Key,
State#state.handle, IO,
State#state.hashtree, State#state.hashtree,
State#state.binary_mode)}]}; State#state.binary_mode)}]};
rolling({call, From}, {key_check, Key}, State) -> rolling(
{call, From}, {key_check, Key}, State = #state{handle = IO})
when ?IS_DEF(IO) ->
{keep_state_and_data, {keep_state_and_data,
[{reply, [{reply,
From, From,
get_mem( get_mem(
Key, Key,
State#state.handle, IO,
State#state.hashtree, State#state.hashtree,
State#state.binary_mode, State#state.binary_mode,
loose_presence)}]}; loose_presence)}]};
@ -612,15 +643,19 @@ rolling({call, From},
{get_positions, _SampleSize, _Index, SampleAcc}, {get_positions, _SampleSize, _Index, SampleAcc},
_State) -> _State) ->
{keep_state_and_data, [{reply, From, SampleAcc}]}; {keep_state_and_data, [{reply, From, SampleAcc}]};
rolling({call, From}, {return_hashtable, IndexList, HashTreeBin}, State) -> rolling(
{call, From},
{return_hashtable, IndexList, HashTreeBin},
State = #state{filename = FN})
when ?IS_DEF(FN) ->
SW = os:timestamp(), SW = os:timestamp(),
Handle = State#state.handle, Handle = State#state.handle,
{ok, BasePos} = file:position(Handle, State#state.last_position), {ok, BasePos} = file:position(Handle, State#state.last_position),
NewName = determine_new_filename(State#state.filename), NewName = determine_new_filename(FN),
ok = perform_write_hash_tables(Handle, HashTreeBin, BasePos), ok = perform_write_hash_tables(Handle, HashTreeBin, BasePos),
ok = write_top_index_table(Handle, BasePos, IndexList), ok = write_top_index_table(Handle, BasePos, IndexList),
file:close(Handle), file:close(Handle),
ok = rename_for_read(State#state.filename, NewName), ok = rename_for_read(FN, NewName),
leveled_log:log(cdb03, [NewName]), leveled_log:log(cdb03, [NewName]),
ets:delete(State#state.hashtree), ets:delete(State#state.hashtree),
{NewHandle, Index, LastKey} = {NewHandle, Index, LastKey} =
@ -646,13 +681,17 @@ rolling(cast, {delete_pending, ManSQN, Inker}, State) ->
{keep_state, {keep_state,
State#state{delete_point=ManSQN, inker=Inker, deferred_delete=true}}. State#state{delete_point=ManSQN, inker=Inker, deferred_delete=true}}.
reader({call, From}, {get_kv, Key}, State) -> reader(
{call, From}, {get_kv, Key}, State = #state{handle = IO})
when ?IS_DEF(IO) ->
Result = Result =
get_withcache(State#state.handle, get_withcache(
IO,
Key, Key,
State#state.hash_index, State#state.hash_index,
State#state.binary_mode, State#state.binary_mode,
State#state.monitor), State#state.monitor
),
{keep_state_and_data, [{reply, From, Result}]}; {keep_state_and_data, [{reply, From, Result}]};
reader({call, From}, {key_check, Key}, State) -> reader({call, From}, {key_check, Key}, State) ->
Result = Result =
@ -673,8 +712,11 @@ reader({call, From}, {get_positions, SampleSize, Index, Acc}, State) ->
{keep_state_and_data, {keep_state_and_data,
[{reply, From, lists:sublist(UpdAcc, SampleSize)}]} [{reply, From, lists:sublist(UpdAcc, SampleSize)}]}
end; end;
reader({call, From}, {direct_fetch, PositionList, Info}, State) -> reader(
H = State#state.handle, {call, From},
{direct_fetch, PositionList, Info},
State = #state{handle = IO})
when ?IS_DEF(IO) ->
FilterFalseKey = FilterFalseKey =
fun(Tpl) -> fun(Tpl) ->
case element(1, Tpl) of case element(1, Tpl) of
@ -687,20 +729,23 @@ reader({call, From}, {direct_fetch, PositionList, Info}, State) ->
case Info of case Info of
key_only -> key_only ->
FM = lists:filtermap( FM =
lists:filtermap(
fun(P) -> fun(P) ->
FilterFalseKey(extract_key(H, P)) end, FilterFalseKey(extract_key(IO, P))
PositionList), end,
PositionList
),
MapFun = fun(T) -> element(1, T) end, MapFun = fun(T) -> element(1, T) end,
{keep_state_and_data, {keep_state_and_data,
[{reply, From, lists:map(MapFun, FM)}]}; [{reply, From, lists:map(MapFun, FM)}]};
key_size -> key_size ->
FilterFun = fun(P) -> FilterFalseKey(extract_key_size(H, P)) end, FilterFun = fun(P) -> FilterFalseKey(extract_key_size(IO, P)) end,
{keep_state_and_data, {keep_state_and_data,
[{reply, From, lists:filtermap(FilterFun, PositionList)}]}; [{reply, From, lists:filtermap(FilterFun, PositionList)}]};
key_value_check -> key_value_check ->
BM = State#state.binary_mode, BM = State#state.binary_mode,
MapFun = fun(P) -> extract_key_value_check(H, P, BM) end, MapFun = fun(P) -> extract_key_value_check(IO, P, BM) end,
% direct_fetch will occur in batches, so it doesn't make sense to % direct_fetch will occur in batches, so it doesn't make sense to
% hibernate the process that is likely to be used again. However, % hibernate the process that is likely to be used again. However,
% a significant amount of unused binary references may have % a significant amount of unused binary references may have
@ -709,12 +754,13 @@ reader({call, From}, {direct_fetch, PositionList, Info}, State) ->
garbage_collect(), garbage_collect(),
{keep_state_and_data, []} {keep_state_and_data, []}
end; end;
reader({call, From}, cdb_complete, State) -> reader(
leveled_log:log(cdb05, [State#state.filename, reader, cdb_ccomplete]), {call, From}, cdb_complete, State = #state{filename = FN, handle = IO})
ok = file:close(State#state.handle), when ?IS_DEF(FN), ?IS_DEF(IO) ->
{stop_and_reply, leveled_log:log(cdb05, [FN, reader, cdb_ccomplete]),
normal, ok = file:close(IO),
[{reply, From, {ok, State#state.filename}}], {stop_and_reply, normal,
[{reply, From, {ok, FN}}],
State#state{handle=undefined}}; State#state{handle=undefined}};
reader({call, From}, check_hashtable, _State) -> reader({call, From}, check_hashtable, _State) ->
{keep_state_and_data, [{reply, From, true}]}; {keep_state_and_data, [{reply, From, true}]};
@ -731,69 +777,77 @@ reader(cast, clerk_complete, _State) ->
{keep_state_and_data, [hibernate]}. {keep_state_and_data, [hibernate]}.
delete_pending({call, From}, {get_kv, Key}, State) -> delete_pending(
{call, From}, {get_kv, Key}, State = #state{handle = IO})
when ?IS_DEF(IO) ->
Result = Result =
get_withcache(State#state.handle, get_withcache(
IO,
Key, Key,
State#state.hash_index, State#state.hash_index,
State#state.binary_mode, State#state.binary_mode,
State#state.monitor), State#state.monitor
),
{keep_state_and_data, [{reply, From, Result}, ?DELETE_TIMEOUT]}; {keep_state_and_data, [{reply, From, Result}, ?DELETE_TIMEOUT]};
delete_pending({call, From}, {key_check, Key}, State) -> delete_pending(
{call, From}, {key_check, Key}, State = #state{handle = IO})
when ?IS_DEF(IO) ->
Result = Result =
get_withcache(State#state.handle, get_withcache(
IO,
Key, Key,
State#state.hash_index, State#state.hash_index,
loose_presence, loose_presence,
State#state.binary_mode, State#state.binary_mode,
{no_monitor, 0}), {no_monitor, 0}
),
{keep_state_and_data, [{reply, From, Result}, ?DELETE_TIMEOUT]}; {keep_state_and_data, [{reply, From, Result}, ?DELETE_TIMEOUT]};
delete_pending({call, From}, cdb_close, State) -> delete_pending(
leveled_log:log(cdb05, [State#state.filename, delete_pending, cdb_close]), {call, From}, cdb_close, State = #state{handle = IO, filename = FN})
close_pendingdelete(State#state.handle, when ?IS_DEF(FN), ?IS_DEF(IO) ->
State#state.filename, leveled_log:log(cdb05, [FN, delete_pending, cdb_close]),
State#state.waste_path), close_pendingdelete(IO, FN, State#state.waste_path),
{stop_and_reply, normal, [{reply, From, ok}]}; {stop_and_reply, normal, [{reply, From, ok}]};
delete_pending(cast, delete_confirmed, State=#state{delete_point=ManSQN}) -> delete_pending(
leveled_log:log(cdb04, [State#state.filename, ManSQN]), cast, delete_confirmed, State = #state{handle = IO, filename = FN})
close_pendingdelete(State#state.handle, when ?IS_DEF(FN), ?IS_DEF(IO) ->
State#state.filename, leveled_log:log(cdb04, [FN, State#state.delete_point]),
State#state.waste_path), close_pendingdelete(IO, FN, State#state.waste_path),
{stop, normal};
delete_pending(cast, destroy, State) ->
leveled_log:log(cdb05, [State#state.filename, delete_pending, destroy]),
close_pendingdelete(State#state.handle,
State#state.filename,
State#state.waste_path),
{stop, normal}; {stop, normal};
delete_pending( delete_pending(
timeout, _, State=#state{delete_point=ManSQN}) when ManSQN > 0 -> cast, destroy, State = #state{handle = IO, filename = FN})
when ?IS_DEF(FN), ?IS_DEF(IO) ->
leveled_log:log(cdb05, [FN, delete_pending, destroy]),
close_pendingdelete(IO, FN, State#state.waste_path),
{stop, normal};
delete_pending(
timeout, _, State=#state{delete_point=ManSQN, handle = IO, filename = FN})
when ManSQN > 0, ?IS_DEF(FN), ?IS_DEF(IO) ->
case is_process_alive(State#state.inker) of case is_process_alive(State#state.inker) of
true -> true ->
ok = ok =
leveled_inker:ink_confirmdelete(State#state.inker, leveled_inker:ink_confirmdelete(
ManSQN, State#state.inker, ManSQN, self()),
self()),
{keep_state_and_data, [?DELETE_TIMEOUT]}; {keep_state_and_data, [?DELETE_TIMEOUT]};
false -> false ->
leveled_log:log(cdb04, [State#state.filename, ManSQN]), leveled_log:log(cdb04, [FN, ManSQN]),
close_pendingdelete(State#state.handle, close_pendingdelete(IO, FN, State#state.waste_path),
State#state.filename,
State#state.waste_path),
{stop, normal} {stop, normal}
end. end.
handle_sync_event({cdb_scan, FilterFun, Acc, StartPos}, From, State) -> handle_sync_event(
{ok, EndPos0} = file:position(State#state.handle, eof), {cdb_scan, FilterFun, Acc, StartPos}, From, State = #state{handle = IO})
when ?IS_DEF(IO) ->
{ok, EndPos0} = file:position(IO, eof),
{ok, StartPos0} = {ok, StartPos0} =
case StartPos of case StartPos of
undefined -> undefined ->
file:position(State#state.handle, ?BASE_POSITION); file:position(IO, ?BASE_POSITION);
StartPos -> StartPos ->
{ok, StartPos} {ok, StartPos}
end, end,
file:position(State#state.handle, StartPos0), file:position(IO, StartPos0),
MaybeEnd = MaybeEnd =
(check_last_key(State#state.last_key) == empty) or (check_last_key(State#state.last_key) == empty) or
(StartPos0 >= (EndPos0 - ?DWORD_SIZE)), (StartPos0 >= (EndPos0 - ?DWORD_SIZE)),
@ -802,11 +856,13 @@ handle_sync_event({cdb_scan, FilterFun, Acc, StartPos}, From, State) ->
true -> true ->
{eof, Acc}; {eof, Acc};
false -> false ->
scan_over_file(State#state.handle, scan_over_file(
IO,
StartPos0, StartPos0,
FilterFun, FilterFun,
Acc, Acc,
State#state.last_key) State#state.last_key
)
end, end,
% The scan may have created a lot of binary references, clear up the % The scan may have created a lot of binary references, clear up the
% reference counters for this process here manually. The cdb process % reference counters for this process here manually. The cdb process
@ -821,20 +877,26 @@ handle_sync_event({cdb_scan, FilterFun, Acc, StartPos}, From, State) ->
{keep_state_and_data, []}; {keep_state_and_data, []};
handle_sync_event(cdb_lastkey, From, State) -> handle_sync_event(cdb_lastkey, From, State) ->
{keep_state_and_data, [{reply, From, State#state.last_key}]}; {keep_state_and_data, [{reply, From, State#state.last_key}]};
handle_sync_event(cdb_firstkey, From, State) -> handle_sync_event(
{ok, EOFPos} = file:position(State#state.handle, eof), cdb_firstkey, From, State = #state{handle = IO})
FilterFun = fun(Key, _V, _P, _O, _Fun) -> {stop, Key} end, when ?IS_DEF(IO) ->
{ok, EOFPos} = file:position(IO, eof),
FirstKey = FirstKey =
case EOFPos of case EOFPos of
?BASE_POSITION -> ?BASE_POSITION ->
empty; empty;
_ -> _ ->
file:position(State#state.handle, ?BASE_POSITION), FindFirstKeyFun =
{_Pos, FirstScanKey} = scan_over_file(State#state.handle, fun(Key, _V, _P, _O, _Fun) -> {stop, Key} end,
file:position(IO, ?BASE_POSITION),
{_Pos, FirstScanKey} =
scan_over_file(
IO,
?BASE_POSITION, ?BASE_POSITION,
FilterFun, FindFirstKeyFun,
empty, empty,
State#state.last_key), State#state.last_key
),
FirstScanKey FirstScanKey
end, end,
{keep_state_and_data, [{reply, From, FirstKey}]}; {keep_state_and_data, [{reply, From, FirstKey}]};
@ -861,14 +923,15 @@ handle_sync_event({put_cachedscore, Score}, From, State) ->
{keep_state, {keep_state,
State#state{cached_score = {Score,os:timestamp()}}, State#state{cached_score = {Score,os:timestamp()}},
[{reply, From, ok}]}; [{reply, From, ok}]};
handle_sync_event(cdb_close, From, State) -> handle_sync_event(
file:close(State#state.handle), cdb_close, From, _State = #state{handle = IO})
when ?IS_DEF(IO) ->
file:close(IO),
{stop_and_reply, normal, [{reply, From, ok}]}. {stop_and_reply, normal, [{reply, From, ok}]}.
terminate(_Reason, _StateName, _State) -> terminate(_Reason, _StateName, _State) ->
ok. ok.
code_change(_OldVsn, StateName, State, _Extra) -> code_change(_OldVsn, StateName, State, _Extra) ->
{ok, StateName, State}. {ok, StateName, State}.
@ -877,7 +940,6 @@ code_change(_OldVsn, StateName, State, _Extra) ->
%%% External functions %%% External functions
%%%============================================================================ %%%============================================================================
finished_rolling(CDB) -> finished_rolling(CDB) ->
RollerFun = RollerFun =
fun(Sleep, FinishedRolling) -> fun(Sleep, FinishedRolling) ->
@ -908,9 +970,9 @@ close_pendingdelete(Handle, Filename, WasteFP) ->
undefined -> undefined ->
ok = file:delete(Filename); ok = file:delete(Filename);
WasteFP -> WasteFP ->
Components = filename:split(Filename), FN = filename:basename(Filename),
NewName = WasteFP ++ lists:last(Components), NewName = filename:join(WasteFP, FN),
file:rename(Filename, NewName) ok = file:rename(Filename, NewName)
end; end;
false -> false ->
% This may happen when there has been a destroy while files are % This may happen when there has been a destroy while files are
@ -1179,7 +1241,7 @@ find_lastkey(Handle, IndexCache) ->
_ -> _ ->
{ok, _} = file:position(Handle, LastPosition), {ok, _} = file:position(Handle, LastPosition),
{KeyLength, _ValueLength} = read_next_2_integers(Handle), {KeyLength, _ValueLength} = read_next_2_integers(Handle),
safe_read_next_key(Handle, KeyLength) safe_read_next(Handle, KeyLength, key)
end. end.
@ -1239,7 +1301,7 @@ extract_kvpair(_H, [], _K, _BinaryMode) ->
extract_kvpair(Handle, [Position|Rest], Key, BinaryMode) -> extract_kvpair(Handle, [Position|Rest], Key, BinaryMode) ->
{ok, _} = file:position(Handle, Position), {ok, _} = file:position(Handle, Position),
{KeyLength, ValueLength} = read_next_2_integers(Handle), {KeyLength, ValueLength} = read_next_2_integers(Handle),
case safe_read_next_keybin(Handle, KeyLength) of case safe_read_next(Handle, KeyLength, keybin) of
{Key, KeyBin} -> % If same key as passed in, then found! {Key, KeyBin} -> % If same key as passed in, then found!
case checkread_next_value(Handle, ValueLength, KeyBin) of case checkread_next_value(Handle, ValueLength, KeyBin) of
{false, _} -> {false, _} ->
@ -1259,12 +1321,12 @@ extract_kvpair(Handle, [Position|Rest], Key, BinaryMode) ->
extract_key(Handle, Position) -> extract_key(Handle, Position) ->
{ok, _} = file:position(Handle, Position), {ok, _} = file:position(Handle, Position),
{KeyLength, _ValueLength} = read_next_2_integers(Handle), {KeyLength, _ValueLength} = read_next_2_integers(Handle),
{safe_read_next_key(Handle, KeyLength)}. {safe_read_next(Handle, KeyLength, key)}.
extract_key_size(Handle, Position) -> extract_key_size(Handle, Position) ->
{ok, _} = file:position(Handle, Position), {ok, _} = file:position(Handle, Position),
{KeyLength, ValueLength} = read_next_2_integers(Handle), {KeyLength, ValueLength} = read_next_2_integers(Handle),
K = safe_read_next_key(Handle, KeyLength), K = safe_read_next(Handle, KeyLength, key),
{K, ValueLength}. {K, ValueLength}.
extract_key_value_check(Handle, Position, BinaryMode) -> extract_key_value_check(Handle, Position, BinaryMode) ->
@ -1279,32 +1341,35 @@ extract_key_value_check(Handle, Position, BinaryMode) ->
end. end.
-spec startup_scan_over_file(file:io_device(), file_location()) -spec startup_scan_over_file(
-> {file_location(), any()}. file:io_device(), integer()) -> {integer(), {ets:tid(), term()}}.
%% @doc %% @doc
%% Scan through the file until there is a failure to crc check an input, and %% Scan through the file until there is a failure to crc check an input, and
%% at that point return the position and the key dictionary scanned so far %% at that point return the position and the key dictionary scanned so far
startup_scan_over_file(Handle, Position) -> startup_scan_over_file(Handle, Position) ->
HashTree = new_hashtree(), Hashtree = new_hashtree(),
{eof, Output} = scan_over_file(Handle, FilterFun = startup_filter(Hashtree),
Position, {eof, LastKey} = scan_over_file(Handle, Position, FilterFun, empty, empty),
fun startup_filter/5,
{HashTree, empty},
empty),
{ok, FinalPos} = file:position(Handle, cur), {ok, FinalPos} = file:position(Handle, cur),
{FinalPos, Output}. {FinalPos, {Hashtree, LastKey}}.
-spec startup_filter(ets:tid()) -> filter_fun().
%% @doc %% @doc
%% Specific filter to be used at startup to build a hashtree for an incomplete %% Specific filter to be used at startup to build a hashtree for an incomplete
%% cdb file, and returns at the end the hashtree and the final Key seen in the %% cdb file, and returns at the end the hashtree and the final Key seen in the
%% journal %% journal
startup_filter(Key, _ValueAsBin, Position, {Hashtree, _LastKey}, _ExtractFun) -> startup_filter(Hashtree) ->
{loop, {put_hashtree(Key, Position, Hashtree), Key}}. FilterFun =
fun(Key, _ValueAsBin, Position, _LastKey, _ExtractFun) ->
put_hashtree(Key, Position, Hashtree),
{loop, Key}
end,
FilterFun.
-spec scan_over_file(file:io_device(), file_location(), -spec scan_over_file
filter_fun(), any(), any()) -> {file_location(), any()}. (file:io_device(), integer(), filter_fun(), term(), any()) ->
{file_location(), term()}.
%% Scan for key changes - scan over file returning applying FilterFun %% Scan for key changes - scan over file returning applying FilterFun
%% The FilterFun should accept as input: %% The FilterFun should accept as input:
%% - Key, ValueBin, Position, Accumulator, Fun (to extract values from Binary) %% - Key, ValueBin, Position, Accumulator, Fun (to extract values from Binary)
@ -1324,7 +1389,8 @@ scan_over_file(Handle, Position, FilterFun, Output, LastKey) ->
{ok, Position} = file:position(Handle, {bof, Position}), {ok, Position} = file:position(Handle, {bof, Position}),
{eof, Output}; {eof, Output};
{Key, ValueAsBin, KeyLength, ValueLength} -> {Key, ValueAsBin, KeyLength, ValueLength} ->
NewPosition = case Key of NewPosition =
case Key of
LastKey -> LastKey ->
eof; eof;
_ -> _ ->
@ -1360,9 +1426,8 @@ check_last_key(empty) ->
check_last_key(_LK) -> check_last_key(_LK) ->
ok. ok.
-spec saferead_keyvalue(
-spec saferead_keyvalue(file:io_device()) file:io_device()) -> false|{any(), binary(), integer(), integer()}.
-> false|{any(), any(), integer(), integer()}.
%% @doc %% @doc
%% Read the Key/Value at this point, returning {ok, Key, Value} %% Read the Key/Value at this point, returning {ok, Key, Value}
%% catch expected exceptions associated with file corruption (or end) and %% catch expected exceptions associated with file corruption (or end) and
@ -1372,11 +1437,11 @@ saferead_keyvalue(Handle) ->
eof -> eof ->
false; false;
{KeyL, ValueL} when is_integer(KeyL), is_integer(ValueL) -> {KeyL, ValueL} when is_integer(KeyL), is_integer(ValueL) ->
case safe_read_next_keybin(Handle, KeyL) of case safe_read_next(Handle, KeyL, keybin) of
false -> false ->
false; false;
{Key, KeyBin} -> {Key, KeyBin} ->
case safe_read_next_value(Handle, ValueL, KeyBin) of case safe_read_next(Handle, ValueL, {value, KeyBin}) of
false -> false ->
false; false;
TrueValue -> TrueValue ->
@ -1388,66 +1453,37 @@ saferead_keyvalue(Handle) ->
false false
end. end.
-spec safe_read_next
-spec safe_read_next_key(file:io_device(), integer()) -> false|term(). (file:io_device(), integer(), key) -> false|term();
%% @doc (file:io_device(), integer(), keybin) -> false|{term(), binary()};
%% Return the next key or have false returned if there is some sort of (file:io_device(), integer(), {value, binary()}) -> false|binary().
%% potentially expected error (e.g. due to file truncation). Note that no
%% CRC check has been performed
safe_read_next_key(Handle, Length) ->
ReadFun = fun(Bin) -> binary_to_term(Bin) end,
safe_read_next(Handle, Length, ReadFun).
-spec safe_read_next_keybin(file:io_device(), integer())
-> false|{term(), binary()}.
%% @doc
%% Return the next key or have false returned if there is some sort of
%% potentially expected error (e.g. due to file truncation). Note that no
%% CRC check has been performed
%% Returns both the Key and the Binary version, the binary version being
%% required for the CRC checking after the value fetch (see
%% safe_read_next_value/3)
safe_read_next_keybin(Handle, Length) ->
ReadFun = fun(Bin) -> {binary_to_term(Bin), Bin} end,
safe_read_next(Handle, Length, ReadFun).
-spec safe_read_next_value(file:io_device(), integer(), binary())
-> binary()|false.
safe_read_next_value(Handle, Length, KeyBin) ->
ReadFun = fun(VBin) -> crccheck(VBin, KeyBin) end,
safe_read_next(Handle, Length, ReadFun).
-type read_output() :: {term(), binary()}|binary()|term()|false.
-type read_fun() :: fun((binary()) -> read_output()).
-spec safe_read_next(file:io_device(), integer(), read_fun())
-> read_output().
%% @doc %% @doc
%% Read the next item of length Length %% Read the next item of length Length
%% Previously catching error:badarg was sufficient to capture errors of %% Previously catching error:badarg was sufficient to capture errors of
%% corruption, but on some OS versions may need to catch error:einval as well %% corruption, but on some OS versions may need to catch error:einval as well
safe_read_next(Handle, Length, ReadFun) -> safe_read_next(Handle, Length, ReadType) ->
ReadFun =
case ReadType of
key ->
fun(Bin) -> binary_to_term(Bin) end;
keybin ->
fun(KBin) -> {binary_to_term(KBin), KBin} end;
{value, KeyBin} ->
fun(VBin) -> crccheck(VBin, KeyBin) end
end,
try try
loose_read(Handle, Length, ReadFun)
catch
error:ReadError ->
leveled_log:log(cdb20, [ReadError, Length]),
false
end.
-spec loose_read(file:io_device(), integer(), read_fun()) -> read_output().
%% @doc
%% Read with minimal error handling (only eof) - to be wrapped in
%% safe_read_next/3 to catch exceptions.
loose_read(Handle, Length, ReadFun) ->
case file:read(Handle, Length) of case file:read(Handle, Length) of
eof -> eof ->
false; false;
{ok, Result} -> {ok, Result} ->
ReadFun(Result) ReadFun(Result)
end
catch
error:ReadError ->
leveled_log:log(cdb20, [ReadError, Length]),
false
end. end.
-spec crccheck(binary()|bitstring(), binary()) -> any(). -spec crccheck(binary()|bitstring(), binary()) -> any().
%% @doc %% @doc
%% CRC chaeck the value which should be a binary, where the first four bytes %% CRC chaeck the value which should be a binary, where the first four bytes
@ -1472,8 +1508,9 @@ crccheck(_V, _KB) ->
calc_crc(KeyBin, Value) -> erlang:crc32(<<KeyBin/binary, Value/binary>>). calc_crc(KeyBin, Value) -> erlang:crc32(<<KeyBin/binary, Value/binary>>).
-spec checkread_next_value(file:io_device(), integer(), binary()) -spec checkread_next_value
-> {boolean(), binary()|crc_wonky}. (file:io_device(), integer(), binary()) ->
{true, binary()}|{false, crc_wonky}.
%% @doc %% @doc
%% Read next string where the string has a CRC prepended - stripping the crc %% Read next string where the string has a CRC prepended - stripping the crc
%% and checking if requested %% and checking if requested
@ -1578,11 +1615,13 @@ search_hash_table(Handle,
leveled_monitor:timing(), leveled_monitor:timing(),
leveled_monitor:timing(), leveled_monitor:timing(),
pos_integer()) -> ok. pos_integer()) -> ok.
maybelog_get_timing(_Monitor, no_timing, no_timing, _CC) -> maybelog_get_timing(
ok; {Pid, _StatsFreq}, IndexTime, ReadTime, CycleCount)
maybelog_get_timing({Pid, _StatsFreq}, IndexTime, ReadTime, CycleCount) -> when is_pid(Pid), is_integer(IndexTime), is_integer(ReadTime) ->
leveled_monitor:add_stat( leveled_monitor:add_stat(
Pid, {cdb_get_update, CycleCount, IndexTime, ReadTime}). Pid, {cdb_get_update, CycleCount, IndexTime, ReadTime});
maybelog_get_timing(_Monitor, _IndexTime, _ReadTime, _CC) ->
ok.
%% Write the actual hashtables at the bottom of the file. Each hash table %% Write the actual hashtables at the bottom of the file. Each hash table
@ -1916,7 +1955,7 @@ dump(FileName) ->
{ok, _} = file:position(Handle, {bof, ?BASE_POSITION}), {ok, _} = file:position(Handle, {bof, ?BASE_POSITION}),
Fn1 = fun(_I, Acc) -> Fn1 = fun(_I, Acc) ->
{KL, VL} = read_next_2_integers(Handle), {KL, VL} = read_next_2_integers(Handle),
{Key, KB} = safe_read_next_keybin(Handle, KL), {Key, KB} = safe_read_next(Handle, KL, keybin),
Value = Value =
case checkread_next_value(Handle, VL, KB) of case checkread_next_value(Handle, VL, KB) of
{true, V0} -> {true, V0} ->
@ -2632,7 +2671,7 @@ safe_read_test() ->
{ok, HandleK} = file:open(TestFN, ?WRITE_OPS), {ok, HandleK} = file:open(TestFN, ?WRITE_OPS),
ok = file:pwrite(HandleK, 0, BinToWrite), ok = file:pwrite(HandleK, 0, BinToWrite),
{ok, _} = file:position(HandleK, 8 + KeyL + ValueL), {ok, _} = file:position(HandleK, 8 + KeyL + ValueL),
?assertMatch(false, safe_read_next_key(HandleK, KeyL)), ?assertMatch(false, safe_read_next(HandleK, KeyL, key)),
ok = file:close(HandleK), ok = file:close(HandleK),
WrongKeyL = endian_flip(KeyL + ValueL), WrongKeyL = endian_flip(KeyL + ValueL),
@ -2749,11 +2788,15 @@ getpositions_sample_test() ->
ok = cdb_close(P2), ok = cdb_close(P2),
file:delete(F2). file:delete(F2).
nonsense_coverage_test() -> nonsense_coverage_test() ->
?assertMatch({ok, reader, #state{}}, code_change(nonsense, ?assertMatch(
{ok, reader, #state{}},
code_change(
nonsense,
reader, reader,
#state{}, #state{max_count=1, max_size=100},
nonsense)). nonsense
)
).
-endif. -endif.

255
src/leveled_codec.erl
View file

@ -10,6 +10,12 @@
-include("leveled.hrl"). -include("leveled.hrl").
-eqwalizer({nowarn_function, convert_to_ledgerv/5}).
-ifdef(TEST).
-export([convert_to_ledgerv/5]).
-endif.
-export([ -export([
inker_reload_strategy/1, inker_reload_strategy/1,
strip_to_seqonly/1, strip_to_seqonly/1,
@ -21,8 +27,10 @@
endkey_passed/2, endkey_passed/2,
key_dominates/2, key_dominates/2,
maybe_reap_expiredkey/2, maybe_reap_expiredkey/2,
to_ledgerkey/3, to_objectkey/3,
to_ledgerkey/5, to_objectkey/5,
to_querykey/3,
to_querykey/5,
from_ledgerkey/1, from_ledgerkey/1,
from_ledgerkey/2, from_ledgerkey/2,
isvalid_ledgerkey/1, isvalid_ledgerkey/1,
@ -33,11 +41,11 @@
from_journalkey/1, from_journalkey/1,
revert_to_keydeltas/2, revert_to_keydeltas/2,
is_full_journalentry/1, is_full_journalentry/1,
split_inkvalue/1,
check_forinkertype/2, check_forinkertype/2,
get_tagstrategy/2, get_tagstrategy/2,
maybe_compress/2, maybe_compress/2,
create_value_for_journal/3, create_value_for_journal/3,
revert_value_from_journal/1,
generate_ledgerkv/5, generate_ledgerkv/5,
get_size/2, get_size/2,
get_keyandobjhash/2, get_keyandobjhash/2,
@ -54,8 +62,9 @@
-type tag() :: -type tag() ::
leveled_head:object_tag()|?IDX_TAG|?HEAD_TAG|atom(). leveled_head:object_tag()|?IDX_TAG|?HEAD_TAG|atom().
-type key() :: -type single_key() :: binary().
binary()|string()|{binary(), binary()}. -type tuple_key() :: {single_key(), single_key()}.
-type key() :: single_key()|tuple_key().
% Keys SHOULD be binary() % Keys SHOULD be binary()
% string() support is a legacy of old tests % string() support is a legacy of old tests
-type sqn() :: -type sqn() ::
@ -75,8 +84,15 @@
-type ledger_status() :: -type ledger_status() ::
tomb|{active, non_neg_integer()|infinity}. tomb|{active, non_neg_integer()|infinity}.
-type primary_key() ::
{leveled_head:object_tag(), key(), single_key(), single_key()|null}.
% Primary key for an object
-type object_key() ::
{tag(), key(), key(), single_key()|null}.
-type query_key() ::
{tag(), key()|null, key()|null, single_key()|null}|all.
-type ledger_key() :: -type ledger_key() ::
{tag(), any(), any(), any()}|all. object_key()|query_key().
-type slimmed_key() :: -type slimmed_key() ::
{binary(), binary()|null}|binary()|null|all. {binary(), binary()|null}|binary()|null|all.
-type ledger_value() :: -type ledger_value() ::
@ -86,7 +102,7 @@
-type ledger_value_v2() :: -type ledger_value_v2() ::
{sqn(), ledger_status(), segment_hash(), metadata(), last_moddate()}. {sqn(), ledger_status(), segment_hash(), metadata(), last_moddate()}.
-type ledger_kv() :: -type ledger_kv() ::
{ledger_key(), ledger_value()}. {object_key(), ledger_value()}.
-type compaction_method() :: -type compaction_method() ::
retain|recovr|recalc. retain|recovr|recalc.
-type compaction_strategy() :: -type compaction_strategy() ::
@ -94,14 +110,14 @@
-type journal_key_tag() :: -type journal_key_tag() ::
?INKT_STND|?INKT_TOMB|?INKT_MPUT|?INKT_KEYD. ?INKT_STND|?INKT_TOMB|?INKT_MPUT|?INKT_KEYD.
-type journal_key() :: -type journal_key() ::
{sqn(), journal_key_tag(), ledger_key()}. {sqn(), journal_key_tag(), primary_key()}.
-type journal_ref() :: -type journal_ref() ::
{ledger_key(), sqn()}. {object_key(), sqn()}.
-type object_spec_v0() :: -type object_spec_v0() ::
{add|remove, key(), key(), key()|null, any()}. {add|remove, key(), single_key(), single_key()|null, metadata()}.
-type object_spec_v1() :: -type object_spec_v1() ::
{add|remove, v1, key(), key(), key()|null, {add|remove, v1, key(), single_key(), single_key()|null,
list(erlang:timestamp())|undefined, any()}. list(erlang:timestamp())|undefined, metadata()}.
-type object_spec() :: -type object_spec() ::
object_spec_v0()|object_spec_v1(). object_spec_v0()|object_spec_v1().
-type compression_method() :: -type compression_method() ::
@ -135,10 +151,14 @@
-export_type([tag/0, -export_type([tag/0,
key/0, key/0,
single_key/0,
sqn/0, sqn/0,
object_spec/0, object_spec/0,
segment_hash/0, segment_hash/0,
ledger_status/0, ledger_status/0,
primary_key/0,
object_key/0,
query_key/0,
ledger_key/0, ledger_key/0,
ledger_value/0, ledger_value/0,
ledger_kv/0, ledger_kv/0,
@ -186,30 +206,26 @@ segment_hash(KeyTuple) when is_tuple(KeyTuple) ->
segment_hash(BinKey). segment_hash(BinKey).
headkey_to_canonicalbinary({?HEAD_TAG, Bucket, Key, SubK}) headkey_to_canonicalbinary({
?HEAD_TAG, Bucket, Key, SubK})
when is_binary(Bucket), is_binary(Key), is_binary(SubK) -> when is_binary(Bucket), is_binary(Key), is_binary(SubK) ->
<<Bucket/binary, Key/binary, SubK/binary>>; <<Bucket/binary, Key/binary, SubK/binary>>;
headkey_to_canonicalbinary({?HEAD_TAG, Bucket, Key, null}) headkey_to_canonicalbinary(
{?HEAD_TAG, Bucket, Key, null})
when is_binary(Bucket), is_binary(Key) -> when is_binary(Bucket), is_binary(Key) ->
<<Bucket/binary, Key/binary>>; <<Bucket/binary, Key/binary>>;
headkey_to_canonicalbinary({?HEAD_TAG, {BucketType, Bucket}, Key, SubKey}) headkey_to_canonicalbinary(
{?HEAD_TAG, {BucketType, Bucket}, Key, SubKey})
when is_binary(BucketType), is_binary(Bucket) -> when is_binary(BucketType), is_binary(Bucket) ->
headkey_to_canonicalbinary({?HEAD_TAG, headkey_to_canonicalbinary(
<<BucketType/binary, Bucket/binary>>, {?HEAD_TAG, <<BucketType/binary, Bucket/binary>>, Key, SubKey}).
Key,
SubKey});
headkey_to_canonicalbinary(Key) when element(1, Key) == ?HEAD_TAG ->
% In unit tests head specs can have non-binary keys, so handle
% this through hashing the whole key
leveled_util:t2b(Key).
-spec to_lookup(ledger_key()) -> maybe_lookup(). -spec to_lookup(ledger_key()) -> maybe_lookup().
%% @doc %% @doc
%% Should it be possible to lookup a key in the merge tree. This is not true %% Should it be possible to lookup a key in the merge tree. This is not true
%% For keys that should only be read through range queries. Direct lookup %% For keys that should only be read through range queries. Direct lookup
%% keys will have presence in bloom filters and other lookup accelerators. %% keys will have presence in bloom filters and other lookup accelerators.
to_lookup(Key) -> to_lookup(Key) when is_tuple(Key) ->
case element(1, Key) of case element(1, Key) of
?IDX_TAG -> ?IDX_TAG ->
no_lookup; no_lookup;
@ -235,12 +251,12 @@ strip_to_keyseqonly({LK, V}) -> {LK, element(1, V)}.
-spec strip_to_indexdetails(ledger_kv()) -> -spec strip_to_indexdetails(ledger_kv()) ->
{integer(), segment_hash(), last_moddate()}. {integer(), segment_hash(), last_moddate()}.
strip_to_indexdetails({_, V}) when tuple_size(V) == 4 -> strip_to_indexdetails({_, {SQN, _, SegmentHash, _}}) ->
% A v1 value % A v1 value
{element(1, V), element(3, V), undefined}; {SQN, SegmentHash, undefined};
strip_to_indexdetails({_, V}) when tuple_size(V) > 4 -> strip_to_indexdetails({_, {SQN, _, SegmentHash, _, LMD}}) ->
% A v2 value should have a fith element - Last Modified Date % A v2 value should have a fith element - Last Modified Date
{element(1, V), element(3, V), element(5, V)}. {SQN, SegmentHash, LMD}.
-spec striphead_to_v1details(ledger_value()) -> ledger_value(). -spec striphead_to_v1details(ledger_value()) -> ledger_value().
striphead_to_v1details(V) -> striphead_to_v1details(V) ->
@ -292,18 +308,25 @@ maybe_accumulate(
maybe_accumulate(T, Acc, Count, Filter, AccFun). maybe_accumulate(T, Acc, Count, Filter, AccFun).
-spec accumulate_index( -spec accumulate_index(
{boolean(), undefined|leveled_runner:mp()}, leveled_runner:acc_fun()) {boolean(), undefined|leveled_runner:mp()},
-> any(). leveled_runner:fold_keys_fun())
-> leveled_penciller:pclacc_fun().
accumulate_index({false, undefined}, FoldKeysFun) -> accumulate_index({false, undefined}, FoldKeysFun) ->
fun({?IDX_TAG, Bucket, _IndexInfo, ObjKey}, _Value, Acc) -> fun(
{?IDX_TAG, Bucket, _IndexInfo, ObjKey}, _Value, Acc)
when ObjKey =/= null ->
FoldKeysFun(Bucket, ObjKey, Acc) FoldKeysFun(Bucket, ObjKey, Acc)
end; end;
accumulate_index({true, undefined}, FoldKeysFun) -> accumulate_index({true, undefined}, FoldKeysFun) ->
fun({?IDX_TAG, Bucket, {_IdxFld, IdxValue}, ObjKey}, _Value, Acc) -> fun(
{?IDX_TAG, Bucket, {_IdxFld, IdxValue}, ObjKey}, _Value, Acc)
when IdxValue =/= null, ObjKey =/= null ->
FoldKeysFun(Bucket, {IdxValue, ObjKey}, Acc) FoldKeysFun(Bucket, {IdxValue, ObjKey}, Acc)
end; end;
accumulate_index({AddTerm, TermRegex}, FoldKeysFun) -> accumulate_index({AddTerm, TermRegex}, FoldKeysFun) ->
fun({?IDX_TAG, Bucket, {_IdxFld, IdxValue}, ObjKey}, _Value, Acc) -> fun(
{?IDX_TAG, Bucket, {_IdxFld, IdxValue}, ObjKey}, _Value, Acc)
when IdxValue =/= null, ObjKey =/= null ->
case re:run(IdxValue, TermRegex) of case re:run(IdxValue, TermRegex) of
nomatch -> nomatch ->
Acc; Acc;
@ -343,17 +366,17 @@ maybe_reap(_, _) ->
false. false.
-spec count_tombs( -spec count_tombs(
list(ledger_kv()), non_neg_integer()|not_counted) -> list(ledger_kv()), non_neg_integer()) ->
non_neg_integer()|not_counted. non_neg_integer().
count_tombs(_List, not_counted) ->
not_counted;
count_tombs([], Count) -> count_tombs([], Count) ->
Count; Count;
count_tombs([{_K, V}|T], Count) when element(2, V) == tomb -> count_tombs([{_K, V}|T], Count) when is_tuple(V) ->
case element(2, V) of
tomb ->
count_tombs(T, Count + 1); count_tombs(T, Count + 1);
count_tombs([_KV|T], Count) -> _ ->
count_tombs(T, Count). count_tombs(T, Count)
end.
-spec from_ledgerkey(atom(), tuple()) -> false|tuple(). -spec from_ledgerkey(atom(), tuple()) -> false|tuple().
%% @doc %% @doc
@ -375,18 +398,37 @@ from_ledgerkey({?HEAD_TAG, Bucket, Key, SubKey}) ->
from_ledgerkey({_Tag, Bucket, Key, _SubKey}) -> from_ledgerkey({_Tag, Bucket, Key, _SubKey}) ->
{Bucket, Key}. {Bucket, Key}.
-spec to_ledgerkey(any(), any(), tag(), any(), any()) -> ledger_key(). -spec to_objectkey(
key(), single_key(), tag(), binary(), binary()) -> object_key().
%% @doc %% @doc
%% Convert something into a ledger key %% Convert something into a ledger key
to_ledgerkey(Bucket, Key, Tag, Field, Value) when Tag == ?IDX_TAG -> to_objectkey(Bucket, Key, Tag, Field, Value) when Tag == ?IDX_TAG ->
{?IDX_TAG, Bucket, {Field, Value}, Key}. {?IDX_TAG, Bucket, {Field, Value}, Key}.
-spec to_ledgerkey(any(), any(), tag()) -> ledger_key(). -if(?OTP_RELEASE >= 26).
-spec to_objectkey
(key(), single_key(), leveled_head:object_tag()) -> primary_key();
(key(), key(), tag()) -> object_key().
-else.
-spec to_objectkey(key(), key()|single_key(), tag()) -> object_key().
-endif.
%% @doc %% @doc
%% Convert something into a ledger key %% Convert something into a ledger key
to_ledgerkey(Bucket, {Key, SubKey}, ?HEAD_TAG) -> to_objectkey(Bucket, {Key, SubKey}, ?HEAD_TAG) ->
{?HEAD_TAG, Bucket, Key, SubKey}; {?HEAD_TAG, Bucket, Key, SubKey};
to_ledgerkey(Bucket, Key, Tag) -> to_objectkey(Bucket, Key, Tag) ->
{Tag, Bucket, Key, null}.
-spec to_querykey(
key(), single_key()|null, tag(), binary(), binary())
-> query_key().
to_querykey(Bucket, Key, Tag, Field, Value) when Tag == ?IDX_TAG ->
{?IDX_TAG, Bucket, {Field, Value}, Key}.
-spec to_querykey(key()|null, key()|null, tag()) -> query_key().
%% @doc
%% Convert something into a ledger query key
to_querykey(Bucket, Key, Tag) ->
{Tag, Bucket, Key, null}. {Tag, Bucket, Key, null}.
%% No spec - due to tests %% No spec - due to tests
@ -399,8 +441,8 @@ isvalid_ledgerkey(_LK) ->
false. false.
-spec endkey_passed( -spec endkey_passed(
ledger_key()|slimmed_key(), query_key()|slimmed_key(),
ledger_key()|slimmed_key()) -> boolean(). object_key()|slimmed_key()) -> boolean().
%% @doc %% @doc
%% Compare a key against a query key, only comparing elements that are non-null %% Compare a key against a query key, only comparing elements that are non-null
%% in the Query key. %% in the Query key.
@ -480,14 +522,19 @@ get_tagstrategy(Tag, Strategy) ->
%%% Manipulate Journal Key and Value %%% Manipulate Journal Key and Value
%%%============================================================================ %%%============================================================================
-spec to_inkerkey(ledger_key(), non_neg_integer()) -> journal_key(). -spec to_inkerkey(primary_key(), non_neg_integer()) -> journal_key().
%% @doc %% @doc
%% convertion from ledger_key to journal_key to allow for the key to be fetched %% convertion from ledger_key to journal_key to allow for the key to be fetched
to_inkerkey(LedgerKey, SQN) -> to_inkerkey(LedgerKey, SQN) ->
{SQN, ?INKT_STND, LedgerKey}. {SQN, ?INKT_STND, LedgerKey}.
-spec to_inkerkv(ledger_key(), non_neg_integer(), any(), journal_keychanges(), -spec to_inkerkv(
compression_method(), boolean()) -> {journal_key(), any()}. primary_key(),
non_neg_integer(),
any(),
journal_keychanges(),
compression_method(), boolean())
-> {journal_key(), binary()}.
%% @doc %% @doc
%% Convert to the correct format of a Journal key and value %% Convert to the correct format of a Journal key and value
to_inkerkv(LedgerKey, SQN, Object, KeyChanges, PressMethod, Compress) -> to_inkerkv(LedgerKey, SQN, Object, KeyChanges, PressMethod, Compress) ->
@ -496,7 +543,7 @@ to_inkerkv(LedgerKey, SQN, Object, KeyChanges, PressMethod, Compress) ->
create_value_for_journal({Object, KeyChanges}, Compress, PressMethod), create_value_for_journal({Object, KeyChanges}, Compress, PressMethod),
{{SQN, InkerType, LedgerKey}, Value}. {{SQN, InkerType, LedgerKey}, Value}.
-spec revert_to_keydeltas(journal_key(), any()) -> {journal_key(), any()}. -spec revert_to_keydeltas(journal_key(), binary()) -> {journal_key(), any()}.
%% @doc %% @doc
%% If we wish to retain key deltas when an object in the Journal has been %% If we wish to retain key deltas when an object in the Journal has been
%% replaced - then this converts a Journal Key and Value into one which has no %% replaced - then this converts a Journal Key and Value into one which has no
@ -575,7 +622,7 @@ serialise_object(Object, false, _Method) ->
serialise_object(Object, true, _Method) -> serialise_object(Object, true, _Method) ->
term_to_binary(Object, [compressed]). term_to_binary(Object, [compressed]).
-spec revert_value_from_journal(binary()) -> {any(), journal_keychanges()}. -spec revert_value_from_journal(binary()) -> {dynamic(), journal_keychanges()}.
%% @doc %% @doc
%% Revert the object back to its deserialised state, along with the list of %% Revert the object back to its deserialised state, along with the list of
%% key changes associated with the change %% key changes associated with the change
@ -661,10 +708,6 @@ decode_valuetype(TypeInt) ->
from_journalkey({SQN, _Type, LedgerKey}) -> from_journalkey({SQN, _Type, LedgerKey}) ->
{SQN, LedgerKey}. {SQN, LedgerKey}.
split_inkvalue(VBin) when is_binary(VBin) ->
revert_value_from_journal(VBin).
check_forinkertype(_LedgerKey, delete) -> check_forinkertype(_LedgerKey, delete) ->
?INKT_TOMB; ?INKT_TOMB;
check_forinkertype(_LedgerKey, head_only) -> check_forinkertype(_LedgerKey, head_only) ->
@ -709,7 +752,7 @@ idx_indexspecs(IndexSpecs, Bucket, Key, SQN, TTL) ->
gen_indexspec(Bucket, Key, IdxOp, IdxField, IdxTerm, SQN, TTL) -> gen_indexspec(Bucket, Key, IdxOp, IdxField, IdxTerm, SQN, TTL) ->
Status = set_status(IdxOp, TTL), Status = set_status(IdxOp, TTL),
{to_ledgerkey(Bucket, Key, ?IDX_TAG, IdxField, IdxTerm), {to_objectkey(Bucket, Key, ?IDX_TAG, IdxField, IdxTerm),
{SQN, Status, no_lookup, null}}. {SQN, Status, no_lookup, null}}.
-spec gen_headspec(object_spec(), integer(), integer()|infinity) -> ledger_kv(). -spec gen_headspec(object_spec(), integer(), integer()|infinity) -> ledger_kv().
@ -717,22 +760,30 @@ gen_indexspec(Bucket, Key, IdxOp, IdxField, IdxTerm, SQN, TTL) ->
%% Take an object_spec as passed in a book_mput, and convert it into to a %% Take an object_spec as passed in a book_mput, and convert it into to a
%% valid ledger key and value. Supports different shaped tuples for different %% valid ledger key and value. Supports different shaped tuples for different
%% versions of the object_spec %% versions of the object_spec
gen_headspec({IdxOp, v1, Bucket, Key, SubKey, LMD, Value}, SQN, TTL) -> gen_headspec(
{IdxOp, v1, Bucket, Key, SubKey, LMD, Value}, SQN, TTL)
when is_binary(Key) ->
% v1 object spec % v1 object spec
Status = set_status(IdxOp, TTL), Status = set_status(IdxOp, TTL),
K = to_ledgerkey(Bucket, {Key, SubKey}, ?HEAD_TAG), K =
case SubKey of
null ->
to_objectkey(Bucket, Key, ?HEAD_TAG);
SKB when is_binary(SKB) ->
to_objectkey(Bucket, {Key, SKB}, ?HEAD_TAG)
end,
{K, {SQN, Status, segment_hash(K), Value, get_last_lastmodification(LMD)}}; {K, {SQN, Status, segment_hash(K), Value, get_last_lastmodification(LMD)}};
gen_headspec({IdxOp, Bucket, Key, SubKey, Value}, SQN, TTL) -> gen_headspec(
% v0 object spec {IdxOp, Bucket, Key, SubKey, Value}, SQN, TTL)
Status = set_status(IdxOp, TTL), when is_binary(Key) ->
K = to_ledgerkey(Bucket, {Key, SubKey}, ?HEAD_TAG), gen_headspec({IdxOp, v1, Bucket, Key, SubKey, undefined, Value}, SQN, TTL).
{K, {SQN, Status, segment_hash(K), Value, undefined}}.
-spec return_proxy(leveled_head:object_tag()|leveled_head:headonly_tag(), -spec return_proxy
leveled_head:object_metadata(), (leveled_head:headonly_tag(), leveled_head:object_metadata(), null, journal_ref())
pid(), journal_ref()) -> leveled_head:object_metadata();
-> proxy_objectbin()|leveled_head:object_metadata(). (leveled_head:object_tag(), leveled_head:object_metadata(), pid(), journal_ref())
-> proxy_objectbin().
%% @doc %% @doc
%% If the object has a value, return the metadata and a proxy through which %% If the object has a value, return the metadata and a proxy through which
%% the applictaion or runner can access the value. If it is a ?HEAD_TAG %% the applictaion or runner can access the value. If it is a ?HEAD_TAG
@ -751,6 +802,9 @@ return_proxy(Tag, ObjMetadata, InkerClone, JournalRef) ->
InkerClone, InkerClone,
JournalRef}}). JournalRef}}).
-spec set_status(
add|remove, non_neg_integer()|infinity) ->
tomb|{active, non_neg_integer()|infinity}.
set_status(add, TTL) -> set_status(add, TTL) ->
{active, TTL}; {active, TTL};
set_status(remove, _TTL) -> set_status(remove, _TTL) ->
@ -758,10 +812,18 @@ set_status(remove, _TTL) ->
tomb. tomb.
-spec generate_ledgerkv( -spec generate_ledgerkv(
tuple(), integer(), any(), integer(), tuple()|infinity) -> primary_key(),
{any(), any(), any(), integer(),
{{integer(), integer()}|no_lookup, integer()}, dynamic(),
list()}. integer(),
non_neg_integer()|infinity) ->
{
key(),
single_key(),
ledger_value_v2(),
{segment_hash(), non_neg_integer()|null},
list(erlang:timestamp())
}.
%% @doc %% @doc
%% Function to extract from an object the information necessary to populate %% Function to extract from an object the information necessary to populate
%% the Penciller's ledger. %% the Penciller's ledger.
@ -776,24 +838,22 @@ set_status(remove, _TTL) ->
%% siblings) %% siblings)
generate_ledgerkv(PrimaryKey, SQN, Obj, Size, TS) -> generate_ledgerkv(PrimaryKey, SQN, Obj, Size, TS) ->
{Tag, Bucket, Key, _} = PrimaryKey, {Tag, Bucket, Key, _} = PrimaryKey,
Status = case Obj of Status = case Obj of delete -> tomb; _ -> {active, TS} end,
delete ->
tomb;
_ ->
{active, TS}
end,
Hash = segment_hash(PrimaryKey), Hash = segment_hash(PrimaryKey),
{MD, LastMods} = leveled_head:extract_metadata(Tag, Size, Obj), {MD, LastMods} = leveled_head:extract_metadata(Tag, Size, Obj),
ObjHash = leveled_head:get_hash(Tag, MD), ObjHash = leveled_head:get_hash(Tag, MD),
Value = {SQN, Value =
{
SQN,
Status, Status,
Hash, Hash,
MD, MD,
get_last_lastmodification(LastMods)}, get_last_lastmodification(LastMods)
},
{Bucket, Key, Value, {Hash, ObjHash}, LastMods}. {Bucket, Key, Value, {Hash, ObjHash}, LastMods}.
-spec get_last_lastmodification(list(erlang:timestamp())|undefined) -spec get_last_lastmodification(
-> pos_integer()|undefined. list(erlang:timestamp())|undefined) -> pos_integer()|undefined.
%% @doc %% @doc
%% Get the highest of the last modifications measured in seconds. This will be %% Get the highest of the last modifications measured in seconds. This will be
%% stored as 4 bytes (unsigned) so will last for another 80 + years %% stored as 4 bytes (unsigned) so will last for another 80 + years
@ -830,10 +890,10 @@ get_keyandobjhash(LK, Value) ->
%% Get the next key to iterate from a given point %% Get the next key to iterate from a given point
next_key(Key) when is_binary(Key) -> next_key(Key) when is_binary(Key) ->
<<Key/binary, 0>>; <<Key/binary, 0>>;
next_key(Key) when is_list(Key) ->
Key ++ [0];
next_key({Type, Bucket}) when is_binary(Type), is_binary(Bucket) -> next_key({Type, Bucket}) when is_binary(Type), is_binary(Bucket) ->
{Type, next_key(Bucket)}. UpdBucket = next_key(Bucket),
true = is_binary(UpdBucket),
{Type, UpdBucket}.
%%%============================================================================ %%%============================================================================
@ -844,6 +904,17 @@ next_key({Type, Bucket}) when is_binary(Type), is_binary(Bucket) ->
-include_lib("eunit/include/eunit.hrl"). -include_lib("eunit/include/eunit.hrl").
-spec convert_to_ledgerv(
leveled_codec:ledger_key(),
integer(),
any(),
integer(),
non_neg_integer()|infinity) -> leveled_codec:ledger_value().
convert_to_ledgerv(PK, SQN, Obj, Size, TS) ->
{_B, _K, MV, _H, _LMs} =
leveled_codec:generate_ledgerkv(PK, SQN, Obj, Size, TS),
MV.
valid_ledgerkey_test() -> valid_ledgerkey_test() ->
UserDefTag = {user_defined, <<"B">>, <<"K">>, null}, UserDefTag = {user_defined, <<"B">>, <<"K">>, null},
?assertMatch(true, isvalid_ledgerkey(UserDefTag)), ?assertMatch(true, isvalid_ledgerkey(UserDefTag)),
@ -870,8 +941,8 @@ indexspecs_test() ->
endkey_passed_test() -> endkey_passed_test() ->
TestKey = {i, null, null, null}, TestKey = {i, null, null, null},
K1 = {i, 123, {"a", "b"}, <<>>}, K1 = {i, <<"123">>, {<<"a">>, <<"b">>}, <<>>},
K2 = {o, 123, {"a", "b"}, <<>>}, K2 = {o, <<"123">>, {<<"a">>, <<"b">>}, <<>>},
?assertMatch(false, endkey_passed(TestKey, K1)), ?assertMatch(false, endkey_passed(TestKey, K1)),
?assertMatch(true, endkey_passed(TestKey, K2)). ?assertMatch(true, endkey_passed(TestKey, K2)).
@ -881,7 +952,7 @@ endkey_passed_test() ->
%% Maybe 5 microseconds per hash %% Maybe 5 microseconds per hash
hashperf_test() -> hashperf_test() ->
OL = lists:map(fun(_X) -> leveled_rand:rand_bytes(8192) end, lists:seq(1, 1000)), OL = lists:map(fun(_X) -> crypto:strong_rand_bytes(8192) end, lists:seq(1, 1000)),
SW = os:timestamp(), SW = os:timestamp(),
_HL = lists:map(fun(Obj) -> erlang:phash2(Obj) end, OL), _HL = lists:map(fun(Obj) -> erlang:phash2(Obj) end, OL),
io:format(user, "1000 object hashes in ~w microseconds~n", io:format(user, "1000 object hashes in ~w microseconds~n",
@ -899,8 +970,8 @@ head_segment_compare_test() ->
headspec_v0v1_test() -> headspec_v0v1_test() ->
% A v0 object spec generates the same outcome as a v1 object spec with the % A v0 object spec generates the same outcome as a v1 object spec with the
% last modified date undefined % last modified date undefined
V1 = {add, v1, <<"B">>, <<"K">>, <<"SK">>, undefined, <<"V">>}, V1 = {add, v1, <<"B">>, <<"K">>, <<"SK">>, undefined, {<<"V">>}},
V0 = {add, <<"B">>, <<"K">>, <<"SK">>, <<"V">>}, V0 = {add, <<"B">>, <<"K">>, <<"SK">>, {<<"V">>}},
TTL = infinity, TTL = infinity,
?assertMatch(true, gen_headspec(V0, 1, TTL) == gen_headspec(V1, 1, TTL)). ?assertMatch(true, gen_headspec(V0, 1, TTL) == gen_headspec(V1, 1, TTL)).

20
src/leveled_ebloom.erl
View file

@ -91,8 +91,8 @@ check_hash({_SegHash, Hash}, BloomBin) when is_binary(BloomBin)->
list(leveled_codec:segment_hash()), tuple(), slot_count()) -> tuple(). list(leveled_codec:segment_hash()), tuple(), slot_count()) -> tuple().
map_hashes([], HashListTuple, _SlotCount) -> map_hashes([], HashListTuple, _SlotCount) ->
HashListTuple; HashListTuple;
map_hashes([Hash|Rest], HashListTuple, SlotCount) -> map_hashes([{_SH, EH}|Rest], HashListTuple, SlotCount) ->
{Slot, [H0, H1]} = split_hash(element(2, Hash), SlotCount), {Slot, [H0, H1]} = split_hash(EH, SlotCount),
SlotHL = element(Slot + 1, HashListTuple), SlotHL = element(Slot + 1, HashListTuple),
map_hashes( map_hashes(
Rest, Rest,
@ -174,11 +174,15 @@ generate_orderedkeys(Seqn, Count, Acc, BucketLow, BucketHigh) ->
BucketExt = BucketExt =
io_lib:format("K~4..0B", [BucketLow + BNumber]), io_lib:format("K~4..0B", [BucketLow + BNumber]),
KeyExt = KeyExt =
io_lib:format("K~8..0B", [Seqn * 100 + leveled_rand:uniform(100)]), io_lib:format("K~8..0B", [Seqn * 100 + rand:uniform(100)]),
LK = leveled_codec:to_ledgerkey("Bucket" ++ BucketExt, "Key" ++ KeyExt, o), LK =
Chunk = leveled_rand:rand_bytes(16), leveled_codec:to_objectkey(
{_B, _K, MV, _H, _LMs} = list_to_binary("Bucket" ++ BucketExt),
leveled_codec:generate_ledgerkv(LK, Seqn, Chunk, 64, infinity), list_to_binary("Key" ++ KeyExt),
o
),
Chunk = crypto:strong_rand_bytes(16),
MV = leveled_codec:convert_to_ledgerv(LK, Seqn, Chunk, 64, infinity),
generate_orderedkeys( generate_orderedkeys(
Seqn + 1, Count - 1, [{LK, MV}|Acc], BucketLow, BucketHigh). Seqn + 1, Count - 1, [{LK, MV}|Acc], BucketLow, BucketHigh).
@ -236,7 +240,7 @@ test_bloom(N, Runs) ->
fun(HashList) -> fun(HashList) ->
HitOrMissFun = HitOrMissFun =
fun (Entry, {HitL, MissL}) -> fun (Entry, {HitL, MissL}) ->
case leveled_rand:uniform() < 0.5 of case rand:uniform() < 0.5 of
true -> true ->
{[Entry|HitL], MissL}; {[Entry|HitL], MissL};
false -> false ->

62
src/leveled_head.erl
View file

@ -82,7 +82,7 @@
-type appdefinable_headfun() :: -type appdefinable_headfun() ::
fun((object_tag(), object_metadata()) -> head()). fun((object_tag(), object_metadata()) -> head()).
-type appdefinable_metadatafun() :: -type appdefinable_metadatafun() ::
fun(({leveled_codec:tag(), non_neg_integer(), any()}) -> fun((leveled_codec:tag(), non_neg_integer(), binary()|delete) ->
{object_metadata(), list(erlang:timestamp())}). {object_metadata(), list(erlang:timestamp())}).
-type appdefinable_indexspecsfun() :: -type appdefinable_indexspecsfun() ::
fun((object_tag(), object_metadata(), object_metadata()|not_present) -> fun((object_tag(), object_metadata(), object_metadata()|not_present) ->
@ -117,10 +117,12 @@
%% @doc %% @doc
%% Convert a key to a binary in a consistent way for the tag. The binary will %% Convert a key to a binary in a consistent way for the tag. The binary will
%% then be used to create the hash %% then be used to create the hash
key_to_canonicalbinary({?RIAK_TAG, Bucket, Key, null}) key_to_canonicalbinary(
{?RIAK_TAG, Bucket, Key, null})
when is_binary(Bucket), is_binary(Key) -> when is_binary(Bucket), is_binary(Key) ->
<<Bucket/binary, Key/binary>>; <<Bucket/binary, Key/binary>>;
key_to_canonicalbinary({?RIAK_TAG, {BucketType, Bucket}, Key, SubKey}) key_to_canonicalbinary(
{?RIAK_TAG, {BucketType, Bucket}, Key, SubKey})
when is_binary(BucketType), is_binary(Bucket) -> when is_binary(BucketType), is_binary(Bucket) ->
key_to_canonicalbinary({?RIAK_TAG, key_to_canonicalbinary({?RIAK_TAG,
<<BucketType/binary, Bucket/binary>>, <<BucketType/binary, Bucket/binary>>,
@ -130,9 +132,11 @@ key_to_canonicalbinary(Key) when element(1, Key) == ?STD_TAG ->
default_key_to_canonicalbinary(Key); default_key_to_canonicalbinary(Key);
key_to_canonicalbinary(Key) -> key_to_canonicalbinary(Key) ->
OverrideFun = OverrideFun =
get_appdefined_function(key_to_canonicalbinary, get_appdefined_function(
key_to_canonicalbinary,
fun default_key_to_canonicalbinary/1, fun default_key_to_canonicalbinary/1,
1), 1
),
OverrideFun(Key). OverrideFun(Key).
default_key_to_canonicalbinary(Key) -> default_key_to_canonicalbinary(Key) ->
@ -162,7 +166,7 @@ default_build_head(_Tag, Metadata) ->
Metadata. Metadata.
-spec extract_metadata(object_tag(), non_neg_integer(), any()) -spec extract_metadata(object_tag(), non_neg_integer(), binary())
-> {object_metadata(), list(erlang:timestamp())}. -> {object_metadata(), list(erlang:timestamp())}.
%% @doc %% @doc
%% Take the inbound object and extract from it the metadata to be stored within %% Take the inbound object and extract from it the metadata to be stored within
@ -239,9 +243,8 @@ defined_objecttags() ->
[?STD_TAG, ?RIAK_TAG]. [?STD_TAG, ?RIAK_TAG].
-spec default_reload_strategy(object_tag()) -spec default_reload_strategy(
-> {object_tag(), object_tag()) -> {object_tag(), leveled_codec:compaction_method()}.
leveled_codec:compaction_method()}.
%% @doc %% @doc
%% State the compaction_method to be used when reloading the Ledger from the %% State the compaction_method to be used when reloading the Ledger from the
%% journal for each object tag. Note, no compaction strategy required for %% journal for each object tag. Note, no compaction strategy required for
@ -249,25 +252,24 @@ defined_objecttags() ->
default_reload_strategy(Tag) -> default_reload_strategy(Tag) ->
{Tag, retain}. {Tag, retain}.
-spec get_size(
-spec get_size(object_tag()|headonly_tag(), object_metadata()) object_tag()|headonly_tag(), object_metadata()) -> non_neg_integer().
-> non_neg_integer().
%% @doc %% @doc
%% Fetch the size from the metadata %% Fetch the size from the metadata
get_size(?RIAK_TAG, RiakObjectMetadata) -> get_size(?RIAK_TAG, {_, _, _, Size}) ->
element(4, RiakObjectMetadata); Size;
get_size(_Tag, ObjectMetadata) -> get_size(_Tag, {_, Size, _}) ->
element(2, ObjectMetadata). Size.
-spec get_hash(object_tag()|headonly_tag(), object_metadata()) -spec get_hash(
-> non_neg_integer(). object_tag()|headonly_tag(), object_metadata()) -> non_neg_integer()|null.
%% @doc %% @doc
%% Fetch the hash from the metadata %% Fetch the hash from the metadata
get_hash(?RIAK_TAG, RiakObjectMetadata) -> get_hash(?RIAK_TAG, {_, _, Hash, _}) ->
element(3, RiakObjectMetadata); Hash;
get_hash(_Tag, ObjectMetadata) -> get_hash(_Tag, {Hash, _, _}) ->
element(1, ObjectMetadata). Hash.
-spec standard_hash(any()) -> non_neg_integer(). -spec standard_hash(any()) -> non_neg_integer().
%% @doc %% @doc
@ -280,9 +282,15 @@ standard_hash(Obj) ->
%%% Handling Override Functions %%% Handling Override Functions
%%%============================================================================ %%%============================================================================
-spec get_appdefined_function( -spec get_appdefined_function
appdefinable_function(), appdefinable_function_fun(), non_neg_integer()) -> (key_to_canonicalbinary, appdefinable_keyfun(), 1) ->
appdefinable_function_fun(). appdefinable_keyfun();
(build_head, appdefinable_headfun(), 2) ->
appdefinable_headfun();
(extract_metadata, appdefinable_metadatafun(), 3) ->
appdefinable_metadatafun();
(diff_indexspecs, appdefinable_indexspecsfun(), 3) ->
appdefinable_indexspecsfun().
%% @doc %% @doc
%% If a keylist of [{function_name, fun()}] has been set as an environment %% If a keylist of [{function_name, fun()}] has been set as an environment
%% variable for a tag, then this FunctionName can be used instead of the %% variable for a tag, then this FunctionName can be used instead of the
@ -300,8 +308,8 @@ get_appdefined_function(FunctionName, DefaultFun, RequiredArity) ->
%%%============================================================================ %%%============================================================================
-spec riak_extract_metadata(binary()|delete, non_neg_integer()) -> -spec riak_extract_metadata(
{riak_metadata(), list()}. binary()|delete, non_neg_integer()) -> {riak_metadata(), list()}.
%% @doc %% @doc
%% Riak extract metadata should extract a metadata object which is a %% Riak extract metadata should extract a metadata object which is a
%% five-tuple of: %% five-tuple of:

317
src/leveled_iclerk.erl
View file

@ -114,10 +114,10 @@
scoring_state :: scoring_state()|undefined, scoring_state :: scoring_state()|undefined,
score_onein = 1 :: pos_integer()}). score_onein = 1 :: pos_integer()}).
-record(candidate, {low_sqn :: integer() | undefined, -record(candidate, {low_sqn :: integer(),
filename :: string() | undefined, filename :: string(),
journal :: pid() | undefined, journal :: pid(),
compaction_perc :: float() | undefined}). compaction_perc :: float()}).
-record(scoring_state, {filter_fun :: leveled_inker:filterfun(), -record(scoring_state, {filter_fun :: leveled_inker:filterfun(),
filter_server :: leveled_inker:filterserver(), filter_server :: leveled_inker:filterserver(),
@ -158,7 +158,13 @@
%% @doc %% @doc
%% Generate a new clerk %% Generate a new clerk
clerk_new(InkerClerkOpts) -> clerk_new(InkerClerkOpts) ->
gen_server:start_link(?MODULE, [leveled_log:get_opts(), InkerClerkOpts], []). {ok, Clerk} =
gen_server:start_link(
?MODULE,
[leveled_log:get_opts(), InkerClerkOpts],
[]
),
{ok, Clerk}.
-spec clerk_compact(pid(), -spec clerk_compact(pid(),
pid(), pid(),
@ -310,60 +316,71 @@ handle_cast({compact, Checker, InitiateFun, CloseFun, FilterFun, Manifest0},
end, end,
ok = clerk_scorefilelist(self(), lists:filter(NotRollingFun, Manifest)), ok = clerk_scorefilelist(self(), lists:filter(NotRollingFun, Manifest)),
ScoringState = ScoringState =
#scoring_state{filter_fun = FilterFun, #scoring_state{
filter_fun = FilterFun,
filter_server = FilterServer, filter_server = FilterServer,
max_sqn = MaxSQN, max_sqn = MaxSQN,
close_fun = CloseFun, close_fun = CloseFun,
start_time = SW}, start_time = SW
},
{noreply, State#state{scored_files = [], scoring_state = ScoringState}}; {noreply, State#state{scored_files = [], scoring_state = ScoringState}};
handle_cast({score_filelist, [Entry|Tail]}, State) -> handle_cast(
{score_filelist, [Entry|Tail]},
State = #state{scoring_state = ScoringState})
when ?IS_DEF(ScoringState) ->
Candidates = State#state.scored_files, Candidates = State#state.scored_files,
{LowSQN, FN, JournalP, _LK} = Entry, {LowSQN, FN, JournalP, _LK} = Entry,
ScoringState = State#state.scoring_state,
CpctPerc = CpctPerc =
case {leveled_cdb:cdb_getcachedscore(JournalP, os:timestamp()), case {leveled_cdb:cdb_getcachedscore(JournalP, os:timestamp()),
leveled_rand:uniform(State#state.score_onein) == 1, rand:uniform(State#state.score_onein) == 1,
State#state.score_onein} of State#state.score_onein} of
{CachedScore, _UseNewScore, ScoreOneIn} {CachedScore, _UseNewScore, ScoreOneIn}
when CachedScore == undefined; ScoreOneIn == 1 -> when CachedScore == undefined; ScoreOneIn == 1 ->
% If caches are not used, always use the current score % If caches are not used, always use the current score
check_single_file(JournalP, check_single_file(
JournalP,
ScoringState#scoring_state.filter_fun, ScoringState#scoring_state.filter_fun,
ScoringState#scoring_state.filter_server, ScoringState#scoring_state.filter_server,
ScoringState#scoring_state.max_sqn, ScoringState#scoring_state.max_sqn,
?SAMPLE_SIZE, ?SAMPLE_SIZE,
?BATCH_SIZE, ?BATCH_SIZE,
State#state.reload_strategy); State#state.reload_strategy
);
{CachedScore, true, _ScoreOneIn} -> {CachedScore, true, _ScoreOneIn} ->
% If caches are used roll the score towards the current score % If caches are used roll the score towards the current score
% Expectation is that this will reduce instances of individual % Expectation is that this will reduce instances of individual
% files being compacted when a run is missed due to cached % files being compacted when a run is missed due to cached
% scores being used in surrounding journals % scores being used in surrounding journals
NewScore = NewScore =
check_single_file(JournalP, check_single_file(
JournalP,
ScoringState#scoring_state.filter_fun, ScoringState#scoring_state.filter_fun,
ScoringState#scoring_state.filter_server, ScoringState#scoring_state.filter_server,
ScoringState#scoring_state.max_sqn, ScoringState#scoring_state.max_sqn,
?SAMPLE_SIZE, ?SAMPLE_SIZE,
?BATCH_SIZE, ?BATCH_SIZE,
State#state.reload_strategy), State#state.reload_strategy
),
(NewScore + CachedScore) / 2; (NewScore + CachedScore) / 2;
{CachedScore, false, _ScoreOneIn} -> {CachedScore, false, _ScoreOneIn} ->
CachedScore CachedScore
end, end,
ok = leveled_cdb:cdb_putcachedscore(JournalP, CpctPerc), ok = leveled_cdb:cdb_putcachedscore(JournalP, CpctPerc),
Candidate = Candidate =
#candidate{low_sqn = LowSQN, #candidate{
low_sqn = LowSQN,
filename = FN, filename = FN,
journal = JournalP, journal = JournalP,
compaction_perc = CpctPerc}, compaction_perc = CpctPerc
},
ok = clerk_scorefilelist(self(), Tail), ok = clerk_scorefilelist(self(), Tail),
{noreply, State#state{scored_files = [Candidate|Candidates]}}; {noreply, State#state{scored_files = [Candidate|Candidates]}};
handle_cast(scoring_complete, State) -> handle_cast(
scoring_complete, State = #state{scoring_state = ScoringState})
when ?IS_DEF(ScoringState) ->
MaxRunLength = State#state.max_run_length, MaxRunLength = State#state.max_run_length,
CDBopts = State#state.cdb_options, CDBopts = State#state.cdb_options,
Candidates = lists:reverse(State#state.scored_files), Candidates = lists:reverse(State#state.scored_files),
ScoringState = State#state.scoring_state,
FilterFun = ScoringState#scoring_state.filter_fun, FilterFun = ScoringState#scoring_state.filter_fun,
FilterServer = ScoringState#scoring_state.filter_server, FilterServer = ScoringState#scoring_state.filter_server,
MaxSQN = ScoringState#scoring_state.max_sqn, MaxSQN = ScoringState#scoring_state.max_sqn,
@ -379,35 +396,45 @@ handle_cast(scoring_complete, State) ->
true -> true ->
BestRun1 = sort_run(BestRun0), BestRun1 = sort_run(BestRun0),
print_compaction_run(BestRun1, ScoreParams), print_compaction_run(BestRun1, ScoreParams),
ManifestSlice = compact_files(BestRun1, ManifestSlice =
compact_files(
BestRun1,
CDBopts, CDBopts,
FilterFun, FilterFun,
FilterServer, FilterServer,
MaxSQN, MaxSQN,
State#state.reload_strategy, State#state.reload_strategy,
State#state.compression_method), State#state.compression_method
FilesToDelete = lists:map(fun(C) -> ),
{C#candidate.low_sqn, FilesToDelete =
lists:map(
fun(C) ->
{
C#candidate.low_sqn,
C#candidate.filename, C#candidate.filename,
C#candidate.journal, C#candidate.journal,
undefined} undefined
}
end, end,
BestRun1), BestRun1
),
leveled_log:log(ic002, [length(FilesToDelete)]), leveled_log:log(ic002, [length(FilesToDelete)]),
ok = CloseFun(FilterServer), ok = CloseFun(FilterServer),
ok = leveled_inker:ink_clerkcomplete(State#state.inker, ok =
ManifestSlice, leveled_inker:ink_clerkcomplete(
FilesToDelete); State#state.inker, ManifestSlice, FilesToDelete);
false -> false ->
ok = CloseFun(FilterServer), ok = CloseFun(FilterServer),
ok = leveled_inker:ink_clerkcomplete(State#state.inker, [], []) ok = leveled_inker:ink_clerkcomplete(State#state.inker, [], [])
end, end,
{noreply, State#state{scoring_state = undefined}, hibernate}; {noreply, State#state{scoring_state = undefined}, hibernate};
handle_cast({trim, PersistedSQN, ManifestAsList}, State) -> handle_cast(
{trim, PersistedSQN, ManifestAsList}, State = #state{inker = Ink})
when ?IS_DEF(Ink) ->
FilesToDelete = FilesToDelete =
leveled_imanifest:find_persistedentries(PersistedSQN, ManifestAsList), leveled_imanifest:find_persistedentries(PersistedSQN, ManifestAsList),
leveled_log:log(ic007, []), leveled_log:log(ic007, []),
ok = leveled_inker:ink_clerkcomplete(State#state.inker, [], FilesToDelete), ok = leveled_inker:ink_clerkcomplete(Ink, [], FilesToDelete),
{noreply, State}; {noreply, State};
handle_cast({hashtable_calc, HashTree, StartPos, CDBpid}, State) -> handle_cast({hashtable_calc, HashTree, StartPos, CDBpid}, State) ->
{IndexList, HashTreeBin} = leveled_cdb:hashtable_calc(HashTree, StartPos), {IndexList, HashTreeBin} = leveled_cdb:hashtable_calc(HashTree, StartPos),
@ -445,9 +472,9 @@ code_change(_OldVsn, State, _Extra) ->
%%% External functions %%% External functions
%%%============================================================================ %%%============================================================================
-spec schedule_compaction(list(integer()), -spec schedule_compaction(
integer(), list(integer()), integer(), {integer(), integer(), integer()}) ->
{integer(), integer(), integer()}) -> integer(). integer().
%% @doc %% @doc
%% Schedule the next compaction event for this store. Chooses a random %% Schedule the next compaction event for this store. Chooses a random
%% interval, and then a random start time within the first third %% interval, and then a random start time within the first third
@ -483,11 +510,11 @@ schedule_compaction(CompactionHours, RunsPerDay, CurrentTS) ->
% today. % today.
RandSelect = RandSelect =
fun(_X) -> fun(_X) ->
{lists:nth(leveled_rand:uniform(TotalHours), CompactionHours), {lists:nth(rand:uniform(TotalHours), CompactionHours),
leveled_rand:uniform(?INTERVALS_PER_HOUR)} rand:uniform(?INTERVALS_PER_HOUR)}
end, end,
RandIntervals = lists:sort(lists:map(RandSelect, RandIntervals =
lists:seq(1, RunsPerDay))), lists:sort(lists:map(RandSelect, lists:seq(1, RunsPerDay))),
% Pick the next interval from the list. The intervals before current time % Pick the next interval from the list. The intervals before current time
% are considered as intervals tomorrow, so will only be next if there are % are considered as intervals tomorrow, so will only be next if there are
@ -508,11 +535,11 @@ schedule_compaction(CompactionHours, RunsPerDay, CurrentTS) ->
% Calculate the offset in seconds to this next interval % Calculate the offset in seconds to this next interval
NextS0 = NextI * (IntervalLength * 60) NextS0 = NextI * (IntervalLength * 60)
- leveled_rand:uniform(IntervalLength * 60), - rand:uniform(IntervalLength * 60),
NextM = NextS0 div 60, NextM = NextS0 div 60,
NextS = NextS0 rem 60, NextS = NextS0 rem 60,
TimeDiff = calendar:time_difference(LocalTime, TimeDiff =
{NextDate, {NextH, NextM, NextS}}), calendar:time_difference(LocalTime, {NextDate, {NextH, NextM, NextS}}),
{Days, {Hours, Mins, Secs}} = TimeDiff, {Days, {Hours, Mins, Secs}} = TimeDiff,
Days * 86400 + Hours * 3600 + Mins * 60 + Secs. Days * 86400 + Hours * 3600 + Mins * 60 + Secs.
@ -521,7 +548,8 @@ schedule_compaction(CompactionHours, RunsPerDay, CurrentTS) ->
%%% Internal functions %%% Internal functions
%%%============================================================================ %%%============================================================================
-spec check_single_file(pid(), -spec check_single_file(
pid(),
leveled_inker:filterfun(), leveled_inker:filterfun(),
leveled_inker:filterserver(), leveled_inker:filterserver(),
leveled_codec:sqn(), leveled_codec:sqn(),
@ -624,8 +652,8 @@ fetch_inbatches(PositionList, BatchSize, CDB, CheckedList) ->
fetch_inbatches(Tail, BatchSize, CDB, CheckedList ++ KL_List). fetch_inbatches(Tail, BatchSize, CDB, CheckedList ++ KL_List).
-spec assess_candidates(list(candidate()), score_parameters()) -spec assess_candidates(
-> {list(candidate()), float()}. list(candidate()), score_parameters()) -> {list(candidate()), float()}.
%% @doc %% @doc
%% For each run length we need to assess all the possible runs of candidates, %% For each run length we need to assess all the possible runs of candidates,
%% to determine which is the best score - to be put forward as the best %% to determine which is the best score - to be put forward as the best
@ -704,10 +732,12 @@ score_run(Run, {MaxRunLength, MR_CT, SF_CT}) ->
(MR_CT - SF_CT) / (MaxRunSize - 1) (MR_CT - SF_CT) / (MaxRunSize - 1)
end, end,
Target = SF_CT + TargetIncr * (length(Run) - 1), Target = SF_CT + TargetIncr * (length(Run) - 1),
RunTotal = lists:foldl(fun(Cand, Acc) -> RunTotal =
Acc + Cand#candidate.compaction_perc end, lists:foldl(
fun(Cand, Acc) -> Acc + Cand#candidate.compaction_perc end,
0.0, 0.0,
Run), Run
),
Target - RunTotal / length(Run). Target - RunTotal / length(Run).
@ -750,26 +780,29 @@ compact_files([Batch|T], CDBopts, ActiveJournal0,
FilterFun, FilterServer, MaxSQN, FilterFun, FilterServer, MaxSQN,
RStrategy, PressMethod, ManSlice0) -> RStrategy, PressMethod, ManSlice0) ->
{SrcJournal, PositionList} = Batch, {SrcJournal, PositionList} = Batch,
KVCs0 = leveled_cdb:cdb_directfetch(SrcJournal, KVCs0 =
PositionList, leveled_cdb:cdb_directfetch(SrcJournal, PositionList, key_value_check),
key_value_check), KVCs1 =
KVCs1 = filter_output(KVCs0, filter_output(KVCs0, FilterFun, FilterServer, MaxSQN, RStrategy),
FilterFun, {ActiveJournal1, ManSlice1} =
FilterServer, write_values(
MaxSQN, KVCs1, CDBopts, ActiveJournal0, ManSlice0, PressMethod),
RStrategy),
{ActiveJournal1, ManSlice1} = write_values(KVCs1,
CDBopts,
ActiveJournal0,
ManSlice0,
PressMethod),
% The inker's clerk will no longer need these (potentially large) binaries, % The inker's clerk will no longer need these (potentially large) binaries,
% so force garbage collection at this point. This will mean when we roll % so force garbage collection at this point. This will mean when we roll
% each CDB file there will be no remaining references to the binaries that % each CDB file there will be no remaining references to the binaries that
% have been transferred and the memory can immediately be cleared % have been transferred and the memory can immediately be cleared
garbage_collect(), garbage_collect(),
compact_files(T, CDBopts, ActiveJournal1, FilterFun, FilterServer, MaxSQN, compact_files(
RStrategy, PressMethod, ManSlice1). T,
CDBopts,
ActiveJournal1,
FilterFun,
FilterServer,
MaxSQN,
RStrategy,
PressMethod,
ManSlice1
).
get_all_positions([], PositionBatches) -> get_all_positions([], PositionBatches) ->
PositionBatches; PositionBatches;
@ -777,23 +810,25 @@ get_all_positions([HeadRef|RestOfBest], PositionBatches) ->
SrcJournal = HeadRef#candidate.journal, SrcJournal = HeadRef#candidate.journal,
Positions = leveled_cdb:cdb_getpositions(SrcJournal, all), Positions = leveled_cdb:cdb_getpositions(SrcJournal, all),
leveled_log:log(ic008, [HeadRef#candidate.filename, length(Positions)]), leveled_log:log(ic008, [HeadRef#candidate.filename, length(Positions)]),
Batches = split_positions_into_batches(lists:sort(Positions), Batches =
SrcJournal, split_positions_into_batches(
[]), lists:sort(Positions), SrcJournal, []
),
get_all_positions(RestOfBest, PositionBatches ++ Batches). get_all_positions(RestOfBest, PositionBatches ++ Batches).
split_positions_into_batches([], _Journal, Batches) -> split_positions_into_batches([], _Journal, Batches) ->
Batches; Batches;
split_positions_into_batches(Positions, Journal, Batches) -> split_positions_into_batches(Positions, Journal, Batches) ->
{ThisBatch, Tail} = if {ThisBatch, Tail} =
if
length(Positions) > ?BATCH_SIZE -> length(Positions) > ?BATCH_SIZE ->
lists:split(?BATCH_SIZE, Positions); lists:split(?BATCH_SIZE, Positions);
true -> true ->
{Positions, []} {Positions, []}
end, end,
split_positions_into_batches(Tail, split_positions_into_batches(
Journal, Tail, Journal, Batches ++ [{Journal, ThisBatch}]
Batches ++ [{Journal, ThisBatch}]). ).
%% @doc %% @doc
@ -918,7 +953,7 @@ clear_waste(State) ->
N = calendar:datetime_to_gregorian_seconds(calendar:local_time()), N = calendar:datetime_to_gregorian_seconds(calendar:local_time()),
DeleteJournalFun = DeleteJournalFun =
fun(DelJ) -> fun(DelJ) ->
LMD = filelib:last_modified(WP ++ DelJ), LMD = {_,_} = filelib:last_modified(WP ++ DelJ),
case N - calendar:datetime_to_gregorian_seconds(LMD) of case N - calendar:datetime_to_gregorian_seconds(LMD) of
LMD_Delta when LMD_Delta >= WRP -> LMD_Delta when LMD_Delta >= WRP ->
ok = file:delete(WP ++ DelJ), ok = file:delete(WP ++ DelJ),
@ -931,12 +966,10 @@ clear_waste(State) ->
lists:foreach(DeleteJournalFun, ClearedJournals) lists:foreach(DeleteJournalFun, ClearedJournals)
end. end.
%%%============================================================================ %%%============================================================================
%%% Test %%% Test
%%%============================================================================ %%%============================================================================
-ifdef(TEST). -ifdef(TEST).
-include_lib("eunit/include/eunit.hrl"). -include_lib("eunit/include/eunit.hrl").
@ -966,24 +999,36 @@ local_time_to_now(DateTime) ->
{Seconds div 1000000, Seconds rem 1000000, 0}. {Seconds div 1000000, Seconds rem 1000000, 0}.
simple_score_test() -> simple_score_test() ->
Run1 = [#candidate{compaction_perc = 75.0}, DummyC =
#candidate{compaction_perc = 75.0}, #candidate{
#candidate{compaction_perc = 76.0}, low_sqn = 1, filename="dummy", journal=self(), compaction_perc = 0
#candidate{compaction_perc = 70.0}], },
Run1 = [DummyC#candidate{compaction_perc = 75.0},
DummyC#candidate{compaction_perc = 75.0},
DummyC#candidate{compaction_perc = 76.0},
DummyC#candidate{compaction_perc = 70.0}],
?assertMatch(-4.0, score_run(Run1, {4, 70.0, 40.0})), ?assertMatch(-4.0, score_run(Run1, {4, 70.0, 40.0})),
Run2 = [#candidate{compaction_perc = 75.0}], Run2 = [DummyC#candidate{compaction_perc = 75.0}],
?assertMatch(-35.0, score_run(Run2, {4, 70.0, 40.0})), ?assertMatch(-35.0, score_run(Run2, {4, 70.0, 40.0})),
?assertEqual(0.0, score_run([], {4, 40.0, 70.0})), ?assertEqual(0.0, score_run([], {4, 40.0, 70.0})),
Run3 = [#candidate{compaction_perc = 100.0}], Run3 = [DummyC#candidate{compaction_perc = 100.0}],
?assertMatch(-60.0, score_run(Run3, {4, 70.0, 40.0})). ?assertMatch(-60.0, score_run(Run3, {4, 70.0, 40.0})).
file_gc_test() -> file_gc_test() ->
State = #state{waste_path="test/test_area/waste/", State =
waste_retention_period=1}, #state{
waste_path="test/test_area/waste/", waste_retention_period = 1
},
ok = filelib:ensure_dir(State#state.waste_path), ok = filelib:ensure_dir(State#state.waste_path),
file:write_file(State#state.waste_path ++ "1.cdb", term_to_binary("Hello")), file:write_file(
filename:join(State#state.waste_path, "1.cdb"),
term_to_binary("Hello")
),
timer:sleep(1100), timer:sleep(1100),
file:write_file(State#state.waste_path ++ "2.cdb", term_to_binary("Hello")), file:write_file(
filename:join(State#state.waste_path, "2.cdb"),
term_to_binary("Hello")
),
clear_waste(State), clear_waste(State),
{ok, ClearedJournals} = file:list_dir(State#state.waste_path), {ok, ClearedJournals} = file:list_dir(State#state.waste_path),
?assertMatch(["2.cdb"], ClearedJournals), ?assertMatch(["2.cdb"], ClearedJournals),
@ -1004,27 +1049,47 @@ find_bestrun_test() ->
%% -define(MAXRUNLENGTH_COMPACTION_TARGET, 60.0). %% -define(MAXRUNLENGTH_COMPACTION_TARGET, 60.0).
%% Tested first with blocks significant as no back-tracking %% Tested first with blocks significant as no back-tracking
Params = {4, 60.0, 40.0}, Params = {4, 60.0, 40.0},
Block1 = [#candidate{compaction_perc = 55.0, filename = "a"}, DummyC =
#candidate{compaction_perc = 65.0, filename = "b"}, #candidate{
#candidate{compaction_perc = 42.0, filename = "c"}, low_sqn = 1, filename="dummy", journal=self(), compaction_perc = 0
#candidate{compaction_perc = 50.0, filename = "d"}], },
Block2 = [#candidate{compaction_perc = 38.0, filename = "e"}, Block1 =
#candidate{compaction_perc = 75.0, filename = "f"}, [
#candidate{compaction_perc = 75.0, filename = "g"}, DummyC#candidate{compaction_perc = 55.0, filename = "a"},
#candidate{compaction_perc = 45.0, filename = "h"}], DummyC#candidate{compaction_perc = 65.0, filename = "b"},
Block3 = [#candidate{compaction_perc = 70.0, filename = "i"}, DummyC#candidate{compaction_perc = 42.0, filename = "c"},
#candidate{compaction_perc = 100.0, filename = "j"}, DummyC#candidate{compaction_perc = 50.0, filename = "d"}
#candidate{compaction_perc = 100.0, filename = "k"}, ],
#candidate{compaction_perc = 100.0, filename = "l"}], Block2 =
Block4 = [#candidate{compaction_perc = 55.0, filename = "m"}, [
#candidate{compaction_perc = 56.0, filename = "n"}, DummyC#candidate{compaction_perc = 38.0, filename = "e"},
#candidate{compaction_perc = 57.0, filename = "o"}, DummyC#candidate{compaction_perc = 75.0, filename = "f"},
#candidate{compaction_perc = 40.0, filename = "p"}], DummyC#candidate{compaction_perc = 75.0, filename = "g"},
Block5 = [#candidate{compaction_perc = 60.0, filename = "q"}, DummyC#candidate{compaction_perc = 45.0, filename = "h"}
#candidate{compaction_perc = 60.0, filename = "r"}], ],
Block3 =
[
DummyC#candidate{compaction_perc = 70.0, filename = "i"},
DummyC#candidate{compaction_perc = 100.0, filename = "j"},
DummyC#candidate{compaction_perc = 100.0, filename = "k"},
DummyC#candidate{compaction_perc = 100.0, filename = "l"}
],
Block4 =
[
DummyC#candidate{compaction_perc = 55.0, filename = "m"},
DummyC#candidate{compaction_perc = 56.0, filename = "n"},
DummyC#candidate{compaction_perc = 57.0, filename = "o"},
DummyC#candidate{compaction_perc = 40.0, filename = "p"}
],
Block5 =
[
DummyC#candidate{compaction_perc = 60.0, filename = "q"},
DummyC#candidate{compaction_perc = 60.0, filename = "r"}
],
CList0 = Block1 ++ Block2 ++ Block3 ++ Block4 ++ Block5, CList0 = Block1 ++ Block2 ++ Block3 ++ Block4 ++ Block5,
?assertMatch(["b", "c", "d", "e"], check_bestrun(CList0, Params)), ?assertMatch(["b", "c", "d", "e"], check_bestrun(CList0, Params)),
CList1 = CList0 ++ [#candidate{compaction_perc = 20.0, filename="s"}], CList1 =
CList0 ++ [DummyC#candidate{compaction_perc = 20.0, filename="s"}],
?assertMatch(["s"], check_bestrun(CList1, Params)), ?assertMatch(["s"], check_bestrun(CList1, Params)),
CList2 = Block4 ++ Block3 ++ Block2 ++ Block1 ++ Block5, CList2 = Block4 ++ Block3 ++ Block2 ++ Block1 ++ Block5,
?assertMatch(["h", "a", "b", "c"], check_bestrun(CList2, Params)), ?assertMatch(["h", "a", "b", "c"], check_bestrun(CList2, Params)),
@ -1219,12 +1284,18 @@ compact_empty_file_test() ->
ok = leveled_cdb:cdb_destroy(CDB2). ok = leveled_cdb:cdb_destroy(CDB2).
compare_candidate_test() -> compare_candidate_test() ->
Candidate1 = #candidate{low_sqn=1}, DummyC =
Candidate2 = #candidate{low_sqn=2}, #candidate{
Candidate3 = #candidate{low_sqn=3}, low_sqn = 1, filename="dummy", journal=self(), compaction_perc = 0
Candidate4 = #candidate{low_sqn=4}, },
?assertMatch([Candidate1, Candidate2, Candidate3, Candidate4], Candidate1 = DummyC#candidate{low_sqn=1},
sort_run([Candidate3, Candidate2, Candidate4, Candidate1])). Candidate2 = DummyC#candidate{low_sqn=2},
Candidate3 = DummyC#candidate{low_sqn=3},
Candidate4 = DummyC#candidate{low_sqn=4},
?assertMatch(
[Candidate1, Candidate2, Candidate3, Candidate4],
sort_run([Candidate3, Candidate2, Candidate4, Candidate1])
).
compact_singlefile_totwosmallfiles_test_() -> compact_singlefile_totwosmallfiles_test_() ->
{timeout, 60, fun compact_singlefile_totwosmallfiles_testto/0}. {timeout, 60, fun compact_singlefile_totwosmallfiles_testto/0}.
@ -1236,24 +1307,31 @@ compact_singlefile_totwosmallfiles_testto() ->
FN1 = leveled_inker:filepath(RP, 1, new_journal), FN1 = leveled_inker:filepath(RP, 1, new_journal),
CDBoptsLarge = #cdb_options{binary_mode=true, max_size=30000000}, CDBoptsLarge = #cdb_options{binary_mode=true, max_size=30000000},
{ok, CDB1} = leveled_cdb:cdb_open_writer(FN1, CDBoptsLarge), {ok, CDB1} = leveled_cdb:cdb_open_writer(FN1, CDBoptsLarge),
lists:foreach(fun(X) -> lists:foreach(
fun(X) ->
LK = test_ledgerkey("Key" ++ integer_to_list(X)), LK = test_ledgerkey("Key" ++ integer_to_list(X)),
Value = leveled_rand:rand_bytes(1024), Value = crypto:strong_rand_bytes(1024),
{IK, IV} = {IK, IV} =
leveled_codec:to_inkerkv(LK, X, Value, leveled_codec:to_inkerkv(LK, X, Value,
{[], infinity}, {[], infinity},
native, true), native, true),
ok = leveled_cdb:cdb_put(CDB1, IK, IV) ok = leveled_cdb:cdb_put(CDB1, IK, IV)
end, end,
lists:seq(1, 1000)), lists:seq(1, 1000)
),
{ok, NewName} = leveled_cdb:cdb_complete(CDB1), {ok, NewName} = leveled_cdb:cdb_complete(CDB1),
{ok, CDBr} = leveled_cdb:cdb_open_reader(NewName), {ok, CDBr} = leveled_cdb:cdb_open_reader(NewName),
CDBoptsSmall = CDBoptsSmall =
#cdb_options{binary_mode=true, max_size=400000, file_path=CP}, #cdb_options{binary_mode=true, max_size=400000, file_path=CP},
BestRun1 = [#candidate{low_sqn=1, BestRun1 =
[
#candidate{
low_sqn=1,
filename=leveled_cdb:cdb_filename(CDBr), filename=leveled_cdb:cdb_filename(CDBr),
journal=CDBr, journal=CDBr,
compaction_perc=50.0}], compaction_perc=50.0
}
],
FakeFilterFun = FakeFilterFun =
fun(_FS, _LK, SQN) -> fun(_FS, _LK, SQN) ->
case SQN rem 2 of case SQN rem 2 of
@ -1262,19 +1340,24 @@ compact_singlefile_totwosmallfiles_testto() ->
end end
end, end,
ManifestSlice = compact_files(BestRun1, ManifestSlice =
compact_files(
BestRun1,
CDBoptsSmall, CDBoptsSmall,
FakeFilterFun, FakeFilterFun,
null, null,
900, 900,
[{?STD_TAG, recovr}], [{?STD_TAG, recovr}],
native), native
),
?assertMatch(2, length(ManifestSlice)), ?assertMatch(2, length(ManifestSlice)),
lists:foreach(fun({_SQN, _FN, CDB, _LK}) -> lists:foreach(
fun({_SQN, _FN, CDB, _LK}) ->
ok = leveled_cdb:cdb_deletepending(CDB), ok = leveled_cdb:cdb_deletepending(CDB),
ok = leveled_cdb:cdb_destroy(CDB) ok = leveled_cdb:cdb_destroy(CDB)
end, end,
ManifestSlice), ManifestSlice
),
ok = leveled_cdb:cdb_deletepending(CDBr), ok = leveled_cdb:cdb_deletepending(CDBr),
ok = leveled_cdb:cdb_destroy(CDBr). ok = leveled_cdb:cdb_destroy(CDBr).
@ -1304,11 +1387,13 @@ size_score_test() ->
end end
end, end,
Score = Score =
size_comparison_score(KeySizeList, size_comparison_score(
KeySizeList,
FilterFun, FilterFun,
CurrentList, CurrentList,
MaxSQN, MaxSQN,
leveled_codec:inker_reload_strategy([])), leveled_codec:inker_reload_strategy([])
),
io:format("Score ~w", [Score]), io:format("Score ~w", [Score]),
?assertMatch(true, Score > 69.0), ?assertMatch(true, Score > 69.0),
?assertMatch(true, Score < 70.0). ?assertMatch(true, Score < 70.0).

43
src/leveled_imanifest.erl
View file

@ -28,9 +28,7 @@
-type manifest() :: list({integer(), list()}). -type manifest() :: list({integer(), list()}).
%% The manifest is divided into blocks by sequence number, with each block %% The manifest is divided into blocks by sequence number, with each block
%% being a list of manifest entries for that SQN range. %% being a list of manifest entries for that SQN range.
-type manifest_entry() :: {integer(), string(), pid()|string(), any()}. -type manifest_entry() :: {integer(), string(), pid(), any()}.
%% The Entry should have a pid() as the third element, but a string() may be
%% used in unit tests
-export_type([manifest/0, manifest_entry/0]). -export_type([manifest/0, manifest_entry/0]).
@ -75,8 +73,8 @@ add_entry(Manifest, Entry, ToEnd) ->
from_list(Man1) from_list(Man1)
end. end.
-spec append_lastkey(manifest(), pid(), leveled_codec:journal_key()) -spec append_lastkey(
-> manifest(). manifest(), pid(), leveled_codec:journal_key()) -> manifest().
%% @doc %% @doc
%% On discovery of the last key in the last journal entry, the manifest can %% On discovery of the last key in the last journal entry, the manifest can
%% be updated through this function to have the last key %% be updated through this function to have the last key
@ -100,7 +98,7 @@ remove_entry(Manifest, Entry) ->
Man0 = lists:keydelete(SQN, 1, to_list(Manifest)), Man0 = lists:keydelete(SQN, 1, to_list(Manifest)),
from_list(Man0). from_list(Man0).
-spec find_entry(integer(), manifest()) -> pid()|string(). -spec find_entry(integer(), manifest()) -> pid().
%% @doc %% @doc
%% Given a SQN find the relevant manifest_entry, returning just the pid() of %% Given a SQN find the relevant manifest_entry, returning just the pid() of
%% the journal file (which may be a string() in unit tests) %% the journal file (which may be a string() in unit tests)
@ -257,18 +255,31 @@ build_testmanifest_aslist() ->
ManifestMapFun = ManifestMapFun =
fun(N) -> fun(N) ->
NStr = integer_to_list(N), NStr = integer_to_list(N),
{max(1, N * 1000), "FN" ++ NStr, "pid" ++ NStr, "LK" ++ NStr} {
max(1, N * 1000),
"FN" ++ NStr,
set_pid(N),
"LK" ++ NStr
}
end, end,
lists:map(ManifestMapFun, lists:reverse(lists:seq(0, 50))). lists:map(ManifestMapFun, lists:reverse(lists:seq(0, 50))).
set_pid(N) ->
lists:flatten(io_lib:format("<0.1~2..0w.0>", [N])).
test_testmanifest(Man0) -> test_testmanifest(Man0) ->
?assertMatch("pid0", find_entry(1, Man0)), P0 = set_pid(0),
?assertMatch("pid0", find_entry(2, Man0)), P1 = set_pid(1),
?assertMatch("pid1", find_entry(1001, Man0)), P20 = set_pid(20),
?assertMatch("pid20", find_entry(20000, Man0)), P50 = set_pid(50),
?assertMatch("pid20", find_entry(20001, Man0)),
?assertMatch("pid20", find_entry(20999, Man0)), ?assertMatch(P0, find_entry(1, Man0)),
?assertMatch("pid50", find_entry(99999, Man0)). ?assertMatch(P0, find_entry(2, Man0)),
?assertMatch(P1, find_entry(1001, Man0)),
?assertMatch(P20, find_entry(20000, Man0)),
?assertMatch(P20, find_entry(20001, Man0)),
?assertMatch(P20, find_entry(20999, Man0)),
?assertMatch(P50, find_entry(99999, Man0)).
buildfromlist_test() -> buildfromlist_test() ->
ManL = build_testmanifest_aslist(), ManL = build_testmanifest_aslist(),
@ -303,7 +314,7 @@ buildrandomfashion_test() ->
ManL0 = build_testmanifest_aslist(), ManL0 = build_testmanifest_aslist(),
RandMapFun = RandMapFun =
fun(X) -> fun(X) ->
{leveled_rand:uniform(), X} {rand:uniform(), X}
end, end,
ManL1 = lists:map(RandMapFun, ManL0), ManL1 = lists:map(RandMapFun, ManL0),
ManL2 = lists:sort(ManL1), ManL2 = lists:sort(ManL1),
@ -317,7 +328,7 @@ buildrandomfashion_test() ->
test_testmanifest(Man0), test_testmanifest(Man0),
?assertMatch(ManL0, to_list(Man0)), ?assertMatch(ManL0, to_list(Man0)),
RandomEntry = lists:nth(leveled_rand:uniform(50), ManL0), RandomEntry = lists:nth(rand:uniform(50), ManL0),
Man1 = remove_entry(Man0, RandomEntry), Man1 = remove_entry(Man0, RandomEntry),
Man2 = add_entry(Man1, RandomEntry, false), Man2 = add_entry(Man1, RandomEntry, false),

229
src/leveled_inker.erl
View file

@ -147,7 +147,7 @@
-record(state, {manifest = [] :: list(), -record(state, {manifest = [] :: list(),
manifest_sqn = 0 :: integer(), manifest_sqn = 0 :: integer(),
journal_sqn = 0 :: integer(), journal_sqn = 0 :: non_neg_integer(),
active_journaldb :: pid() | undefined, active_journaldb :: pid() | undefined,
pending_removals = [] :: list(), pending_removals = [] :: list(),
registered_snapshots = [] :: list(registered_snapshot()), registered_snapshots = [] :: list(registered_snapshot()),
@ -159,7 +159,9 @@
is_snapshot = false :: boolean(), is_snapshot = false :: boolean(),
compression_method = native :: lz4|native|none, compression_method = native :: lz4|native|none,
compress_on_receipt = false :: boolean(), compress_on_receipt = false :: boolean(),
snap_timeout :: pos_integer() | undefined, % in seconds snap_timeout = 0 :: non_neg_integer(),
% in seconds, 0 for snapshots
% (only relevant for primary Inker)
source_inker :: pid() | undefined, source_inker :: pid() | undefined,
shutdown_loops = ?SHUTDOWN_LOOPS :: non_neg_integer()}). shutdown_loops = ?SHUTDOWN_LOOPS :: non_neg_integer()}).
@ -196,20 +198,26 @@
%% The inker will need to know what the reload strategy is, to inform the %% The inker will need to know what the reload strategy is, to inform the
%% clerk about the rules to enforce during compaction. %% clerk about the rules to enforce during compaction.
ink_start(InkerOpts) -> ink_start(InkerOpts) ->
gen_server:start_link(?MODULE, [leveled_log:get_opts(), InkerOpts], []). {ok, Inker} =
gen_server:start_link(
?MODULE, [leveled_log:get_opts(), InkerOpts], []),
{ok, Inker}.
-spec ink_snapstart(inker_options()) -> {ok, pid()}. -spec ink_snapstart(inker_options()) -> {ok, pid()}.
%% @doc %% @doc
%% Don't link on startup as snapshot %% Don't link on startup as snapshot
ink_snapstart(InkerOpts) -> ink_snapstart(InkerOpts) ->
gen_server:start(?MODULE, [leveled_log:get_opts(), InkerOpts], []). {ok, Inker} =
gen_server:start(
?MODULE, [leveled_log:get_opts(), InkerOpts], []),
{ok, Inker}.
-spec ink_put(pid(), -spec ink_put(pid(),
leveled_codec:ledger_key(), leveled_codec:ledger_key(),
any(), any(),
leveled_codec:journal_keychanges(), leveled_codec:journal_keychanges(),
boolean()) -> boolean()) ->
{ok, integer(), integer()}. {ok, non_neg_integer(), pos_integer()}.
%% @doc %% @doc
%% PUT an object into the journal, returning the sequence number for the PUT %% PUT an object into the journal, returning the sequence number for the PUT
%% as well as the size of the object (information required by the ledger). %% as well as the size of the object (information required by the ledger).
@ -504,14 +512,13 @@ ink_getclerkpid(Pid) ->
init([LogOpts, InkerOpts]) -> init([LogOpts, InkerOpts]) ->
leveled_log:save(LogOpts), leveled_log:save(LogOpts),
leveled_rand:seed(),
case {InkerOpts#inker_options.root_path, case {InkerOpts#inker_options.root_path,
InkerOpts#inker_options.start_snapshot} of InkerOpts#inker_options.start_snapshot,
{undefined, true} -> InkerOpts#inker_options.source_inker} of
{undefined, true, SrcInker} when ?IS_DEF(SrcInker) ->
%% monitor the bookie, and close the snapshot when bookie %% monitor the bookie, and close the snapshot when bookie
%% exits %% exits
BookieMonitor = erlang:monitor(process, InkerOpts#inker_options.bookies_pid), BookieMonitor = erlang:monitor(process, InkerOpts#inker_options.bookies_pid),
SrcInker = InkerOpts#inker_options.source_inker,
{Manifest, {Manifest,
ActiveJournalDB, ActiveJournalDB,
JournalSQN} = ink_registersnapshot(SrcInker, self()), JournalSQN} = ink_registersnapshot(SrcInker, self()),
@ -522,7 +529,7 @@ init([LogOpts, InkerOpts]) ->
bookie_monref = BookieMonitor, bookie_monref = BookieMonitor,
is_snapshot = true}}; is_snapshot = true}};
%% Need to do something about timeout %% Need to do something about timeout
{_RootPath, false} -> {_RootPath, false, _SrcInker} ->
start_from_file(InkerOpts) start_from_file(InkerOpts)
end. end.
@ -557,10 +564,12 @@ handle_call({fold,
Manifest = lists:reverse(leveled_imanifest:to_list(State#state.manifest)), Manifest = lists:reverse(leveled_imanifest:to_list(State#state.manifest)),
Folder = Folder =
fun() -> fun() ->
fold_from_sequence(StartSQN, fold_from_sequence(
StartSQN,
{FilterFun, InitAccFun, FoldFun}, {FilterFun, InitAccFun, FoldFun},
Acc, Acc,
Manifest) Manifest
)
end, end,
case By of case By of
as_ink -> as_ink ->
@ -583,24 +592,20 @@ handle_call(get_manifest, _From, State) ->
handle_call(print_manifest, _From, State) -> handle_call(print_manifest, _From, State) ->
leveled_imanifest:printer(State#state.manifest), leveled_imanifest:printer(State#state.manifest),
{reply, ok, State}; {reply, ok, State};
handle_call({compact, handle_call(
Checker, {compact, Checker, InitiateFun, CloseFun, FilterFun},
InitiateFun, _From,
CloseFun, State=#state{is_snapshot=Snap})
FilterFun}, when Snap == false ->
_From, State=#state{is_snapshot=Snap}) when Snap == false ->
Clerk = State#state.clerk, Clerk = State#state.clerk,
Manifest = leveled_imanifest:to_list(State#state.manifest), Manifest = leveled_imanifest:to_list(State#state.manifest),
leveled_iclerk:clerk_compact(State#state.clerk, leveled_iclerk:clerk_compact(
Checker, Clerk, Checker, InitiateFun, CloseFun, FilterFun, Manifest),
InitiateFun,
CloseFun,
FilterFun,
Manifest),
{reply, {ok, Clerk}, State#state{compaction_pending=true}}; {reply, {ok, Clerk}, State#state{compaction_pending=true}};
handle_call(compaction_pending, _From, State) -> handle_call(compaction_pending, _From, State) ->
{reply, State#state.compaction_pending, State}; {reply, State#state.compaction_pending, State};
handle_call({trim, PersistedSQN}, _From, State=#state{is_snapshot=Snap}) handle_call(
{trim, PersistedSQN}, _From, State=#state{is_snapshot=Snap})
when Snap == false -> when Snap == false ->
Manifest = leveled_imanifest:to_list(State#state.manifest), Manifest = leveled_imanifest:to_list(State#state.manifest),
ok = leveled_iclerk:clerk_trim(State#state.clerk, PersistedSQN, Manifest), ok = leveled_iclerk:clerk_trim(State#state.clerk, PersistedSQN, Manifest),
@ -625,7 +630,8 @@ handle_call(roll, _From, State=#state{is_snapshot=Snap}) when Snap == false ->
manifest_sqn = NewManSQN, manifest_sqn = NewManSQN,
active_journaldb = NewJournalP}} active_journaldb = NewJournalP}}
end; end;
handle_call({backup, BackupPath}, _from, State) handle_call(
{backup, BackupPath}, _from, State)
when State#state.is_snapshot == true -> when State#state.is_snapshot == true ->
SW = os:timestamp(), SW = os:timestamp(),
BackupJFP = filepath(filename:join(BackupPath, ?JOURNAL_FP), journal_dir), BackupJFP = filepath(filename:join(BackupPath, ?JOURNAL_FP), journal_dir),
@ -665,7 +671,7 @@ handle_call({backup, BackupPath}, _from, State)
leveled_log:log(i0022, [RFN]), leveled_log:log(i0022, [RFN]),
RemoveFile = filename:join(BackupJFP, RFN), RemoveFile = filename:join(BackupJFP, RFN),
case filelib:is_file(RemoveFile) case filelib:is_file(RemoveFile)
and not filelib:is_dir(RemoveFile) of andalso not filelib:is_dir(RemoveFile) of
true -> true ->
ok = file:delete(RemoveFile); ok = file:delete(RemoveFile);
false -> false ->
@ -699,12 +705,13 @@ handle_call(get_clerkpid, _From, State) ->
handle_call(close, _From, State=#state{is_snapshot=Snap}) when Snap == true -> handle_call(close, _From, State=#state{is_snapshot=Snap}) when Snap == true ->
ok = ink_releasesnapshot(State#state.source_inker, self()), ok = ink_releasesnapshot(State#state.source_inker, self()),
{stop, normal, ok, State}; {stop, normal, ok, State};
handle_call(ShutdownType, From, State) handle_call(
when ShutdownType == close; ShutdownType == doom -> ShutdownType, From, State = #state{clerk = Clerk})
when ?IS_DEF(Clerk) ->
case ShutdownType of case ShutdownType of
doom -> doom ->
leveled_log:log(i0018, []); leveled_log:log(i0018, []);
_ -> close ->
ok ok
end, end,
leveled_log:log(i0005, [ShutdownType]), leveled_log:log(i0005, [ShutdownType]),
@ -714,9 +721,10 @@ handle_call(ShutdownType, From, State)
gen_server:cast(self(), {maybe_defer_shutdown, ShutdownType, From}), gen_server:cast(self(), {maybe_defer_shutdown, ShutdownType, From}),
{noreply, State}. {noreply, State}.
handle_cast(
handle_cast({clerk_complete, ManifestSnippet, FilesToDelete}, State) -> {clerk_complete, ManifestSnippet, FilesToDelete},
CDBOpts = State#state.cdb_options, State = #state{cdb_options = CDBOpts})
when ?IS_DEF(CDBOpts) ->
DropFun = DropFun =
fun(E, Acc) -> fun(E, Acc) ->
leveled_imanifest:remove_entry(Acc, E) leveled_imanifest:remove_entry(Acc, E)
@ -854,8 +862,10 @@ handle_cast({complete_shutdown, ShutdownType, From}, State) ->
{stop, normal, State}. {stop, normal, State}.
%% handle the bookie stopping and stop this snapshot %% handle the bookie stopping and stop this snapshot
handle_info({'DOWN', BookieMonRef, process, _BookiePid, _Info}, handle_info(
State=#state{bookie_monref = BookieMonRef}) -> {'DOWN', BookieMonRef, process, _BookiePid, _Info},
State=#state{bookie_monref = BookieMonRef, source_inker = SrcInker})
when ?IS_DEF(SrcInker) ->
%% Monitor only registered on snapshots %% Monitor only registered on snapshots
ok = ink_releasesnapshot(State#state.source_inker, self()), ok = ink_releasesnapshot(State#state.source_inker, self()),
{stop, normal, State}; {stop, normal, State};
@ -879,7 +889,10 @@ code_change(_OldVsn, State, _Extra) ->
-spec start_from_file(inker_options()) -> {ok, ink_state()}. -spec start_from_file(inker_options()) -> {ok, ink_state()}.
%% @doc %% @doc
%% Start an Inker from the state on disk (i.e. not a snapshot). %% Start an Inker from the state on disk (i.e. not a snapshot).
start_from_file(InkOpts) -> start_from_file(
InkOpts =
#inker_options{root_path = RootPath, snaptimeout_long = SnapTimeout})
when ?IS_DEF(RootPath), ?IS_DEF(SnapTimeout) ->
% Setting the correct CDB options is important when starting the inker, in % Setting the correct CDB options is important when starting the inker, in
% particular for waste retention which is determined by the CDB options % particular for waste retention which is determined by the CDB options
% with which the file was last opened % with which the file was last opened
@ -926,9 +939,8 @@ start_from_file(InkOpts) ->
{Manifest, {Manifest,
ManifestSQN, ManifestSQN,
JournalSQN, JournalSQN,
ActiveJournal} = build_manifest(ManifestFilenames, ActiveJournal} =
RootPath, build_manifest(ManifestFilenames, RootPath, CDBopts),
CDBopts),
{ok, #state{manifest = Manifest, {ok, #state{manifest = Manifest,
manifest_sqn = ManifestSQN, manifest_sqn = ManifestSQN,
journal_sqn = JournalSQN, journal_sqn = JournalSQN,
@ -971,7 +983,8 @@ get_cdbopts(InkOpts)->
CDBopts#cdb_options{waste_path = WasteFP}. CDBopts#cdb_options{waste_path = WasteFP}.
-spec put_object(leveled_codec:ledger_key(), -spec put_object(
leveled_codec:primary_key(),
any(), any(),
leveled_codec:journal_keychanges(), leveled_codec:journal_keychanges(),
boolean(), boolean(),
@ -983,39 +996,50 @@ get_cdbopts(InkOpts)->
%% only Journal. %% only Journal.
%% The reply contains the byte_size of the object, using the size calculated %% The reply contains the byte_size of the object, using the size calculated
%% to store the object. %% to store the object.
put_object(LedgerKey, Object, KeyChanges, Sync, State) -> put_object(
LedgerKey,
Object,
KeyChanges,
Sync,
State =
#state{
active_journaldb = ActiveJournal,
cdb_options = CDBOpts,
root_path = RP
})
when ?IS_DEF(ActiveJournal), ?IS_DEF(CDBOpts), ?IS_DEF(RP) ->
NewSQN = State#state.journal_sqn + 1, NewSQN = State#state.journal_sqn + 1,
ActiveJournal = State#state.active_journaldb,
{JournalKey, JournalBin} = {JournalKey, JournalBin} =
leveled_codec:to_inkerkv(LedgerKey, leveled_codec:to_inkerkv(
LedgerKey,
NewSQN, NewSQN,
Object, Object,
KeyChanges, KeyChanges,
State#state.compression_method, State#state.compression_method,
State#state.compress_on_receipt), State#state.compress_on_receipt
case leveled_cdb:cdb_put(ActiveJournal, ),
JournalKey, PutR = leveled_cdb:cdb_put(ActiveJournal, JournalKey, JournalBin, Sync),
JournalBin, case PutR of
Sync) of
ok -> ok ->
{ok, {ok, State#state{journal_sqn=NewSQN}, byte_size(JournalBin)};
State#state{journal_sqn=NewSQN},
byte_size(JournalBin)};
roll -> roll ->
SWroll = os:timestamp(), SWroll = os:timestamp(),
{NewJournalP, Manifest1, NewManSQN} = {NewJournalP, Manifest1, NewManSQN} =
roll_active(ActiveJournal, roll_active(
ActiveJournal,
State#state.manifest, State#state.manifest,
NewSQN, NewSQN,
State#state.cdb_options, State#state.cdb_options,
State#state.root_path, State#state.root_path,
State#state.manifest_sqn), State#state.manifest_sqn
),
leveled_log:log_timer(i0008, [], SWroll), leveled_log:log_timer(i0008, [], SWroll),
ok = leveled_cdb:cdb_put(NewJournalP, ok =
JournalKey, leveled_cdb:cdb_put(
JournalBin), NewJournalP, JournalKey, JournalBin),
{rolling, {rolling,
State#state{journal_sqn=NewSQN, State#state{
journal_sqn=NewSQN,
manifest=Manifest1, manifest=Manifest1,
manifest_sqn = NewManSQN, manifest_sqn = NewManSQN,
active_journaldb=NewJournalP}, active_journaldb=NewJournalP},
@ -1023,7 +1047,8 @@ put_object(LedgerKey, Object, KeyChanges, Sync, State) ->
end. end.
-spec get_object(leveled_codec:ledger_key(), -spec get_object(
leveled_codec:ledger_key(),
integer(), integer(),
leveled_imanifest:manifest()) -> any(). leveled_imanifest:manifest()) -> any().
%% @doc %% @doc
@ -1041,14 +1066,19 @@ get_object(LedgerKey, SQN, Manifest, ToIgnoreKeyChanges) ->
leveled_codec:from_inkerkv(Obj, ToIgnoreKeyChanges). leveled_codec:from_inkerkv(Obj, ToIgnoreKeyChanges).
-spec roll_active(pid(), leveled_imanifest:manifest(), -spec roll_active(
integer(), #cdb_options{}, string(), integer()) -> pid(),
{pid(), leveled_imanifest:manifest(), integer()}. leveled_imanifest:manifest(),
integer(),
#cdb_options{},
string(),
integer()) -> {pid(), leveled_imanifest:manifest(), integer()}.
%% @doc %% @doc
%% Roll the active journal, and start a new active journal, updating the %% Roll the active journal, and start a new active journal, updating the
%% manifest %% manifest
roll_active(ActiveJournal, Manifest, NewSQN, CDBopts, RootPath, ManifestSQN) -> roll_active(ActiveJournal, Manifest, NewSQN, CDBopts, RootPath, ManifestSQN) ->
LastKey = leveled_cdb:cdb_lastkey(ActiveJournal), case leveled_cdb:cdb_lastkey(ActiveJournal) of
LastKey when LastKey =/= empty ->
ok = leveled_cdb:cdb_roll(ActiveJournal), ok = leveled_cdb:cdb_roll(ActiveJournal),
Manifest0 = Manifest0 =
leveled_imanifest:append_lastkey(Manifest, ActiveJournal, LastKey), leveled_imanifest:append_lastkey(Manifest, ActiveJournal, LastKey),
@ -1056,11 +1086,14 @@ roll_active(ActiveJournal, Manifest, NewSQN, CDBopts, RootPath, ManifestSQN) ->
start_new_activejournal(NewSQN, RootPath, CDBopts), start_new_activejournal(NewSQN, RootPath, CDBopts),
{_, _, NewJournalP, _} = ManEntry, {_, _, NewJournalP, _} = ManEntry,
Manifest1 = leveled_imanifest:add_entry(Manifest0, ManEntry, true), Manifest1 = leveled_imanifest:add_entry(Manifest0, ManEntry, true),
ok = leveled_imanifest:writer(Manifest1, ManifestSQN + 1, RootPath), ok =
leveled_imanifest:writer(Manifest1, ManifestSQN + 1, RootPath),
{NewJournalP, Manifest1, ManifestSQN + 1}. {NewJournalP, Manifest1, ManifestSQN + 1}
end.
-spec key_check(leveled_codec:ledger_key(), -spec key_check(
leveled_codec:primary_key(),
integer(), integer(),
leveled_imanifest:manifest()) -> missing|probably. leveled_imanifest:manifest()) -> missing|probably.
%% @doc %% @doc
@ -1081,23 +1114,20 @@ key_check(LedgerKey, SQN, Manifest) ->
%% Selects the correct manifest to open, and then starts a process for each %% Selects the correct manifest to open, and then starts a process for each
%% file in the manifest, storing the PID for that process within the manifest. %% file in the manifest, storing the PID for that process within the manifest.
%% Opens an active journal if one is not present. %% Opens an active journal if one is not present.
build_manifest(ManifestFilenames, build_manifest(ManifestFilenames, RootPath, CDBopts) ->
RootPath,
CDBopts) ->
% Find the manifest with a highest Manifest sequence number % Find the manifest with a highest Manifest sequence number
% Open it and read it to get the current Confirmed Manifest % Open it and read it to get the current Confirmed Manifest
ManifestRegex = "(?<MSQN>[0-9]+)\\." ++ leveled_imanifest:complete_filex(), ManifestRegex = "(?<MSQN>[0-9]+)\\." ++ leveled_imanifest:complete_filex(),
ValidManSQNs = sequencenumbers_fromfilenames(ManifestFilenames, ValidManSQNs =
ManifestRegex, sequencenumbers_fromfilenames(
'MSQN'), ManifestFilenames, ManifestRegex, 'MSQN'),
{Manifest, {Manifest, ManifestSQN} =
ManifestSQN} = case length(ValidManSQNs) of case length(ValidManSQNs) of
0 -> 0 ->
{[], 1}; {[], 1};
_ -> _ ->
PersistedManSQN = lists:max(ValidManSQNs), PersistedManSQN = lists:max(ValidManSQNs),
M1 = leveled_imanifest:reader(PersistedManSQN, M1 = leveled_imanifest:reader(PersistedManSQN, RootPath),
RootPath),
{M1, PersistedManSQN} {M1, PersistedManSQN}
end, end,
@ -1105,11 +1135,10 @@ build_manifest(ManifestFilenames,
% a valid active journal at the head of the manifest % a valid active journal at the head of the manifest
OpenManifest = open_all_manifest(Manifest, RootPath, CDBopts), OpenManifest = open_all_manifest(Manifest, RootPath, CDBopts),
{ActiveLowSQN, {ActiveLowSQN, _FN, ActiveJournal, _LK} =
_FN, leveled_imanifest:head_entry(OpenManifest),
ActiveJournal, JournalSQN =
_LK} = leveled_imanifest:head_entry(OpenManifest), case leveled_cdb:cdb_lastkey(ActiveJournal) of
JournalSQN = case leveled_cdb:cdb_lastkey(ActiveJournal) of
empty -> empty ->
ActiveLowSQN; ActiveLowSQN;
{JSQN, _Type, _LastKey} -> {JSQN, _Type, _LastKey} ->
@ -1146,7 +1175,8 @@ close_allmanifest([H|ManifestT]) ->
close_allmanifest(ManifestT). close_allmanifest(ManifestT).
-spec open_all_manifest(leveled_imanifest:manifest(), list(), #cdb_options{}) -spec open_all_manifest(
leveled_imanifest:manifest(), list(), #cdb_options{})
-> leveled_imanifest:manifest(). -> leveled_imanifest:manifest().
%% @doc %% @doc
%% Open all the files in the manifets, and updating the manifest with the PIDs %% Open all the files in the manifets, and updating the manifest with the PIDs
@ -1185,24 +1215,21 @@ open_all_manifest(Man0, RootPath, CDBOpts) ->
true -> true ->
leveled_log:log(i0012, [HeadFN]), leveled_log:log(i0012, [HeadFN]),
{ok, HeadR} = leveled_cdb:cdb_open_reader(CompleteHeadFN), {ok, HeadR} = leveled_cdb:cdb_open_reader(CompleteHeadFN),
LastKey = leveled_cdb:cdb_lastkey(HeadR), LastKey = {LastSQN, _, _} = leveled_cdb:cdb_lastkey(HeadR),
LastSQN = element(1, LastKey), ManToHead =
ManToHead = leveled_imanifest:add_entry(OpenedTail, leveled_imanifest:add_entry(
{HeadSQN, OpenedTail,
HeadFN, {HeadSQN, HeadFN, HeadR, LastKey},
HeadR, true
LastKey}, ),
true), NewManEntry =
NewManEntry = start_new_activejournal(LastSQN + 1, start_new_activejournal(LastSQN + 1, RootPath, CDBOpts),
RootPath,
CDBOpts),
leveled_imanifest:add_entry(ManToHead, NewManEntry, true); leveled_imanifest:add_entry(ManToHead, NewManEntry, true);
false -> false ->
{ok, HeadW} = leveled_cdb:cdb_open_writer(PendingHeadFN, {ok, HeadW} =
CDBOpts), leveled_cdb:cdb_open_writer(PendingHeadFN, CDBOpts),
leveled_imanifest:add_entry(OpenedTail, leveled_imanifest:add_entry(
{HeadSQN, HeadFN, HeadW, HeadLK}, OpenedTail, {HeadSQN, HeadFN, HeadW, HeadLK}, true)
true)
end. end.
@ -1288,7 +1315,8 @@ foldfile_between_sequence(MinSQN, MaxSQN, FoldFuns,
sequencenumbers_fromfilenames(Filenames, Regex, IntName) -> sequencenumbers_fromfilenames(Filenames, Regex, IntName) ->
lists:foldl(fun(FN, Acc) -> lists:foldl(
fun(FN, Acc) ->
case re:run(FN, case re:run(FN,
Regex, Regex,
[{capture, [IntName], list}]) of [{capture, [IntName], list}]) of
@ -1296,7 +1324,8 @@ sequencenumbers_fromfilenames(Filenames, Regex, IntName) ->
Acc; Acc;
{match, [Int]} when is_list(Int) -> {match, [Int]} when is_list(Int) ->
Acc ++ [list_to_integer(Int)] Acc ++ [list_to_integer(Int)]
end end, end
end,
[], [],
Filenames). Filenames).
@ -1525,7 +1554,7 @@ compact_journal_testto(WRP, ExpectedFiles) ->
PK = "KeyZ" ++ integer_to_list(X), PK = "KeyZ" ++ integer_to_list(X),
{ok, SQN, _} = ink_put(Ink1, {ok, SQN, _} = ink_put(Ink1,
test_ledgerkey(PK), test_ledgerkey(PK),
leveled_rand:rand_bytes(10000), crypto:strong_rand_bytes(10000),
{[], infinity}, {[], infinity},
false), false),
{SQN, test_ledgerkey(PK)} {SQN, test_ledgerkey(PK)}

4
src/leveled_log.erl
View file

@ -377,7 +377,7 @@ get_opts() ->
} }
end. end.
-spec return_settings() -> {log_level(), list(string())}. -spec return_settings() -> {log_level(), list(atom())}.
%% @doc %% @doc
%% Return the settings outside of the record %% Return the settings outside of the record
return_settings() -> return_settings() ->
@ -454,7 +454,7 @@ log_timer(LogRef, Subs, StartTime, SupportedLevels) ->
-spec log_randomtimer(atom(), list(), erlang:timestamp(), float()) -> ok. -spec log_randomtimer(atom(), list(), erlang:timestamp(), float()) -> ok.
log_randomtimer(LogReference, Subs, StartTime, RandomProb) -> log_randomtimer(LogReference, Subs, StartTime, RandomProb) ->
R = leveled_rand:uniform(), R = rand:uniform(),
case R < RandomProb of case R < RandomProb of
true -> true ->
log_timer(LogReference, Subs, StartTime); log_timer(LogReference, Subs, StartTime);

15
src/leveled_monitor.erl
View file

@ -136,13 +136,13 @@
{leveled_pmanifest:lsm_level(), #sst_fetch_timings{}}. {leveled_pmanifest:lsm_level(), #sst_fetch_timings{}}.
-type log_type() :: -type log_type() ::
bookie_head|bookie_get|bookie_put|bookie_snap|pcl_fetch|sst_fetch|cdb_get. bookie_head|bookie_get|bookie_put|bookie_snap|pcl_fetch|sst_fetch|cdb_get.
-type pcl_level() :: mem|leveled_pmanifest:lsm_level(). -type pcl_level() :: memory|leveled_pmanifest:lsm_level().
-type sst_fetch_type() :: -type sst_fetch_type() ::
fetch_cache|slot_cachedblock|slot_noncachedblock|not_found. fetch_cache|slot_cachedblock|slot_noncachedblock|not_found.
-type microsecs() :: pos_integer(). -type microsecs() :: pos_integer().
-type byte_size() :: pos_integer(). -type byte_size() :: pos_integer().
-type monitor() :: {no_monitor, 0}|{pid(), 0..100}. -type monitor() :: {no_monitor, 0}|{pid(), 0..100}.
-type timing() :: no_timing|pos_integer(). -type timing() :: no_timing|microsecs().
-type bookie_get_update() :: -type bookie_get_update() ::
@ -173,8 +173,10 @@
-spec monitor_start(pos_integer(), list(log_type())) -> {ok, pid()}. -spec monitor_start(pos_integer(), list(log_type())) -> {ok, pid()}.
monitor_start(LogFreq, LogOrder) -> monitor_start(LogFreq, LogOrder) ->
{ok, Monitor} =
gen_server:start_link( gen_server:start_link(
?MODULE, [leveled_log:get_opts(), LogFreq, LogOrder], []). ?MODULE, [leveled_log:get_opts(), LogFreq, LogOrder], []),
{ok, Monitor}.
-spec add_stat(pid(), statistic()) -> ok. -spec add_stat(pid(), statistic()) -> ok.
add_stat(Watcher, Statistic) -> add_stat(Watcher, Statistic) ->
@ -204,7 +206,7 @@ log_remove(Pid, ForcedLogs) ->
-spec maybe_time(monitor()) -> erlang:timestamp()|no_timing. -spec maybe_time(monitor()) -> erlang:timestamp()|no_timing.
maybe_time({_Pid, TimingProbability}) -> maybe_time({_Pid, TimingProbability}) ->
case leveled_rand:uniform(100) of case rand:uniform(100) of
N when N =< TimingProbability -> N when N =< TimingProbability ->
os:timestamp(); os:timestamp();
_ -> _ ->
@ -230,16 +232,15 @@ get_defaults() ->
init([LogOpts, LogFrequency, LogOrder]) -> init([LogOpts, LogFrequency, LogOrder]) ->
leveled_log:save(LogOpts), leveled_log:save(LogOpts),
leveled_rand:seed(),
RandomLogOrder = RandomLogOrder =
lists:map( lists:map(
fun({_R, SL}) -> SL end, fun({_R, SL}) -> SL end,
lists:keysort( lists:keysort(
1, 1,
lists:map( lists:map(
fun(L) -> {leveled_rand:uniform(), L} end, fun(L) -> {rand:uniform(), L} end,
LogOrder))), LogOrder))),
InitialJitter = leveled_rand:uniform(2 * 1000 * LogFrequency), InitialJitter = rand:uniform(2 * 1000 * LogFrequency),
erlang:send_after(InitialJitter, self(), report_next_stats), erlang:send_after(InitialJitter, self(), report_next_stats),
{ok, #state{log_frequency = LogFrequency, log_order = RandomLogOrder}}. {ok, #state{log_frequency = LogFrequency, log_order = RandomLogOrder}}.

202
src/leveled_pclerk.erl
View file

@ -48,8 +48,8 @@
-define(MIN_TIMEOUT, 200). -define(MIN_TIMEOUT, 200).
-define(GROOMING_PERC, 50). -define(GROOMING_PERC, 50).
-record(state, {owner :: pid() | undefined, -record(state, {owner :: pid()|undefined,
root_path :: string() | undefined, root_path :: string()|undefined,
pending_deletions = dict:new() :: dict:dict(), pending_deletions = dict:new() :: dict:dict(),
sst_options :: sst_options() sst_options :: sst_options()
}). }).
@ -123,18 +123,20 @@ handle_call(close, _From, State) ->
handle_cast(prompt, State) -> handle_cast(prompt, State) ->
handle_info(timeout, State); handle_info(timeout, State);
handle_cast({push_work, Work}, State) -> handle_cast(
{push_work, Work}, State = #state{root_path = RP, owner = PCL})
when ?IS_DEF(RP), is_pid(PCL) ->
{ManifestSQN, Deletions} = {ManifestSQN, Deletions} =
handle_work( handle_work(Work, RP, State#state.sst_options, PCL),
Work,
State#state.root_path, State#state.sst_options, State#state.owner),
PDs = dict:store(ManifestSQN, Deletions, State#state.pending_deletions), PDs = dict:store(ManifestSQN, Deletions, State#state.pending_deletions),
leveled_log:log(pc022, [ManifestSQN]), leveled_log:log(pc022, [ManifestSQN]),
{noreply, State#state{pending_deletions = PDs}, ?MIN_TIMEOUT}; {noreply, State#state{pending_deletions = PDs}, ?MIN_TIMEOUT};
handle_cast({prompt_deletions, ManifestSQN}, State) -> handle_cast(
{Deletions, UpdD} = return_deletions(ManifestSQN, {prompt_deletions, ManifestSQN}, State = #state{owner = PCL})
State#state.pending_deletions), when is_pid(PCL) ->
ok = notify_deletions(Deletions, State#state.owner), {Deletions, UpdD} =
return_deletions(ManifestSQN, State#state.pending_deletions),
ok = notify_deletions(Deletions, PCL),
{noreply, State#state{pending_deletions = UpdD}, ?MIN_TIMEOUT}; {noreply, State#state{pending_deletions = UpdD}, ?MIN_TIMEOUT};
handle_cast({log_level, LogLevel}, State) -> handle_cast({log_level, LogLevel}, State) ->
ok = leveled_log:set_loglevel(LogLevel), ok = leveled_log:set_loglevel(LogLevel),
@ -152,8 +154,8 @@ handle_cast({remove_logs, ForcedLogs}, State) ->
SSTopts0 = SSTopts#sst_options{log_options = leveled_log:get_opts()}, SSTopts0 = SSTopts#sst_options{log_options = leveled_log:get_opts()},
{noreply, State#state{sst_options = SSTopts0}}. {noreply, State#state{sst_options = SSTopts0}}.
handle_info(timeout, State) -> handle_info(timeout, State = #state{owner = PCL}) when is_pid(PCL) ->
ok = leveled_penciller:pcl_workforclerk(State#state.owner), ok = leveled_penciller:pcl_workforclerk(PCL),
% When handling work, the clerk can collect a large number of binary % When handling work, the clerk can collect a large number of binary
% references, so proactively GC this process before receiving any future % references, so proactively GC this process before receiving any future
% work. In under pressure clusters, clerks with large binary memory % work. In under pressure clusters, clerks with large binary memory
@ -207,7 +209,7 @@ merge(SrcLevel, Manifest, RootPath, OptsSST) ->
[SrcLevel + 1, FCnt, MnHBS, MnHS, MnLHS, MnBVHS]) [SrcLevel + 1, FCnt, MnHBS, MnHS, MnLHS, MnBVHS])
end, end,
SelectMethod = SelectMethod =
case leveled_rand:uniform(100) of case rand:uniform(100) of
R when R =< ?GROOMING_PERC -> R when R =< ?GROOMING_PERC ->
{grooming, fun grooming_scorer/1}; {grooming, fun grooming_scorer/1};
_ -> _ ->
@ -220,16 +222,22 @@ merge(SrcLevel, Manifest, RootPath, OptsSST) ->
leveled_pmanifest:merge_lookup( leveled_pmanifest:merge_lookup(
Manifest, Manifest,
SrcLevel + 1, SrcLevel + 1,
Src#manifest_entry.start_key, leveled_pmanifest:entry_startkey(Src),
Src#manifest_entry.end_key leveled_pmanifest:entry_endkey(Src)
), ),
Candidates = length(SinkList), Candidates = length(SinkList),
leveled_log:log(pc008, [SrcLevel, Candidates]), leveled_log:log(pc008, [SrcLevel, Candidates]),
case Candidates of case Candidates of
0 -> 0 ->
NewLevel = SrcLevel + 1, NewLevel = SrcLevel + 1,
leveled_log:log(pc009, [Src#manifest_entry.filename, NewLevel]), leveled_log:log(
leveled_sst:sst_switchlevels(Src#manifest_entry.owner, NewLevel), pc009,
[leveled_pmanifest:entry_filename(Src), NewLevel]
),
leveled_sst:sst_switchlevels(
leveled_pmanifest:entry_owner(Src),
NewLevel
),
Man0 = Man0 =
leveled_pmanifest:switch_manifest_entry( leveled_pmanifest:switch_manifest_entry(
Manifest, Manifest,
@ -249,7 +257,11 @@ merge(SrcLevel, Manifest, RootPath, OptsSST) ->
notify_deletions([], _Penciller) -> notify_deletions([], _Penciller) ->
ok; ok;
notify_deletions([Head|Tail], Penciller) -> notify_deletions([Head|Tail], Penciller) ->
ok = leveled_sst:sst_setfordelete(Head#manifest_entry.owner, Penciller), ok =
leveled_sst:sst_setfordelete(
leveled_pmanifest:entry_owner(Head),
Penciller
),
notify_deletions(Tail, Penciller). notify_deletions(Tail, Penciller).
@ -259,9 +271,12 @@ notify_deletions([Head|Tail], Penciller) ->
%% SrcLevel is the level of the src sst file, the sink should be srcLevel + 1 %% SrcLevel is the level of the src sst file, the sink should be srcLevel + 1
perform_merge(Manifest, Src, SinkList, SrcLevel, RootPath, NewSQN, OptsSST) -> perform_merge(Manifest, Src, SinkList, SrcLevel, RootPath, NewSQN, OptsSST) ->
leveled_log:log(pc010, [Src#manifest_entry.filename, NewSQN]), leveled_log:log(pc010, [leveled_pmanifest:entry_filename(Src), NewSQN]),
SrcList = [{next, Src, all}], SrcList = [{next, Src, all}],
MaxSQN = leveled_sst:sst_getmaxsequencenumber(Src#manifest_entry.owner), MaxSQN =
leveled_sst:sst_getmaxsequencenumber(
leveled_pmanifest:entry_owner(Src)
),
SinkLevel = SrcLevel + 1, SinkLevel = SrcLevel + 1,
SinkBasement = leveled_pmanifest:is_basement(Manifest, SinkLevel), SinkBasement = leveled_pmanifest:is_basement(Manifest, SinkLevel),
Additions = Additions =
@ -319,13 +334,8 @@ do_merge(KL1, KL2, SinkLevel, SinkB, RP, NewSQN, MaxSQN, OptsSST, Additions) ->
{ok, Pid, Reply, Bloom} -> {ok, Pid, Reply, Bloom} ->
{{KL1Rem, KL2Rem}, SmallestKey, HighestKey} = Reply, {{KL1Rem, KL2Rem}, SmallestKey, HighestKey} = Reply,
Entry = Entry =
#manifest_entry{ leveled_pmanifest:new_entry(
start_key=SmallestKey, SmallestKey, HighestKey, Pid, FileName, Bloom),
end_key=HighestKey,
owner=Pid,
filename=FileName,
bloom=Bloom
},
leveled_log:log_timer(pc015, [], TS1), leveled_log:log_timer(pc015, [], TS1),
do_merge( do_merge(
KL1Rem, KL2Rem, KL1Rem, KL2Rem,
@ -340,7 +350,8 @@ do_merge(KL1, KL2, SinkLevel, SinkB, RP, NewSQN, MaxSQN, OptsSST, Additions) ->
list(leveled_pmanifest:manifest_entry())) list(leveled_pmanifest:manifest_entry()))
-> leveled_pmanifest:manifest_entry(). -> leveled_pmanifest:manifest_entry().
grooming_scorer([ME | MEs]) -> grooming_scorer([ME | MEs]) ->
InitTombCount = leveled_sst:sst_gettombcount(ME#manifest_entry.owner), InitTombCount =
leveled_sst:sst_gettombcount(leveled_pmanifest:entry_owner(ME)),
{HighestTC, BestME} = grooming_scorer(InitTombCount, ME, MEs), {HighestTC, BestME} = grooming_scorer(InitTombCount, ME, MEs),
leveled_log:log(pc024, [HighestTC]), leveled_log:log(pc024, [HighestTC]),
BestME. BestME.
@ -348,7 +359,8 @@ grooming_scorer([ME | MEs]) ->
grooming_scorer(HighestTC, BestME, []) -> grooming_scorer(HighestTC, BestME, []) ->
{HighestTC, BestME}; {HighestTC, BestME};
grooming_scorer(HighestTC, BestME, [ME | MEs]) -> grooming_scorer(HighestTC, BestME, [ME | MEs]) ->
TombCount = leveled_sst:sst_gettombcount(ME#manifest_entry.owner), TombCount =
leveled_sst:sst_gettombcount(leveled_pmanifest:entry_owner(ME)),
case TombCount > HighestTC of case TombCount > HighestTC of
true -> true ->
grooming_scorer(TombCount, ME, MEs); grooming_scorer(TombCount, ME, MEs);
@ -385,11 +397,17 @@ generate_randomkeys(Count, Acc, BucketLow, BRange) ->
BNumber = BNumber =
lists:flatten( lists:flatten(
io_lib:format("~4..0B", io_lib:format("~4..0B",
[BucketLow + leveled_rand:uniform(BRange)])), [BucketLow + rand:uniform(BRange)])),
KNumber = KNumber =
lists:flatten( lists:flatten(
io_lib:format("~4..0B", [leveled_rand:uniform(1000)])), io_lib:format("~4..0B", [rand:uniform(1000)])),
K = {o, "Bucket" ++ BNumber, "Key" ++ KNumber, null}, K =
{
o,
list_to_binary("Bucket" ++ BNumber),
list_to_binary("Key" ++ KNumber),
null
},
RandKey = {K, {Count + 1, RandKey = {K, {Count + 1,
{active, infinity}, {active, infinity},
leveled_codec:segment_hash(K), leveled_codec:segment_hash(K),
@ -415,7 +433,6 @@ grooming_score_test() ->
3, 3,
999999, 999999,
#sst_options{}, #sst_options{},
true,
true), true),
{ok, PidL3_1B, _, _} = {ok, PidL3_1B, _, _} =
leveled_sst:sst_newmerge("test/test_area/ledger_files/", leveled_sst:sst_newmerge("test/test_area/ledger_files/",
@ -427,7 +444,6 @@ grooming_score_test() ->
3, 3,
999999, 999999,
#sst_options{}, #sst_options{},
true,
true), true),
{ok, PidL3_2, _, _} = {ok, PidL3_2, _, _} =
@ -439,102 +455,116 @@ grooming_score_test() ->
3, 3,
999999, 999999,
#sst_options{}, #sst_options{},
true,
true), true),
{ok, PidL3_2NC, _, _} = DSK = {o, <<"B">>, <<"SK">>, null},
leveled_sst:sst_newmerge("test/test_area/ledger_files/", DEK = {o, <<"E">>, <<"EK">>, null},
"2NC_L3.sst", ME1 = leveled_pmanifest:new_entry(DSK, DEK, PidL3_1, "dummyL3_1", none),
KL3_L3, ME1B = leveled_pmanifest:new_entry(DSK, DEK, PidL3_1B, "dummyL3_1B", none),
KL4_L3, ME2 = leveled_pmanifest:new_entry(DSK, DEK, PidL3_2, "dummyL3_2", none),
false,
3,
999999,
#sst_options{},
true,
false),
ME1 = #manifest_entry{owner=PidL3_1},
ME1B = #manifest_entry{owner=PidL3_1B},
ME2 = #manifest_entry{owner=PidL3_2},
ME2NC = #manifest_entry{owner=PidL3_2NC},
?assertMatch(ME1, grooming_scorer([ME1, ME2])), ?assertMatch(ME1, grooming_scorer([ME1, ME2])),
?assertMatch(ME1, grooming_scorer([ME2, ME1])), ?assertMatch(ME1, grooming_scorer([ME2, ME1])),
% prefer the file with the tombstone % prefer the file with the tombstone
?assertMatch(ME2NC, grooming_scorer([ME1, ME2NC])),
?assertMatch(ME2NC, grooming_scorer([ME2NC, ME1])),
% not_counted > 1 - we will merge files in unexpected (i.e. legacy)
% format first
?assertMatch(ME1B, grooming_scorer([ME1B, ME2])), ?assertMatch(ME1B, grooming_scorer([ME1B, ME2])),
?assertMatch(ME2, grooming_scorer([ME2, ME1B])), ?assertMatch(ME2, grooming_scorer([ME2, ME1B])),
% If the file with the tombstone is in the basement, it will have % If the file with the tombstone is in the basement, it will have
% no tombstone so the first file will be chosen % no tombstone so the first file will be chosen
lists:foreach(fun(P) -> leveled_sst:sst_clear(P) end, lists:foreach(fun(P) -> leveled_sst:sst_clear(P) end,
[PidL3_1, PidL3_1B, PidL3_2, PidL3_2NC]). [PidL3_1, PidL3_1B, PidL3_2]).
merge_file_test() -> merge_file_test() ->
ok = filelib:ensure_dir("test/test_area/ledger_files/"), ok = filelib:ensure_dir("test/test_area/ledger_files/"),
KL1_L1 = lists:sort(generate_randomkeys(8000, 0, 1000)), KL1_L1 = lists:sort(generate_randomkeys(8000, 0, 1000)),
{ok, PidL1_1, _, _} = {ok, PidL1_1, _, _} =
leveled_sst:sst_new("test/test_area/ledger_files/", leveled_sst:sst_new(
"test/test_area/ledger_files/",
"KL1_L1.sst", "KL1_L1.sst",
1, 1,
KL1_L1, KL1_L1,
999999, 999999,
#sst_options{}), #sst_options{}
),
KL1_L2 = lists:sort(generate_randomkeys(8000, 0, 250)), KL1_L2 = lists:sort(generate_randomkeys(8000, 0, 250)),
{ok, PidL2_1, _, _} = {ok, PidL2_1, _, _} =
leveled_sst:sst_new("test/test_area/ledger_files/", leveled_sst:sst_new(
"test/test_area/ledger_files/",
"KL1_L2.sst", "KL1_L2.sst",
2, 2,
KL1_L2, KL1_L2,
999999, 999999,
#sst_options{}), #sst_options{}
),
KL2_L2 = lists:sort(generate_randomkeys(8000, 250, 250)), KL2_L2 = lists:sort(generate_randomkeys(8000, 250, 250)),
{ok, PidL2_2, _, _} = {ok, PidL2_2, _, _} =
leveled_sst:sst_new("test/test_area/ledger_files/", leveled_sst:sst_new(
"test/test_area/ledger_files/",
"KL2_L2.sst", "KL2_L2.sst",
2, 2,
KL2_L2, KL2_L2,
999999, 999999,
#sst_options{press_method = lz4}), #sst_options{press_method = lz4}
),
KL3_L2 = lists:sort(generate_randomkeys(8000, 500, 250)), KL3_L2 = lists:sort(generate_randomkeys(8000, 500, 250)),
{ok, PidL2_3, _, _} = {ok, PidL2_3, _, _} =
leveled_sst:sst_new("test/test_area/ledger_files/", leveled_sst:sst_new(
"test/test_area/ledger_files/",
"KL3_L2.sst", "KL3_L2.sst",
2, 2,
KL3_L2, KL3_L2,
999999, 999999,
#sst_options{press_method = lz4}), #sst_options{press_method = lz4}
),
KL4_L2 = lists:sort(generate_randomkeys(8000, 750, 250)), KL4_L2 = lists:sort(generate_randomkeys(8000, 750, 250)),
{ok, PidL2_4, _, _} = {ok, PidL2_4, _, _} =
leveled_sst:sst_new("test/test_area/ledger_files/", leveled_sst:sst_new(
"test/test_area/ledger_files/",
"KL4_L2.sst", "KL4_L2.sst",
2, 2,
KL4_L2, KL4_L2,
999999, 999999,
#sst_options{press_method = lz4}), #sst_options{press_method = lz4}
E1 = #manifest_entry{owner = PidL1_1, ),
filename = "./KL1_L1.sst", E1 =
end_key = lists:last(KL1_L1), leveled_pmanifest:new_entry(
start_key = lists:nth(1, KL1_L1)}, lists:nth(1, KL1_L1),
E2 = #manifest_entry{owner = PidL2_1, lists:last(KL1_L1),
filename = "./KL1_L2.sst", PidL1_1,
end_key = lists:last(KL1_L2), "./KL1_L1.sst",
start_key = lists:nth(1, KL1_L2)}, none
E3 = #manifest_entry{owner = PidL2_2, ),
filename = "./KL2_L2.sst", E2 =
end_key = lists:last(KL2_L2), leveled_pmanifest:new_entry(
start_key = lists:nth(1, KL2_L2)}, lists:nth(1, KL1_L2),
E4 = #manifest_entry{owner = PidL2_3, lists:last(KL1_L2),
filename = "./KL3_L2.sst", PidL2_1,
end_key = lists:last(KL3_L2), "./KL1_L2.sst",
start_key = lists:nth(1, KL3_L2)}, none
E5 = #manifest_entry{owner = PidL2_4, ),
filename = "./KL4_L2.sst", E3 =
end_key = lists:last(KL4_L2), leveled_pmanifest:new_entry(
start_key = lists:nth(1, KL4_L2)}, lists:nth(1, KL2_L2),
lists:last(KL2_L2),
PidL2_2,
"./KL2_L2.sst",
none
),
E4 =
leveled_pmanifest:new_entry(
lists:nth(1, KL3_L2),
lists:last(KL3_L2),
PidL2_3,
"./KL3_L2.sst",
none
),
E5 =
leveled_pmanifest:new_entry(
lists:nth(1, KL4_L2),
lists:last(KL4_L2),
PidL2_4,
"./KL4_L2.sst",
none
),
Man0 = leveled_pmanifest:new_manifest(), Man0 = leveled_pmanifest:new_manifest(),
Man1 = leveled_pmanifest:insert_manifest_entry(Man0, 1, 2, E2), Man1 = leveled_pmanifest:insert_manifest_entry(Man0, 1, 2, E2),

1040
src/leveled_penciller.erl
View file

File diff suppressed because it is too large Load diff

791
src/leveled_pmanifest.erl
View file

File diff suppressed because it is too large Load diff

108
src/leveled_pmem.erl
View file

@ -41,9 +41,12 @@
cache_full/1 cache_full/1
]). ]).
% Test functions to ignore for equalizer - due to array issues
-eqwalizer({nowarn_function, index_performance_test/0}).
-define(MAX_CACHE_LINES, 31). % Must be less than 128 -define(MAX_CACHE_LINES, 31). % Must be less than 128
-type index_array() :: list(array:array())|[]|none. -type index_array() :: list(array:array(binary()))|none.
-export_type([index_array/0]). -export_type([index_array/0]).
@ -58,7 +61,7 @@ cache_full(L0Cache) ->
length(L0Cache) == ?MAX_CACHE_LINES. length(L0Cache) == ?MAX_CACHE_LINES.
-spec prepare_for_index( -spec prepare_for_index(
array:array(), leveled_codec:segment_hash()) -> array:array(). array:array(binary()), leveled_codec:segment_hash()) -> array:array().
%% @doc %% @doc
%% Add the hash of a key to the index. This is 'prepared' in the sense that %% Add the hash of a key to the index. This is 'prepared' in the sense that
%% this index is not use until it is loaded into the main index. %% this index is not use until it is loaded into the main index.
@ -73,18 +76,22 @@ prepare_for_index(IndexArray, Hash) ->
Bin = array:get(Slot, IndexArray), Bin = array:get(Slot, IndexArray),
array:set(Slot, <<Bin/binary, H0:24/integer>>, IndexArray). array:set(Slot, <<Bin/binary, H0:24/integer>>, IndexArray).
-spec add_to_index(array:array(), index_array(), integer()) -> index_array(). -spec add_to_index(
array:array(binary()), index_array(), integer()) -> index_array().
%% @doc %% @doc
%% Expand the penciller's current index array with the details from a new %% Expand the penciller's current index array with the details from a new
%% ledger cache tree sent from the Bookie. The tree will have a cache slot %% ledger cache tree sent from the Bookie. The tree will have a cache slot
%% which is the index of this ledger_cache in the list of the ledger_caches %% which is the index of this ledger_cache in the list of the ledger_caches
add_to_index(LM1Array, L0Index, CacheSlot) when CacheSlot < 128 -> add_to_index(
LM1Array, L0Index, CacheSlot)
when CacheSlot < 128, L0Index =/= none ->
[LM1Array|L0Index]. [LM1Array|L0Index].
-spec new_index() -> array:array(). -spec new_index() -> array:array(binary()).
%% @doc %% @doc
%% Create a new index array %% Create a new index array
new_index() -> new_index() ->
% eqwalizer:ignore - array does contain binary()
array:new([{size, 256}, {default, <<>>}]). array:new([{size, 256}, {default, <<>>}]).
-spec check_index(leveled_codec:segment_hash(), index_array()) -spec check_index(leveled_codec:segment_hash(), index_array())
@ -92,7 +99,7 @@ new_index() ->
%% @doc %% @doc
%% return a list of positions in the list of cache arrays that may contain the %% return a list of positions in the list of cache arrays that may contain the
%% key associated with the hash being checked %% key associated with the hash being checked
check_index(Hash, L0Index) -> check_index(Hash, L0Index) when L0Index =/= none ->
{Slot, H0} = split_hash(Hash), {Slot, H0} = split_hash(Hash),
{_L, Positions} = {_L, Positions} =
lists:foldl( lists:foldl(
@ -239,19 +246,14 @@ check_slotlist(Key, _Hash, CheckList, TreeList) ->
-include_lib("eunit/include/eunit.hrl"). -include_lib("eunit/include/eunit.hrl").
generate_randomkeys_aslist(Seqn, Count, BucketRangeLow, BucketRangeHigh) -> generate_randomkeys_aslist(Seqn, Count, BucketRangeLow, BucketRangeHigh) ->
lists:ukeysort(1, lists:ukeysort(
generate_randomkeys(Seqn, 1,
Count, generate_randomkeys(Seqn, Count, [], BucketRangeLow, BucketRangeHigh)
[], ).
BucketRangeLow,
BucketRangeHigh)).
generate_randomkeys(Seqn, Count, BucketRangeLow, BucketRangeHigh) -> generate_randomkeys(Seqn, Count, BucketRangeLow, BucketRangeHigh) ->
KVL = generate_randomkeys(Seqn, KVL =
Count, generate_randomkeys(Seqn, Count, [], BucketRangeLow, BucketRangeHigh),
[],
BucketRangeLow,
BucketRangeHigh),
leveled_tree:from_orderedlist(lists:ukeysort(1, KVL), ?CACHE_TYPE). leveled_tree:from_orderedlist(lists:ukeysort(1, KVL), ?CACHE_TYPE).
generate_randomkeys(_Seqn, 0, Acc, _BucketLow, _BucketHigh) -> generate_randomkeys(_Seqn, 0, Acc, _BucketLow, _BucketHigh) ->
@ -260,22 +262,25 @@ generate_randomkeys(Seqn, Count, Acc, BucketLow, BRange) ->
BNumber = BNumber =
lists:flatten( lists:flatten(
io_lib:format("~4..0B", io_lib:format("~4..0B",
[BucketLow + leveled_rand:uniform(BRange)])), [BucketLow + rand:uniform(BRange)])),
KNumber = KNumber =
lists:flatten(io_lib:format("~4..0B", [leveled_rand:uniform(1000)])), lists:flatten(io_lib:format("~4..0B", [rand:uniform(1000)])),
{K, V} = {{o, "Bucket" ++ BNumber, "Key" ++ KNumber, null}, {K, V} =
{Seqn, {active, infinity}, null}}, {
generate_randomkeys(Seqn + 1, {o,
Count - 1, list_to_binary("Bucket" ++ BNumber),
[{K, V}|Acc], list_to_binary("Key" ++ KNumber),
BucketLow, null},
BRange). {Seqn, {active, infinity}, null}
},
generate_randomkeys(Seqn + 1, Count - 1, [{K, V}|Acc], BucketLow, BRange).
compare_method_test() -> compare_method_test() ->
R = lists:foldl(fun(_X, {LedgerSQN, L0Size, L0TreeList}) -> R =
LM1 = generate_randomkeys(LedgerSQN + 1, lists:foldl(
2000, 1, 500), fun(_X, {LedgerSQN, L0Size, L0TreeList}) ->
LM1 = generate_randomkeys(LedgerSQN + 1, 2000, 1, 500),
add_to_cache( add_to_cache(
L0Size, L0Size,
{LM1, LedgerSQN + 1, LedgerSQN + 2000}, {LM1, LedgerSQN + 1, LedgerSQN + 2000},
@ -310,30 +315,39 @@ compare_method_test() ->
end end
end, end,
S0 = lists:foldl(fun({Key, _V}, Acc) -> S0 =
R0 = lists:foldl(FindKeyFun(Key), lists:foldl(
{false, not_found}, fun({Key, _V}, Acc) ->
TreeList), R0 =
[R0|Acc] end, lists:foldl(
FindKeyFun(Key), {false, not_found}, TreeList),
[R0|Acc]
end,
[], [],
TestList), TestList)
,
PosList = lists:seq(1, length(TreeList)), PosList = lists:seq(1, length(TreeList)),
S1 = lists:foldl(fun({Key, _V}, Acc) -> S1 =
lists:foldl(
fun({Key, _V}, Acc) ->
R0 = check_levelzero(Key, PosList, TreeList), R0 = check_levelzero(Key, PosList, TreeList),
[R0|Acc] [R0|Acc]
end, end,
[], [],
TestList), TestList
),
?assertMatch(S0, S1), ?assertMatch(S0, S1),
StartKey = {o, "Bucket0100", null, null}, StartKey = {o, <<"Bucket0100">>, null, null},
EndKey = {o, "Bucket0200", null, null}, EndKey = {o, <<"Bucket0200">>, null, null},
SWa = os:timestamp(), SWa = os:timestamp(),
FetchFun = fun(Slot) -> lists:nth(Slot, TreeList) end, FetchFun = fun(Slot) -> lists:nth(Slot, TreeList) end,
DumpList = to_list(length(TreeList), FetchFun), DumpList = to_list(length(TreeList), FetchFun),
Q0 = lists:foldl(fun({K, V}, Acc) -> Q0 =
lists:foldl(
fun({K, V}, Acc) ->
P = leveled_codec:endkey_passed(EndKey, K), P = leveled_codec:endkey_passed(EndKey, K),
case {K, P} of case {K, P} of
{K, false} when K >= StartKey -> {K, false} when K >= StartKey ->
@ -343,17 +357,19 @@ compare_method_test() ->
end end
end, end,
[], [],
DumpList), DumpList
),
Tree = leveled_tree:from_orderedlist(lists:ukeysort(1, Q0), ?CACHE_TYPE), Tree = leveled_tree:from_orderedlist(lists:ukeysort(1, Q0), ?CACHE_TYPE),
Sz0 = leveled_tree:tsize(Tree), Sz0 = leveled_tree:tsize(Tree),
io:format("Crude method took ~w microseconds resulting in tree of " ++ io:format(
"size ~w~n", "Crude method took ~w microseconds resulting in tree of size ~w~n",
[timer:now_diff(os:timestamp(), SWa), Sz0]), [timer:now_diff(os:timestamp(), SWa), Sz0]
),
SWb = os:timestamp(), SWb = os:timestamp(),
Q1 = merge_trees(StartKey, EndKey, TreeList, leveled_tree:empty(?CACHE_TYPE)), Q1 = merge_trees(StartKey, EndKey, TreeList, leveled_tree:empty(?CACHE_TYPE)),
Sz1 = length(Q1), Sz1 = length(Q1),
io:format("Merge method took ~w microseconds resulting in tree of " ++ io:format(
"size ~w~n", "Merge method took ~w microseconds resulting in tree of size ~w~n",
[timer:now_diff(os:timestamp(), SWb), Sz1]), [timer:now_diff(os:timestamp(), SWb), Sz1]),
?assertMatch(Sz0, Sz1). ?assertMatch(Sz0, Sz1).

28
src/leveled_rand.erl
View file

@ -1,28 +0,0 @@
%% Generalized random module that offers a backwards compatible API
%% around some of the changes in rand, crypto and for time units.
-module(leveled_rand).
%% API
-export([
uniform/0,
uniform/1,
seed/0,
rand_bytes/1
]).
%%%===================================================================
%%% New (r19+) rand style functions
%%%===================================================================
uniform() ->
rand:uniform().
uniform(N) ->
rand:uniform(N).
seed() ->
ok.
rand_bytes(Size) ->
crypto:strong_rand_bytes(Size).

139
src/leveled_runner.erl
View file

@ -39,8 +39,7 @@
-define(CHECKJOURNAL_PROB, 0.2). -define(CHECKJOURNAL_PROB, 0.2).
-type key_range() -type key_range()
:: {leveled_codec:ledger_key()|null, :: {leveled_codec:query_key(), leveled_codec:query_key()}.
leveled_codec:ledger_key()|null}.
-type foldacc() :: any(). -type foldacc() :: any().
% Can't currently be specific about what an acc might be % Can't currently be specific about what an acc might be
@ -56,15 +55,15 @@
:: fun((leveled_codec:key(), leveled_codec:key()) -> accumulate|pass). :: fun((leveled_codec:key(), leveled_codec:key()) -> accumulate|pass).
-type snap_fun() -type snap_fun()
:: fun(() -> {ok, pid(), pid()|null}). :: fun(() -> {ok, pid(), pid()|null, fun(() -> ok)}).
-type runner_fun() -type runner_fun()
:: fun(() -> foldacc()). :: fun(() -> foldacc()).
-type acc_fun() -type objectacc_fun()
:: fun((leveled_codec:key(), any(), foldacc()) -> foldacc()). :: fun((leveled_codec:object_key(), any(), foldacc()) -> foldacc()).
-type mp() -type mp()
:: {re_pattern, term(), term(), term(), term()}. :: {re_pattern, term(), term(), term(), term()}.
-export_type([acc_fun/0, mp/0]). -export_type([fold_keys_fun/0, mp/0]).
%%%============================================================================ %%%============================================================================
%%% External functions %%% External functions
@ -76,8 +75,8 @@
%% @doc %% @doc
%% Fold over a bucket accumulating the count of objects and their total sizes %% Fold over a bucket accumulating the count of objects and their total sizes
bucket_sizestats(SnapFun, Bucket, Tag) -> bucket_sizestats(SnapFun, Bucket, Tag) ->
StartKey = leveled_codec:to_ledgerkey(Bucket, null, Tag), StartKey = leveled_codec:to_querykey(Bucket, null, Tag),
EndKey = leveled_codec:to_ledgerkey(Bucket, null, Tag), EndKey = leveled_codec:to_querykey(Bucket, null, Tag),
AccFun = accumulate_size(), AccFun = accumulate_size(),
Runner = Runner =
fun() -> fun() ->
@ -132,7 +131,7 @@ bucket_list(SnapFun, Tag, FoldBucketsFun, InitAcc, MaxBuckets) ->
-spec index_query(snap_fun(), -spec index_query(snap_fun(),
{leveled_codec:ledger_key(), {leveled_codec:ledger_key(),
leveled_codec:ledger_key(), leveled_codec:ledger_key(),
{boolean(), undefined|mp()|iodata()}}, {boolean(), undefined|mp()}},
{fold_keys_fun(), foldacc()}) {fold_keys_fun(), foldacc()})
-> {async, runner_fun()}. -> {async, runner_fun()}.
%% @doc %% @doc
@ -165,7 +164,7 @@ index_query(SnapFun, {StartKey, EndKey, TermHandling}, FoldAccT) ->
-spec bucketkey_query(snap_fun(), -spec bucketkey_query(snap_fun(),
leveled_codec:tag(), leveled_codec:tag(),
leveled_codec:key()|null, leveled_codec:key()|null,
key_range(), {leveled_codec:single_key()|null, leveled_codec:single_key()|null},
{fold_keys_fun(), foldacc()}, {fold_keys_fun(), foldacc()},
leveled_codec:regular_expression()) leveled_codec:regular_expression())
-> {async, runner_fun()}. -> {async, runner_fun()}.
@ -175,8 +174,8 @@ bucketkey_query(SnapFun, Tag, Bucket,
{StartKey, EndKey}, {StartKey, EndKey},
{FoldKeysFun, InitAcc}, {FoldKeysFun, InitAcc},
TermRegex) -> TermRegex) ->
SK = leveled_codec:to_ledgerkey(Bucket, StartKey, Tag), SK = leveled_codec:to_querykey(Bucket, StartKey, Tag),
EK = leveled_codec:to_ledgerkey(Bucket, EndKey, Tag), EK = leveled_codec:to_querykey(Bucket, EndKey, Tag),
AccFun = accumulate_keys(FoldKeysFun, TermRegex), AccFun = accumulate_keys(FoldKeysFun, TermRegex),
Runner = Runner =
fun() -> fun() ->
@ -203,8 +202,8 @@ bucketkey_query(SnapFun, Tag, Bucket, FunAcc) ->
%% @doc %% @doc
%% Fold over the keys under a given Tag accumulating the hashes %% Fold over the keys under a given Tag accumulating the hashes
hashlist_query(SnapFun, Tag, JournalCheck) -> hashlist_query(SnapFun, Tag, JournalCheck) ->
StartKey = leveled_codec:to_ledgerkey(null, null, Tag), StartKey = leveled_codec:to_querykey(null, null, Tag),
EndKey = leveled_codec:to_ledgerkey(null, null, Tag), EndKey = leveled_codec:to_querykey(null, null, Tag),
Runner = Runner =
fun() -> fun() ->
{ok, LedgerSnapshot, JournalSnapshot, AfterFun} = SnapFun(), {ok, LedgerSnapshot, JournalSnapshot, AfterFun} = SnapFun(),
@ -217,9 +216,10 @@ hashlist_query(SnapFun, Tag, JournalCheck) ->
end, end,
{async, Runner}. {async, Runner}.
-spec tictactree(snap_fun(), -spec tictactree(
snap_fun(),
{leveled_codec:tag(), leveled_codec:key(), tuple()}, {leveled_codec:tag(), leveled_codec:key(), tuple()},
boolean(), atom(), fold_filter_fun()) boolean(), leveled_tictac:tree_size(), fold_filter_fun())
-> {async, runner_fun()}. -> {async, runner_fun()}.
%% @doc %% @doc
%% Return a merkle tree from the fold, directly accessing hashes cached in the %% Return a merkle tree from the fold, directly accessing hashes cached in the
@ -246,14 +246,14 @@ tictactree(SnapFun, {Tag, Bucket, Query}, JournalCheck, TreeSize, Filter) ->
case Tag of case Tag of
?IDX_TAG -> ?IDX_TAG ->
{IdxFld, StartIdx, EndIdx} = Query, {IdxFld, StartIdx, EndIdx} = Query,
KeyDefFun = fun leveled_codec:to_ledgerkey/5, KeyDefFun = fun leveled_codec:to_querykey/5,
{KeyDefFun(Bucket, null, ?IDX_TAG, IdxFld, StartIdx), {KeyDefFun(Bucket, null, ?IDX_TAG, IdxFld, StartIdx),
KeyDefFun(Bucket, null, ?IDX_TAG, IdxFld, EndIdx), KeyDefFun(Bucket, null, ?IDX_TAG, IdxFld, EndIdx),
EnsureKeyBinaryFun}; EnsureKeyBinaryFun};
_ -> _ ->
{StartOKey, EndOKey} = Query, {StartOKey, EndOKey} = Query,
{leveled_codec:to_ledgerkey(Bucket, StartOKey, Tag), {leveled_codec:to_querykey(Bucket, StartOKey, Tag),
leveled_codec:to_ledgerkey(Bucket, EndOKey, Tag), leveled_codec:to_querykey(Bucket, EndOKey, Tag),
fun(K, H) -> fun(K, H) ->
V = {is_hash, H}, V = {is_hash, H},
EnsureKeyBinaryFun(K, V) EnsureKeyBinaryFun(K, V)
@ -279,8 +279,8 @@ tictactree(SnapFun, {Tag, Bucket, Query}, JournalCheck, TreeSize, Filter) ->
%% function to each proxy object %% function to each proxy object
foldheads_allkeys(SnapFun, Tag, FoldFun, JournalCheck, foldheads_allkeys(SnapFun, Tag, FoldFun, JournalCheck,
SegmentList, LastModRange, MaxObjectCount) -> SegmentList, LastModRange, MaxObjectCount) ->
StartKey = leveled_codec:to_ledgerkey(null, null, Tag), StartKey = leveled_codec:to_querykey(null, null, Tag),
EndKey = leveled_codec:to_ledgerkey(null, null, Tag), EndKey = leveled_codec:to_querykey(null, null, Tag),
foldobjects(SnapFun, foldobjects(SnapFun,
Tag, Tag,
[{StartKey, EndKey}], [{StartKey, EndKey}],
@ -298,8 +298,8 @@ foldheads_allkeys(SnapFun, Tag, FoldFun, JournalCheck,
%% @doc %% @doc
%% Fold over all objects for a given tag %% Fold over all objects for a given tag
foldobjects_allkeys(SnapFun, Tag, FoldFun, key_order) -> foldobjects_allkeys(SnapFun, Tag, FoldFun, key_order) ->
StartKey = leveled_codec:to_ledgerkey(null, null, Tag), StartKey = leveled_codec:to_querykey(null, null, Tag),
EndKey = leveled_codec:to_ledgerkey(null, null, Tag), EndKey = leveled_codec:to_querykey(null, null, Tag),
foldobjects(SnapFun, foldobjects(SnapFun,
Tag, Tag,
[{StartKey, EndKey}], [{StartKey, EndKey}],
@ -335,7 +335,7 @@ foldobjects_allkeys(SnapFun, Tag, FoldObjectsFun, sqn_order) ->
_ -> _ ->
{VBin, _VSize} = ExtractFun(JVal), {VBin, _VSize} = ExtractFun(JVal),
{Obj, _IdxSpecs} = {Obj, _IdxSpecs} =
leveled_codec:split_inkvalue(VBin), leveled_codec:revert_value_from_journal(VBin),
ToLoop = ToLoop =
case SQN of case SQN of
MaxSQN -> stop; MaxSQN -> stop;
@ -353,11 +353,16 @@ foldobjects_allkeys(SnapFun, Tag, FoldObjectsFun, sqn_order) ->
Folder = Folder =
fun() -> fun() ->
{ok, LedgerSnapshot, JournalSnapshot, AfterFun} = SnapFun(), {ok, LedgerSnapshot, JournalSnapshot, AfterFun} =
{ok, JournalSQN} = leveled_inker:ink_getjournalsqn(JournalSnapshot), case SnapFun() of
{ok, LS, JS, AF} when is_pid(JS) ->
{ok, LS, JS, AF}
end,
{ok, JournalSQN} =
leveled_inker:ink_getjournalsqn(JournalSnapshot),
IsValidFun = IsValidFun =
fun(Bucket, Key, SQN) -> fun(Bucket, Key, SQN) ->
LedgerKey = leveled_codec:to_ledgerkey(Bucket, Key, Tag), LedgerKey = leveled_codec:to_objectkey(Bucket, Key, Tag),
CheckSQN = CheckSQN =
leveled_penciller:pcl_checksequencenumber( leveled_penciller:pcl_checksequencenumber(
LedgerSnapshot, LedgerKey, SQN), LedgerSnapshot, LedgerKey, SQN),
@ -438,9 +443,9 @@ foldheads_bybucket(SnapFun,
%% and passing those objects into the fold function %% and passing those objects into the fold function
foldobjects_byindex(SnapFun, {Tag, Bucket, Field, FromTerm, ToTerm}, FoldFun) -> foldobjects_byindex(SnapFun, {Tag, Bucket, Field, FromTerm, ToTerm}, FoldFun) ->
StartKey = StartKey =
leveled_codec:to_ledgerkey(Bucket, null, ?IDX_TAG, Field, FromTerm), leveled_codec:to_querykey(Bucket, null, ?IDX_TAG, Field, FromTerm),
EndKey = EndKey =
leveled_codec:to_ledgerkey(Bucket, null, ?IDX_TAG, Field, ToTerm), leveled_codec:to_querykey(Bucket, null, ?IDX_TAG, Field, ToTerm),
foldobjects(SnapFun, foldobjects(SnapFun,
Tag, Tag,
[{StartKey, EndKey}], [{StartKey, EndKey}],
@ -457,38 +462,39 @@ foldobjects_byindex(SnapFun, {Tag, Bucket, Field, FromTerm, ToTerm}, FoldFun) ->
get_nextbucket(_NextB, _NextK, _Tag, _LS, BKList, {Limit, Limit}) -> get_nextbucket(_NextB, _NextK, _Tag, _LS, BKList, {Limit, Limit}) ->
lists:reverse(BKList); lists:reverse(BKList);
get_nextbucket(NextBucket, NextKey, Tag, LedgerSnapshot, BKList, {C, L}) -> get_nextbucket(NextBucket, NextKey, Tag, LedgerSnapshot, BKList, {C, L}) ->
StartKey = leveled_codec:to_ledgerkey(NextBucket, NextKey, Tag), StartKey = leveled_codec:to_querykey(NextBucket, NextKey, Tag),
EndKey = leveled_codec:to_ledgerkey(null, null, Tag), EndKey = leveled_codec:to_querykey(null, null, Tag),
ExtractFun = ExtractFun =
fun(LK, V, _Acc) -> fun(LK, V, _Acc) ->
{leveled_codec:from_ledgerkey(LK), V} {leveled_codec:from_ledgerkey(LK), V}
end, end,
R = leveled_penciller:pcl_fetchnextkey(LedgerSnapshot, R =
StartKey, leveled_penciller:pcl_fetchnextkey(
EndKey, LedgerSnapshot, StartKey, EndKey, ExtractFun, null),
ExtractFun,
null),
case R of case R of
{1, null} -> {1, null} ->
leveled_log:log(b0008,[]), leveled_log:log(b0008,[]),
BKList; BKList;
{0, {{B, K}, _V}} -> {0, {{B, K}, _V}} when is_binary(B); is_tuple(B) ->
leveled_log:log(b0009,[B]), leveled_log:log(b0009,[B]),
get_nextbucket(leveled_codec:next_key(B), get_nextbucket(
leveled_codec:next_key(B),
null, null,
Tag, Tag,
LedgerSnapshot, LedgerSnapshot,
[{B, K}|BKList], [{B, K}|BKList],
{C + 1, L}) {C + 1, L}
)
end. end.
-spec foldobjects(snap_fun(), -spec foldobjects(
snap_fun(),
atom(), atom(),
list(), list(),
fold_objects_fun()|{fold_objects_fun(), foldacc()}, fold_objects_fun()|{fold_objects_fun(), foldacc()},
false|{true, boolean()}, false|list(integer())) -> false|{true, boolean()}, false|list(integer()))
{async, runner_fun()}. -> {async, runner_fun()}.
foldobjects(SnapFun, Tag, KeyRanges, FoldObjFun, DeferredFetch, SegmentList) -> foldobjects(SnapFun, Tag, KeyRanges, FoldObjFun, DeferredFetch, SegmentList) ->
foldobjects(SnapFun, Tag, KeyRanges, foldobjects(SnapFun, Tag, KeyRanges,
FoldObjFun, DeferredFetch, SegmentList, false, false). FoldObjFun, DeferredFetch, SegmentList, false, false).
@ -534,14 +540,16 @@ foldobjects(SnapFun, Tag, KeyRanges, FoldObjFun, DeferredFetch,
FoldFun, JournalSnapshot, Tag, DeferredFetch), FoldFun, JournalSnapshot, Tag, DeferredFetch),
FoldFunGen = FoldFunGen =
fun({StartKey, EndKey}, FoldAcc) -> fun({StartKey, EndKey}, FoldAcc) ->
leveled_penciller:pcl_fetchkeysbysegment(LedgerSnapshot, leveled_penciller:pcl_fetchkeysbysegment(
LedgerSnapshot,
StartKey, StartKey,
EndKey, EndKey,
AccFun, AccFun,
FoldAcc, FoldAcc,
SegmentList, SegmentList,
LastModRange, LastModRange,
LimitByCount) LimitByCount
)
end, end,
ListFoldFun = ListFoldFun =
fun(KeyRange, Acc) -> fun(KeyRange, Acc) ->
@ -567,9 +575,7 @@ accumulate_hashes(JournalCheck, InkerClone) ->
fun(B, K, H, Acc) -> fun(B, K, H, Acc) ->
[{B, K, H}|Acc] [{B, K, H}|Acc]
end, end,
get_hashaccumulator(JournalCheck, get_hashaccumulator(JournalCheck, InkerClone, AddKeyFun).
InkerClone,
AddKeyFun).
accumulate_tree(FilterFun, JournalCheck, InkerClone, HashFun) -> accumulate_tree(FilterFun, JournalCheck, InkerClone, HashFun) ->
AddKeyFun = AddKeyFun =
@ -581,15 +587,13 @@ accumulate_tree(FilterFun, JournalCheck, InkerClone, HashFun) ->
Tree Tree
end end
end, end,
get_hashaccumulator(JournalCheck, get_hashaccumulator(JournalCheck, InkerClone, AddKeyFun).
InkerClone,
AddKeyFun).
get_hashaccumulator(JournalCheck, InkerClone, AddKeyFun) -> get_hashaccumulator(JournalCheck, InkerClone, AddKeyFun) ->
AccFun = AccFun =
fun(LK, V, Acc) -> fun(LK, V, Acc) ->
{B, K, H} = leveled_codec:get_keyandobjhash(LK, V), {B, K, H} = leveled_codec:get_keyandobjhash(LK, V),
Check = leveled_rand:uniform() < ?CHECKJOURNAL_PROB, Check = rand:uniform() < ?CHECKJOURNAL_PROB,
case JournalCheck and Check of case JournalCheck and Check of
true -> true ->
case check_presence(LK, V, InkerClone) of case check_presence(LK, V, InkerClone) of
@ -604,11 +608,11 @@ get_hashaccumulator(JournalCheck, InkerClone, AddKeyFun) ->
end, end,
AccFun. AccFun.
-spec accumulate_objects(fold_objects_fun(), -spec accumulate_objects
pid()|null, (fold_objects_fun(), pid(), leveled_head:object_tag(), false|{true, boolean()})
leveled_codec:tag(), -> objectacc_fun();
false|{true, boolean()}) (fold_objects_fun(), null, leveled_head:headonly_tag(), {true, false})
-> acc_fun(). -> objectacc_fun().
accumulate_objects(FoldObjectsFun, InkerClone, Tag, DeferredFetch) -> accumulate_objects(FoldObjectsFun, InkerClone, Tag, DeferredFetch) ->
AccFun = AccFun =
fun(LK, V, Acc) -> fun(LK, V, Acc) ->
@ -630,24 +634,23 @@ accumulate_objects(FoldObjectsFun, InkerClone, Tag, DeferredFetch) ->
{B0, K0, _T0} -> {B0, K0, _T0} ->
{B0, K0} {B0, K0}
end, end,
JK = {leveled_codec:to_ledgerkey(B, K, Tag), SQN}, JK = {leveled_codec:to_objectkey(B, K, Tag), SQN},
case DeferredFetch of case DeferredFetch of
{true, JournalCheck} -> {true, JournalCheck} when MD =/= null ->
ProxyObj = ProxyObj =
leveled_codec:return_proxy(Tag, MD, InkerClone, JK), leveled_codec:return_proxy(Tag, MD, InkerClone, JK),
case JournalCheck of case {JournalCheck, InkerClone} of
true -> {true, InkerClone} when is_pid(InkerClone) ->
InJournal = InJournal =
leveled_inker:ink_keycheck(InkerClone, leveled_inker:ink_keycheck(
LK, InkerClone, LK, SQN),
SQN),
case InJournal of case InJournal of
probably -> probably ->
FoldObjectsFun(B, K, ProxyObj, Acc); FoldObjectsFun(B, K, ProxyObj, Acc);
missing -> missing ->
Acc Acc
end; end;
false -> {false, _} ->
FoldObjectsFun(B, K, ProxyObj, Acc) FoldObjectsFun(B, K, ProxyObj, Acc)
end; end;
false -> false ->
@ -730,10 +733,10 @@ throw_test() ->
fun() -> fun() ->
error error
end, end,
?assertMatch({ok, ['1']}, ?assertMatch({ok, ['1']}, wrap_runner(CompletedFolder, AfterAction)),
wrap_runner(CompletedFolder, AfterAction)), ?assertException(
?assertException(throw, stop_fold, throw, stop_fold, wrap_runner(StoppedFolder, AfterAction)
wrap_runner(StoppedFolder, AfterAction)). ).
-endif. -endif.

957
src/leveled_sst.erl
View file

File diff suppressed because it is too large Load diff

49
src/leveled_tictac.erl
View file

@ -101,7 +101,7 @@
width :: integer(), width :: integer(),
segment_count :: integer(), segment_count :: integer(),
level1 :: level1_map(), level1 :: level1_map(),
level2 :: array:array() level2 :: array:array(binary())
}). }).
-type level1_map() :: #{non_neg_integer() => binary()}|binary(). -type level1_map() :: #{non_neg_integer() => binary()}|binary().
@ -161,6 +161,8 @@ new_tree(TreeID, Size, UseMap) ->
width = Width, width = Width,
segment_count = Width * ?L2_CHUNKSIZE, segment_count = Width * ?L2_CHUNKSIZE,
level1 = Lv1Init, level1 = Lv1Init,
% array values are indeed all binaries
% eqwalizer:ignore
level2 = Lv2Init level2 = Lv2Init
}. }.
@ -196,13 +198,16 @@ import_tree(ExportedTree) ->
[{<<"level1">>, L1Base64}, [{<<"level1">>, L1Base64},
{<<"level2">>, {struct, L2List}}]} = ExportedTree, {<<"level2">>, {struct, L2List}}]} = ExportedTree,
L1Bin = base64:decode(L1Base64), L1Bin = base64:decode(L1Base64),
Sizes = lists:map(fun(SizeTag) -> {SizeTag, get_size(SizeTag)} end, Sizes =
?VALID_SIZES), lists:map(
fun(SizeTag) -> {SizeTag, get_size(SizeTag)} end,
?VALID_SIZES
),
Width = byte_size(L1Bin) div ?HASH_SIZE, Width = byte_size(L1Bin) div ?HASH_SIZE,
{Size, _Width} = lists:keyfind(Width, 2, Sizes), {Size, _Width} = lists:keyfind(Width, 2, Sizes),
%% assert that side is indeed the provided width %% assert that side is indeed the provided width
true = get_size(Size) == Width, true = get_size(Size) == Width,
Lv2Init = array:new([{size, Width}]), Lv2Init = array:new([{size, Width}, {default, ?EMPTY}]),
FoldFun = FoldFun =
fun({X, EncodedL2SegBin}, L2Array) -> fun({X, EncodedL2SegBin}, L2Array) ->
L2SegBin = zlib:uncompress(base64:decode(EncodedL2SegBin)), L2SegBin = zlib:uncompress(base64:decode(EncodedL2SegBin)),
@ -216,6 +221,8 @@ import_tree(ExportedTree) ->
width = Width, width = Width,
segment_count = Width * ?L2_CHUNKSIZE, segment_count = Width * ?L2_CHUNKSIZE,
level1 = to_level1_map(L1Bin), level1 = to_level1_map(L1Bin),
% array values are indeed all binaries
% eqwalizer:ignore
level2 = Lv2 level2 = Lv2
}. }.
@ -229,12 +236,14 @@ import_tree(ExportedTree) ->
add_kv(TicTacTree, Key, Value, BinExtractFun) -> add_kv(TicTacTree, Key, Value, BinExtractFun) ->
add_kv(TicTacTree, Key, Value, BinExtractFun, false). add_kv(TicTacTree, Key, Value, BinExtractFun, false).
-spec add_kv( -spec add_kv
tictactree(), term(), term(), bin_extract_fun(), boolean()) (tictactree(), term(), term(), bin_extract_fun(), true) ->
-> tictactree()|{tictactree(), integer()}. {tictactree(), integer()};
(tictactree(), term(), term(), bin_extract_fun(), false) ->
tictactree().
%% @doc %% @doc
%% add_kv with ability to return segment ID of Key added %% add_kv with ability to return segment ID of Key added
add_kv(TicTacTree, Key, Value, BinExtractFun, ReturnSegment) -> add_kv(TicTacTree, Key, Value, BinExtractFun, true) ->
{BinK, BinV} = BinExtractFun(Key, Value), {BinK, BinV} = BinExtractFun(Key, Value),
{SegHash, SegChangeHash} = tictac_hash(BinK, BinV), {SegHash, SegChangeHash} = tictac_hash(BinK, BinV),
Segment = get_segment(SegHash, TicTacTree#tictactree.segment_count), Segment = get_segment(SegHash, TicTacTree#tictactree.segment_count),
@ -249,12 +258,9 @@ add_kv(TicTacTree, Key, Value, BinExtractFun, ReturnSegment) ->
replace_segment( replace_segment(
SegLeaf1Upd, SegLeaf2Upd, L1Extract, L2Extract, TicTacTree SegLeaf1Upd, SegLeaf2Upd, L1Extract, L2Extract, TicTacTree
), ),
case ReturnSegment of
true ->
{UpdatedTree, Segment}; {UpdatedTree, Segment};
false -> add_kv(TicTacTree, Key, Value, BinExtractFun, false) ->
UpdatedTree element(1, add_kv(TicTacTree, Key, Value, BinExtractFun, true)).
end.
-spec alter_segment(integer(), integer(), tictactree()) -> tictactree(). -spec alter_segment(integer(), integer(), tictactree()) -> tictactree().
%% @doc %% @doc
@ -373,16 +379,13 @@ get_segment(Hash, TreeSize) ->
%% has already been taken. If the value is not a pre-extracted hash just use %% has already been taken. If the value is not a pre-extracted hash just use
%% erlang:phash2. If an exportable hash of the value is required this should %% erlang:phash2. If an exportable hash of the value is required this should
%% be managed through the add_kv ExtractFun providing a pre-prepared Hash. %% be managed through the add_kv ExtractFun providing a pre-prepared Hash.
tictac_hash(BinKey, Val) when is_binary(BinKey) -> tictac_hash(
BinKey, {is_hash, HashedVal})
when is_binary(BinKey), is_integer(HashedVal) ->
{HashKeyToSeg, AltHashKey} = keyto_doublesegment32(BinKey), {HashKeyToSeg, AltHashKey} = keyto_doublesegment32(BinKey),
HashVal = {HashKeyToSeg, AltHashKey bxor HashedVal};
case Val of tictac_hash(BinKey, ValToHash) when is_binary(BinKey) ->
{is_hash, HashedVal} -> tictac_hash(BinKey, {is_hash, erlang:phash2(ValToHash)}).
HashedVal;
_ ->
erlang:phash2(Val)
end,
{HashKeyToSeg, AltHashKey bxor HashVal}.
-spec keyto_doublesegment32( -spec keyto_doublesegment32(
binary()) -> {non_neg_integer(), non_neg_integer()}. binary()) -> {non_neg_integer(), non_neg_integer()}.
@ -961,7 +964,7 @@ timing_test() ->
timing_tester(KeyCount, SegCount, SmallSize, LargeSize) -> timing_tester(KeyCount, SegCount, SmallSize, LargeSize) ->
SegList = SegList =
lists:map(fun(_C) -> lists:map(fun(_C) ->
leveled_rand:uniform(get_size(SmallSize) * ?L2_CHUNKSIZE - 1) rand:uniform(get_size(SmallSize) * ?L2_CHUNKSIZE - 1)
end, end,
lists:seq(1, SegCount)), lists:seq(1, SegCount)),
KeyToSegFun = KeyToSegFun =

30
src/leveled_tree.erl
View file

@ -29,9 +29,7 @@
-define(SKIP_WIDTH, 16). -define(SKIP_WIDTH, 16).
-type tree_type() :: tree|idxt|skpl. -type tree_type() :: tree|idxt|skpl.
-type leveled_tree() :: {tree_type(), -type leveled_tree() :: {tree_type(), integer(), any()}.
integer(), % length
any()}.
-export_type([leveled_tree/0]). -export_type([leveled_tree/0]).
@ -104,7 +102,7 @@ match(Key, {tree, _L, Tree}) ->
{_NK, SL, _Iter} -> {_NK, SL, _Iter} ->
lookup_match(Key, SL) lookup_match(Key, SL)
end; end;
match(Key, {idxt, _L, {TLI, IDX}}) -> match(Key, {idxt, _L, {TLI, IDX}}) when is_tuple(TLI) ->
Iter = tree_iterator_from(Key, IDX), Iter = tree_iterator_from(Key, IDX),
case tree_next(Iter) of case tree_next(Iter) of
none -> none ->
@ -136,7 +134,7 @@ search(Key, {tree, _L, Tree}, StartKeyFun) ->
{K, V} {K, V}
end end
end; end;
search(Key, {idxt, _L, {TLI, IDX}}, StartKeyFun) -> search(Key, {idxt, _L, {TLI, IDX}}, StartKeyFun) when is_tuple(TLI) ->
Iter = tree_iterator_from(Key, IDX), Iter = tree_iterator_from(Key, IDX),
case tree_next(Iter) of case tree_next(Iter) of
none -> none ->
@ -235,14 +233,13 @@ to_list({tree, _L, Tree}) ->
Acc ++ SL Acc ++ SL
end, end,
lists:foldl(FoldFun, [], tree_to_list(Tree)); lists:foldl(FoldFun, [], tree_to_list(Tree));
to_list({idxt, _L, {TLI, _IDX}}) -> to_list({idxt, _L, {TLI, _IDX}}) when is_tuple(TLI) ->
lists:append(tuple_to_list(TLI)); lists:append(tuple_to_list(TLI));
to_list({skpl, _L, SkipList}) -> to_list({skpl, _L, SkipList}) when is_list(SkipList) ->
FoldFun = FoldFun =
fun({_M, SL}, Acc) -> fun({_M, SL}, Acc) ->
[SL|Acc] [SL|Acc]
end, end,
Lv1List = lists:reverse(lists:foldl(FoldFun, [], SkipList)), Lv1List = lists:reverse(lists:foldl(FoldFun, [], SkipList)),
Lv0List = lists:reverse(lists:foldl(FoldFun, [], lists:append(Lv1List))), Lv0List = lists:reverse(lists:foldl(FoldFun, [], lists:append(Lv1List))),
lists:append(Lv0List). lists:append(Lv0List).
@ -580,13 +577,13 @@ generate_randomkeys(Seqn, Count, BucketRangeLow, BucketRangeHigh) ->
generate_randomkeys(_Seqn, 0, Acc, _BucketLow, _BucketHigh) -> generate_randomkeys(_Seqn, 0, Acc, _BucketLow, _BucketHigh) ->
Acc; Acc;
generate_randomkeys(Seqn, Count, Acc, BucketLow, BRange) -> generate_randomkeys(Seqn, Count, Acc, BucketLow, BRange) ->
BRand = leveled_rand:uniform(BRange), BRand = rand:uniform(BRange),
BNumber = BNumber =
lists:flatten( lists:flatten(
io_lib:format("K~4..0B", [BucketLow + BRand])), io_lib:format("K~4..0B", [BucketLow + BRand])),
KNumber = KNumber =
lists:flatten( lists:flatten(
io_lib:format("K~8..0B", [leveled_rand:uniform(1000)])), io_lib:format("K~8..0B", [rand:uniform(1000)])),
{K, V} = {K, V} =
{{o_kv, {{o_kv,
{<<"btype">>, list_to_binary("Bucket" ++ BNumber)}, {<<"btype">>, list_to_binary("Bucket" ++ BNumber)},
@ -608,7 +605,7 @@ generate_simplekeys(Seqn, Count, Acc) ->
KNumber = KNumber =
list_to_binary( list_to_binary(
lists:flatten( lists:flatten(
io_lib:format("K~8..0B", [leveled_rand:uniform(100000)]))), io_lib:format("K~8..0B", [rand:uniform(100000)]))),
generate_simplekeys(Seqn + 1, Count - 1, [{KNumber, Seqn}|Acc]). generate_simplekeys(Seqn + 1, Count - 1, [{KNumber, Seqn}|Acc]).
@ -958,14 +955,13 @@ search_range_idx_test() ->
{o_rkv,"Bucket1","Key1",null}, {o_rkv,"Bucket1","Key1",null},
"<0.320.0>","./16_1_6.sst", none}}]}, "<0.320.0>","./16_1_6.sst", none}}]},
{1,{{o_rkv,"Bucket1","Key1",null},1,nil,nil}}}}, {1,{{o_rkv,"Bucket1","Key1",null},1,nil,nil}}}},
StartKeyFun = R =
fun(ME) -> search_range(
ME#manifest_entry.start_key {o_rkv, "Bucket", null, null},
end,
R = search_range({o_rkv, "Bucket", null, null},
{o_rkv, "Bucket", null, null}, {o_rkv, "Bucket", null, null},
Tree, Tree,
StartKeyFun), fun leveled_pmanifest:entry_startkey/1
),
?assertMatch(1, length(R)). ?assertMatch(1, length(R)).
-endif. -endif.

2
src/leveled_util.erl
View file

@ -23,7 +23,7 @@
%% Credit to %% Credit to
%% https://github.com/afiskon/erlang-uuid-v4/blob/master/src/uuid.erl %% https://github.com/afiskon/erlang-uuid-v4/blob/master/src/uuid.erl
generate_uuid() -> generate_uuid() ->
<<A:32, B:16, C:16, D:16, E:48>> = leveled_rand:rand_bytes(16), <<A:32, B:16, C:16, D:16, E:48>> = crypto:strong_rand_bytes(16),
L = io_lib:format("~8.16.0b-~4.16.0b-4~3.16.0b-~4.16.0b-~12.16.0b", L = io_lib:format("~8.16.0b-~4.16.0b-4~3.16.0b-~4.16.0b-~12.16.0b",
[A, B, C band 16#0fff, D band 16#3fff bor 16#8000, E]), [A, B, C band 16#0fff, D band 16#3fff bor 16#8000, E]),
binary_to_list(list_to_binary(L)). binary_to_list(list_to_binary(L)).

4
test/end_to_end/appdefined_SUITE.erl
View file

@ -73,7 +73,7 @@ application_defined_tag_tester(KeyCount, Tag, Functions, ExpectMD) ->
[{bespoke_tag1, retain}, {bespoke_tag2, retain}]}, [{bespoke_tag1, retain}, {bespoke_tag2, retain}]},
{override_functions, Functions}], {override_functions, Functions}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1), {ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
Value = leveled_rand:rand_bytes(512), Value = crypto:strong_rand_bytes(512),
MapFun = MapFun =
fun(C) -> fun(C) ->
{C, object_generator(C, Value)} {C, object_generator(C, Value)}
@ -119,7 +119,7 @@ application_defined_tag_tester(KeyCount, Tag, Functions, ExpectMD) ->
object_generator(Count, V) -> object_generator(Count, V) ->
Hash = erlang:phash2({count, V}), Hash = erlang:phash2({count, V}),
Random = leveled_rand:uniform(1000), Random = rand:uniform(1000),
Key = list_to_binary(leveled_util:generate_uuid()), Key = list_to_binary(leveled_util:generate_uuid()),
Bucket = <<"B">>, Bucket = <<"B">>,
{Bucket, {Bucket,

263
test/end_to_end/basic_SUITE.erl
View file

@ -55,15 +55,19 @@ simple_put_fetch_head_delete(_Config) ->
simple_test_withlog(LogLevel, ForcedLogs) -> simple_test_withlog(LogLevel, ForcedLogs) ->
RootPath = testutil:reset_filestructure(), RootPath = testutil:reset_filestructure(),
StartOpts1 = [{root_path, RootPath}, StartOpts1 =
[
{root_path, RootPath},
{sync_strategy, testutil:sync_strategy()}, {sync_strategy, testutil:sync_strategy()},
{log_level, LogLevel}, {log_level, LogLevel},
{forced_logs, ForcedLogs}], {forced_logs, ForcedLogs},
{max_pencillercachesize, 200}
],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1), {ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{TestObject, TestSpec} = testutil:generate_testobject(), {TestObject, TestSpec} = testutil:generate_testobject(),
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec), ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
testutil:check_formissingobject(Bookie1, "Bucket1", "Key2"), testutil:check_formissingobject(Bookie1, <<"Bucket1">>, <<"Key2">>),
ok = leveled_bookie:book_close(Bookie1), ok = leveled_bookie:book_close(Bookie1),
StartOpts2 = [{root_path, RootPath}, StartOpts2 = [{root_path, RootPath},
{max_journalsize, 3000000}, {max_journalsize, 3000000},
@ -78,29 +82,49 @@ simple_test_withlog(LogLevel, ForcedLogs) ->
ChkList1 = lists:sublist(lists:sort(ObjList1), 100), ChkList1 = lists:sublist(lists:sort(ObjList1), 100),
testutil:check_forlist(Bookie2, ChkList1), testutil:check_forlist(Bookie2, ChkList1),
testutil:check_forobject(Bookie2, TestObject), testutil:check_forobject(Bookie2, TestObject),
testutil:check_formissingobject(Bookie2, "Bucket1", "Key2"), testutil:check_formissingobject(Bookie2, <<"Bucket1">>, <<"Key2">>),
ok = leveled_bookie:book_put(Bookie2, "Bucket1", "Key2", "Value2", ok =
[{add, "Index1", "Term1"}]), leveled_bookie:book_put(
{ok, "Value2"} = leveled_bookie:book_get(Bookie2, "Bucket1", "Key2"), Bookie2,
{ok, {62888926, S, undefined}} = <<"Bucket1">>,
leveled_bookie:book_head(Bookie2, "Bucket1", "Key2"), <<"Key2">>,
true = (S == 58) or (S == 60), <<"Value2">>,
[{add, <<"Index1">>, <<"Term1">>}]
),
{ok, <<"Value2">>} =
leveled_bookie:book_get(Bookie2, <<"Bucket1">>, <<"Key2">>),
{ok, {2220864, S, undefined}} =
leveled_bookie:book_head(Bookie2, <<"Bucket1">>, <<"Key2">>),
true = (S == 63) or (S == 65),
% After OTP 26 the object is 58 bytes not 60 % After OTP 26 the object is 58 bytes not 60
testutil:check_formissingobject(Bookie2, "Bucket1", "Key2"), testutil:check_formissingobject(Bookie2, <<"Bucket1">>, <<"Key2">>),
ok = leveled_bookie:book_put(Bookie2, "Bucket1", "Key2", <<"Value2">>, ok =
[{remove, "Index1", "Term1"}, leveled_bookie:book_put(
{add, "Index1", <<"Term2">>}]), Bookie2,
{ok, <<"Value2">>} = leveled_bookie:book_get(Bookie2, "Bucket1", "Key2"), <<"Bucket1">>,
<<"Key2">>,
<<"Value2">>,
[{remove, <<"Index1">>, <<"Term1">>},
{add, <<"Index1">>, <<"Term2">>}]
),
{ok, <<"Value2">>} =
leveled_bookie:book_get(Bookie2, <<"Bucket1">>, <<"Key2">>),
ok = leveled_bookie:book_close(Bookie2), ok = leveled_bookie:book_close(Bookie2),
{ok, Bookie3} = leveled_bookie:book_start(StartOpts2), {ok, Bookie3} = leveled_bookie:book_start(StartOpts2),
{ok, <<"Value2">>} = leveled_bookie:book_get(Bookie3, "Bucket1", "Key2"), {ok, <<"Value2">>} =
ok = leveled_bookie:book_delete(Bookie3, "Bucket1", "Key2", leveled_bookie:book_get(Bookie3, <<"Bucket1">>, <<"Key2">>),
[{remove, "Index1", "Term1"}]), ok =
not_found = leveled_bookie:book_get(Bookie3, "Bucket1", "Key2"), leveled_bookie:book_delete(
not_found = leveled_bookie:book_head(Bookie3, "Bucket1", "Key2"), Bookie3,
<<"Bucket1">>,
<<"Key2">>,
[{remove, <<"Index1">>, <<"Term1">>}]
),
not_found = leveled_bookie:book_get(Bookie3, <<"Bucket1">>, <<"Key2">>),
not_found = leveled_bookie:book_head(Bookie3, <<"Bucket1">>, <<"Key2">>),
ok = leveled_bookie:book_close(Bookie3), ok = leveled_bookie:book_close(Bookie3),
{ok, Bookie4} = leveled_bookie:book_start(StartOpts2), {ok, Bookie4} = leveled_bookie:book_start(StartOpts2),
not_found = leveled_bookie:book_get(Bookie4, "Bucket1", "Key2"), not_found = leveled_bookie:book_get(Bookie4, <<"Bucket1">>, <<"Key2">>),
ok = leveled_bookie:book_destroy(Bookie4). ok = leveled_bookie:book_destroy(Bookie4).
many_put_fetch_head(_Config) -> many_put_fetch_head(_Config) ->
@ -168,7 +192,7 @@ many_put_fetch_head(_Config) ->
not_found = leveled_bookie:book_sqn(Bookie3, not_found = leveled_bookie:book_sqn(Bookie3,
testutil:get_bucket(TestObject), testutil:get_bucket(TestObject),
testutil:get_key(TestObject)), testutil:get_key(TestObject)),
testutil:check_formissingobject(Bookie3, "Bookie1", "MissingKey0123"), testutil:check_formissingobject(Bookie3, <<"Bookie1">>, <<"MissingKey0123">>),
ok = leveled_bookie:book_destroy(Bookie3). ok = leveled_bookie:book_destroy(Bookie3).
bigjournal_littlejournal(_Config) -> bigjournal_littlejournal(_Config) ->
@ -181,7 +205,7 @@ bigjournal_littlejournal(_Config) ->
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1), {ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
ObjL1 = ObjL1 =
testutil:generate_objects(100, 1, [], testutil:generate_objects(100, 1, [],
leveled_rand:rand_bytes(10000), crypto:strong_rand_bytes(10000),
fun() -> [] end, <<"B">>), fun() -> [] end, <<"B">>),
testutil:riakload(Bookie1, ObjL1), testutil:riakload(Bookie1, ObjL1),
ok = leveled_bookie:book_close(Bookie1), ok = leveled_bookie:book_close(Bookie1),
@ -189,7 +213,7 @@ bigjournal_littlejournal(_Config) ->
{ok, Bookie2} = leveled_bookie:book_start(StartOpts2), {ok, Bookie2} = leveled_bookie:book_start(StartOpts2),
ObjL2 = ObjL2 =
testutil:generate_objects(10, 1000, [], testutil:generate_objects(10, 1000, [],
leveled_rand:rand_bytes(10000), crypto:strong_rand_bytes(10000),
fun() -> [] end, <<"B">>), fun() -> [] end, <<"B">>),
testutil:riakload(Bookie2, ObjL2), testutil:riakload(Bookie2, ObjL2),
testutil:check_forlist(Bookie2, ObjL1), testutil:check_forlist(Bookie2, ObjL1),
@ -214,7 +238,7 @@ bigsst_littlesst(_Config) ->
100000, 100000,
1, 1,
[], [],
leveled_rand:rand_bytes(100), crypto:strong_rand_bytes(100),
fun() -> [] end, fun() -> [] end,
<<"B">>) <<"B">>)
), ),
@ -260,13 +284,16 @@ journal_compaction_tester(Restart, WRP) ->
ChkList1 = lists:sublist(lists:sort(ObjList1), 10000), ChkList1 = lists:sublist(lists:sort(ObjList1), 10000),
testutil:check_forlist(Bookie0, ChkList1), testutil:check_forlist(Bookie0, ChkList1),
testutil:check_forobject(Bookie0, TestObject), testutil:check_forobject(Bookie0, TestObject),
{B2, K2, V2, Spec2, MD} = {"Bucket2", {B2, K2, V2, Spec2, MD} =
"Key2", {
"Value2", <<"Bucket2">>,
<<"Key2">>,
<<"Value2">>,
[], [],
[{"MDK2", "MDV2"}]}, [{<<"MDK2">>, <<"MDV2">>}]
{TestObject2, TestSpec2} = testutil:generate_testobject(B2, K2, },
V2, Spec2, MD), {TestObject2, TestSpec2} =
testutil:generate_testobject(B2, K2, V2, Spec2, MD),
ok = testutil:book_riakput(Bookie0, TestObject2, TestSpec2), ok = testutil:book_riakput(Bookie0, TestObject2, TestSpec2),
ok = leveled_bookie:book_compactjournal(Bookie0, 30000), ok = leveled_bookie:book_compactjournal(Bookie0, 30000),
testutil:check_forlist(Bookie0, ChkList1), testutil:check_forlist(Bookie0, ChkList1),
@ -277,13 +304,15 @@ journal_compaction_tester(Restart, WRP) ->
testutil:check_forobject(Bookie0, TestObject2), testutil:check_forobject(Bookie0, TestObject2),
%% Delete some of the objects %% Delete some of the objects
ObjListD = testutil:generate_objects(10000, 2), ObjListD = testutil:generate_objects(10000, 2),
lists:foreach(fun({_R, O, _S}) -> lists:foreach(
fun({_R, O, _S}) ->
testutil:book_riakdelete(Bookie0, testutil:book_riakdelete(Bookie0,
testutil:get_bucket(O), testutil:get_bucket(O),
testutil:get_key(O), testutil:get_key(O),
[]) [])
end, end,
ObjListD), ObjListD
),
%% Now replace all the other objects %% Now replace all the other objects
ObjList2 = testutil:generate_objects(40000, 10002), ObjList2 = testutil:generate_objects(40000, 10002),
@ -539,11 +568,11 @@ fetchput_snapshot(_Config) ->
% smaller due to replacements and files deleting % smaller due to replacements and files deleting
% This is dependent on the sleep though (yuk) % This is dependent on the sleep though (yuk)
{B1Size, B1Count} = testutil:check_bucket_stats(Bookie2, "Bucket1"), {B1Size, B1Count} = testutil:check_bucket_stats(Bookie2, <<"Bucket1">>),
true = B1Size > 0, true = B1Size > 0,
true = B1Count == 1, true = B1Count == 1,
{B1Size, B1Count} = testutil:check_bucket_stats(Bookie2, "Bucket1"), {B1Size, B1Count} = testutil:check_bucket_stats(Bookie2, <<"Bucket1">>),
{BSize, BCount} = testutil:check_bucket_stats(Bookie2, "Bucket"), {BSize, BCount} = testutil:check_bucket_stats(Bookie2, <<"Bucket">>),
true = BSize > 0, true = BSize > 0,
true = BCount == 180000, true = BCount == 180000,
@ -622,82 +651,78 @@ load_and_count(JournalSize, BookiesMemSize, PencillerMemSize) ->
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
io:format("Loading initial small objects~n"), io:format("Loading initial small objects~n"),
G1 = fun testutil:generate_smallobjects/2, G1 = fun testutil:generate_smallobjects/2,
lists:foldl(fun(_X, Acc) -> lists:foldl(
testutil:load_objects(5000, fun(_X, Acc) ->
[Acc + 2], testutil:load_objects(
Bookie1, 5000, [Acc + 2], Bookie1, TestObject, G1),
TestObject,
G1),
{_S, Count} = {_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"), testutil:check_bucket_stats(Bookie1, <<"Bucket">>),
if if
Acc + 5000 == Count -> Acc + 5000 == Count ->
ok ok
end, end,
Acc + 5000 end, Acc + 5000 end,
0, 0,
lists:seq(1, 20)), lists:seq(1, 20)
),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
io:format("Loading larger compressible objects~n"), io:format("Loading larger compressible objects~n"),
G2 = fun testutil:generate_compressibleobjects/2, G2 = fun testutil:generate_compressibleobjects/2,
lists:foldl(fun(_X, Acc) -> lists:foldl(
testutil:load_objects(5000, fun(_X, Acc) ->
[Acc + 2], testutil:load_objects(
Bookie1, 5000, [Acc + 2], Bookie1, TestObject, G2),
TestObject,
G2),
{_S, Count} = {_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"), testutil:check_bucket_stats(Bookie1, <<"Bucket">>),
if if
Acc + 5000 == Count -> Acc + 5000 == Count ->
ok ok
end, end,
Acc + 5000 end, Acc + 5000 end,
100000, 100000,
lists:seq(1, 20)), lists:seq(1, 20)
),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
io:format("Replacing small objects~n"), io:format("Replacing small objects~n"),
lists:foldl(fun(_X, Acc) -> lists:foldl(
testutil:load_objects(5000, fun(_X, Acc) ->
[Acc + 2], testutil:load_objects(
Bookie1, 5000, [Acc + 2], Bookie1, TestObject, G1),
TestObject,
G1),
{_S, Count} = {_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"), testutil:check_bucket_stats(Bookie1, <<"Bucket">>),
if if
Count == 200000 -> Count == 200000 ->
ok ok
end, end,
Acc + 5000 end, Acc + 5000 end,
0, 0,
lists:seq(1, 20)), lists:seq(1, 20)
),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
io:format("Loading more small objects~n"), io:format("Loading more small objects~n"),
io:format("Now with unused snapshot so deletions are blocked~n"), io:format("Now with unused snapshot so deletions are blocked~n"),
{ok, PclClone, null} = {ok, PclClone, null} =
leveled_bookie:book_snapshot(Bookie1, ledger, undefined, true), leveled_bookie:book_snapshot(Bookie1, ledger, undefined, true),
lists:foldl(fun(_X, Acc) -> lists:foldl(
testutil:load_objects(5000, fun(_X, Acc) ->
[Acc + 2], testutil:load_objects(
Bookie1, 5000, [Acc + 2], Bookie1, TestObject, G2),
TestObject,
G2),
{_S, Count} = {_S, Count} =
testutil:check_bucket_stats(Bookie1, "Bucket"), testutil:check_bucket_stats(Bookie1, <<"Bucket">>),
if if
Acc + 5000 == Count -> Acc + 5000 == Count ->
ok ok
end, end,
Acc + 5000 end, Acc + 5000 end,
200000, 200000,
lists:seq(1, 20)), lists:seq(1, 20)
),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
ok = leveled_penciller:pcl_close(PclClone), ok = leveled_penciller:pcl_close(PclClone),
{_S, 300000} = testutil:check_bucket_stats(Bookie1, "Bucket"), {_S, 300000} = testutil:check_bucket_stats(Bookie1, <<"Bucket">>),
ok = leveled_bookie:book_close(Bookie1), ok = leveled_bookie:book_close(Bookie1),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1), {ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
{_, 300000} = testutil:check_bucket_stats(Bookie2, "Bucket"), {_, 300000} = testutil:check_bucket_stats(Bookie2, <<"Bucket">>),
ok = leveled_bookie:book_close(Bookie2), ok = leveled_bookie:book_close(Bookie2),
@ -722,21 +747,19 @@ load_and_count_withdelete(_Config) ->
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
io:format("Loading initial small objects~n"), io:format("Loading initial small objects~n"),
G1 = fun testutil:generate_smallobjects/2, G1 = fun testutil:generate_smallobjects/2,
lists:foldl(fun(_X, Acc) -> lists:foldl(
testutil:load_objects(5000, fun(_X, Acc) ->
[Acc + 2], testutil:load_objects(
Bookie1, 5000, [Acc + 2], Bookie1, TestObject, G1),
TestObject, {_S, Count} = testutil:check_bucket_stats(Bookie1, <<"Bucket">>),
G1),
{_S, Count} = testutil:check_bucket_stats(Bookie1,
"Bucket"),
if if
Acc + 5000 == Count -> Acc + 5000 == Count ->
ok ok
end, end,
Acc + 5000 end, Acc + 5000 end,
0, 0,
lists:seq(1, 20)), lists:seq(1, 20)
),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
{BucketD, KeyD} = {BucketD, KeyD} =
{testutil:get_bucket(TestObject), testutil:get_key(TestObject)}, {testutil:get_bucket(TestObject), testutil:get_key(TestObject)},
@ -746,21 +769,19 @@ load_and_count_withdelete(_Config) ->
{_, 0} = testutil:check_bucket_stats(Bookie1, BucketD), {_, 0} = testutil:check_bucket_stats(Bookie1, BucketD),
io:format("Loading larger compressible objects~n"), io:format("Loading larger compressible objects~n"),
G2 = fun testutil:generate_compressibleobjects/2, G2 = fun testutil:generate_compressibleobjects/2,
lists:foldl(fun(_X, Acc) -> lists:foldl(
testutil:load_objects(5000, fun(_X, Acc) ->
[Acc + 2], testutil:load_objects(
Bookie1, 5000, [Acc + 2], Bookie1, no_check, G2),
no_check, {_S, Count} = testutil:check_bucket_stats(Bookie1, <<"Bucket">>),
G2),
{_S, Count} = testutil:check_bucket_stats(Bookie1,
"Bucket"),
if if
Acc + 5000 == Count -> Acc + 5000 == Count ->
ok ok
end, end,
Acc + 5000 end, Acc + 5000 end,
100000, 100000,
lists:seq(1, 20)), lists:seq(1, 20)
),
not_found = testutil:book_riakget(Bookie1, BucketD, KeyD), not_found = testutil:book_riakget(Bookie1, BucketD, KeyD),
ok = leveled_bookie:book_close(Bookie1), ok = leveled_bookie:book_close(Bookie1),
@ -780,11 +801,8 @@ space_clear_ondelete(_Config) ->
{sync_strategy, testutil:sync_strategy()}], {sync_strategy, testutil:sync_strategy()}],
{ok, Book1} = leveled_bookie:book_start(StartOpts1), {ok, Book1} = leveled_bookie:book_start(StartOpts1),
G2 = fun testutil:generate_compressibleobjects/2, G2 = fun testutil:generate_compressibleobjects/2,
testutil:load_objects(20000, testutil:load_objects(
[uuid, uuid, uuid, uuid], 20000, [uuid, uuid, uuid, uuid], Book1, no_check, G2),
Book1,
no_check,
G2),
FoldKeysFun = fun(B, K, Acc) -> [{B, K}|Acc] end, FoldKeysFun = fun(B, K, Acc) -> [{B, K}|Acc] end,
@ -808,10 +826,9 @@ space_clear_ondelete(_Config) ->
FoldObjectsFun = fun(B, K, ObjBin, Acc) -> FoldObjectsFun = fun(B, K, ObjBin, Acc) ->
[{B, K, erlang:phash2(ObjBin)}|Acc] end, [{B, K, erlang:phash2(ObjBin)}|Acc] end,
{async, HTreeF1} = leveled_bookie:book_objectfold(Book1, {async, HTreeF1} =
?RIAK_TAG, leveled_bookie:book_objectfold(
{FoldObjectsFun, []}, Book1, ?RIAK_TAG, {FoldObjectsFun, []}, false),
false),
% This query does not Snap PreFold - and so will not prevent % This query does not Snap PreFold - and so will not prevent
% pending deletes from prompting actual deletes % pending deletes from prompting actual deletes
@ -822,30 +839,32 @@ space_clear_ondelete(_Config) ->
% Delete the keys % Delete the keys
SW2 = os:timestamp(), SW2 = os:timestamp(),
lists:foreach(fun({Bucket, Key}) -> lists:foreach(
testutil:book_riakdelete(Book1, fun({Bucket, Key}) ->
Bucket, testutil:book_riakdelete(Book1, Bucket, Key, [])
Key,
[])
end, end,
KL1), KL1),
io:format("Deletion took ~w microseconds for 80K keys~n", io:format(
"Deletion took ~w microseconds for 80K keys~n",
[timer:now_diff(os:timestamp(), SW2)]), [timer:now_diff(os:timestamp(), SW2)]),
ok = leveled_bookie:book_compactjournal(Book1, 30000), ok = leveled_bookie:book_compactjournal(Book1, 30000),
F = fun leveled_bookie:book_islastcompactionpending/1, F = fun leveled_bookie:book_islastcompactionpending/1,
lists:foldl(fun(X, Pending) -> lists:foldl(
fun(X, Pending) ->
case Pending of case Pending of
false -> false ->
false; false;
true -> true ->
io:format("Loop ~w waiting for journal " io:format(
++ "compaction to complete~n", [X]), "Loop ~w waiting for journal "
"compaction to complete~n",
[X]
),
timer:sleep(20000), timer:sleep(20000),
F(Book1) F(Book1)
end end, end
end,
true, true,
lists:seq(1, 15)), lists:seq(1, 15)),
io:format("Waiting for journal deletes - blocked~n"), io:format("Waiting for journal deletes - blocked~n"),
@ -1113,7 +1132,7 @@ many_put_fetch_switchcompression_tester(CompressionMethod) ->
%% Change method back again %% Change method back again
{ok, Bookie3} = leveled_bookie:book_start(StartOpts1), {ok, Bookie3} = leveled_bookie:book_start(StartOpts1),
testutil:check_formissingobject(Bookie3, "Bookie1", "MissingKey0123"), testutil:check_formissingobject(Bookie3, <<"Bookie1">>, "MissingKey0123"),
lists:foreach( lists:foreach(
fun(CL) -> ok = testutil:check_forlist(Bookie3, CL) end, CL2s), fun(CL) -> ok = testutil:check_forlist(Bookie3, CL) end, CL2s),
lists:foreach( lists:foreach(
@ -1244,10 +1263,12 @@ bigpcl_bucketlist(_Config) ->
MapFun = MapFun =
fun(B) -> fun(B) ->
testutil:generate_objects(ObjectCount, 1, [], testutil:generate_objects(
leveled_rand:rand_bytes(100), ObjectCount, 1, [],
crypto:strong_rand_bytes(100),
fun() -> [] end, fun() -> [] end,
B) B
)
end, end,
ObjLofL = lists:map(MapFun, BucketList), ObjLofL = lists:map(MapFun, BucketList),
lists:foreach(fun(ObjL) -> testutil:riakload(Bookie1, ObjL) end, ObjLofL), lists:foreach(fun(ObjL) -> testutil:riakload(Bookie1, ObjL) end, ObjLofL),
@ -1263,11 +1284,15 @@ bigpcl_bucketlist(_Config) ->
FBAccT = {BucketFold, sets:new()}, FBAccT = {BucketFold, sets:new()},
{async, BucketFolder1} = {async, BucketFolder1} =
leveled_bookie:book_headfold(Bookie1, leveled_bookie:book_headfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{bucket_list, BucketList}, {bucket_list, BucketList},
FBAccT, FBAccT,
false, false, false), false,
false,
false
),
{FoldTime1, BucketList1} = timer:tc(BucketFolder1, []), {FoldTime1, BucketList1} = timer:tc(BucketFolder1, []),
true = BucketCount == sets:size(BucketList1), true = BucketCount == sets:size(BucketList1),
@ -1276,11 +1301,15 @@ bigpcl_bucketlist(_Config) ->
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1), {ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
{async, BucketFolder2} = {async, BucketFolder2} =
leveled_bookie:book_headfold(Bookie2, leveled_bookie:book_headfold(
Bookie2,
?RIAK_TAG, ?RIAK_TAG,
{bucket_list, BucketList}, {bucket_list, BucketList},
FBAccT, FBAccT,
false, false, false), false,
false,
false
),
{FoldTime2, BucketList2} = timer:tc(BucketFolder2, []), {FoldTime2, BucketList2} = timer:tc(BucketFolder2, []),
true = BucketCount == sets:size(BucketList2), true = BucketCount == sets:size(BucketList2),

328
test/end_to_end/iterator_SUITE.erl
View file

@ -57,7 +57,7 @@ expiring_indexes(_Config) ->
Indexes9 = testutil:get_randomindexes_generator(2), Indexes9 = testutil:get_randomindexes_generator(2),
TempRiakObjects = TempRiakObjects =
testutil:generate_objects( testutil:generate_objects(
KeyCount, binary_uuid, [], V9, Indexes9, "riakBucket"), KeyCount, binary_uuid, [], V9, Indexes9, <<"riakBucket">>),
IBKL1 = testutil:stdload_expiring(Bookie1, KeyCount, Future), IBKL1 = testutil:stdload_expiring(Bookie1, KeyCount, Future),
lists:foreach( lists:foreach(
@ -147,11 +147,13 @@ expiring_indexes(_Config) ->
Bookie1, B0, K0, 5, <<"value">>, leveled_util:integer_now() + 10), Bookie1, B0, K0, 5, <<"value">>, leveled_util:integer_now() + 10),
timer:sleep(1000), timer:sleep(1000),
{async, Folder2} = IndexFold(), {async, Folder2} = IndexFold(),
leveled_bookie:book_indexfold(Bookie1, leveled_bookie:book_indexfold(
Bookie1,
B0, B0,
{FoldFun, InitAcc}, {FoldFun, InitAcc},
{<<"temp_int">>, 5, 8}, {<<"temp_int">>, 5, 8},
{true, undefined}), {true, undefined}
),
QR2 = Folder2(), QR2 = Folder2(),
io:format("Query with additional entry length ~w~n", [length(QR2)]), io:format("Query with additional entry length ~w~n", [length(QR2)]),
true = lists:sort(QR2) == lists:sort([{5, B0, K0}|LoadedEntriesInRange]), true = lists:sort(QR2) == lists:sort([{5, B0, K0}|LoadedEntriesInRange]),
@ -208,11 +210,9 @@ breaking_folds(_Config) ->
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1), {ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
ObjectGen = testutil:get_compressiblevalue_andinteger(), ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = testutil:get_randomindexes_generator(8), IndexGen = testutil:get_randomindexes_generator(8),
ObjL1 = testutil:generate_objects(KeyCount, ObjL1 =
binary_uuid, testutil:generate_objects(
[], KeyCount, binary_uuid, [], ObjectGen, IndexGen),
ObjectGen,
IndexGen),
testutil:riakload(Bookie1, ObjL1), testutil:riakload(Bookie1, ObjL1),
% Find all keys index, and then same again but stop at a midpoint using a % Find all keys index, and then same again but stop at a midpoint using a
@ -261,7 +261,6 @@ breaking_folds(_Config) ->
io:format("Index fold with result size ~w~n", [length(KeyList2)]), io:format("Index fold with result size ~w~n", [length(KeyList2)]),
true = KeyCount div 2 == length(KeyList2), true = KeyCount div 2 == length(KeyList2),
HeadFoldFun = HeadFoldFun =
fun(_B, K, PO, Acc) -> fun(_B, K, PO, Acc) ->
{proxy_object, _MDBin, Size, _FF} = binary_to_term(PO), {proxy_object, _MDBin, Size, _FF} = binary_to_term(PO),
@ -287,10 +286,14 @@ breaking_folds(_Config) ->
end end
end, end,
{async, HeadFolderToMidK} = {async, HeadFolderToMidK} =
leveled_bookie:book_headfold(Bookie1, leveled_bookie:book_headfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{FoldThrowFun(HeadFoldFun), []}, {FoldThrowFun(HeadFoldFun), []},
true, true, false), true,
true,
false
),
KeySizeList2 = lists:reverse(CatchingFold(HeadFolderToMidK)), KeySizeList2 = lists:reverse(CatchingFold(HeadFolderToMidK)),
io:format("Head fold with result size ~w~n", [length(KeySizeList2)]), io:format("Head fold with result size ~w~n", [length(KeySizeList2)]),
true = KeyCount div 2 == length(KeySizeList2), true = KeyCount div 2 == length(KeySizeList2),
@ -300,21 +303,25 @@ breaking_folds(_Config) ->
[{K,byte_size(V)}|Acc] [{K,byte_size(V)}|Acc]
end, end,
{async, ObjectFolderKO} = {async, ObjectFolderKO} =
leveled_bookie:book_objectfold(Bookie1, leveled_bookie:book_objectfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{ObjFoldFun, []}, {ObjFoldFun, []},
false, false,
key_order), key_order
),
ObjSizeList1 = lists:reverse(ObjectFolderKO()), ObjSizeList1 = lists:reverse(ObjectFolderKO()),
io:format("Obj fold with result size ~w~n", [length(ObjSizeList1)]), io:format("Obj fold with result size ~w~n", [length(ObjSizeList1)]),
true = KeyCount == length(ObjSizeList1), true = KeyCount == length(ObjSizeList1),
{async, ObjFolderToMidK} = {async, ObjFolderToMidK} =
leveled_bookie:book_objectfold(Bookie1, leveled_bookie:book_objectfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{FoldThrowFun(ObjFoldFun), []}, {FoldThrowFun(ObjFoldFun), []},
false, false,
key_order), key_order
),
ObjSizeList2 = lists:reverse(CatchingFold(ObjFolderToMidK)), ObjSizeList2 = lists:reverse(CatchingFold(ObjFolderToMidK)),
io:format("Object fold with result size ~w~n", [length(ObjSizeList2)]), io:format("Object fold with result size ~w~n", [length(ObjSizeList2)]),
true = KeyCount div 2 == length(ObjSizeList2), true = KeyCount div 2 == length(ObjSizeList2),
@ -324,11 +331,13 @@ breaking_folds(_Config) ->
% that was terminated by reaching a point in the key range .. as results % that was terminated by reaching a point in the key range .. as results
% will not be passed to the fold function in key order % will not be passed to the fold function in key order
{async, ObjectFolderSO} = {async, ObjectFolderSO} =
leveled_bookie:book_objectfold(Bookie1, leveled_bookie:book_objectfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{ObjFoldFun, []}, {ObjFoldFun, []},
false, false,
sqn_order), sqn_order
),
ObjSizeList1_SO = lists:reverse(ObjectFolderSO()), ObjSizeList1_SO = lists:reverse(ObjectFolderSO()),
io:format("Obj fold with result size ~w~n", [length(ObjSizeList1_SO)]), io:format("Obj fold with result size ~w~n", [length(ObjSizeList1_SO)]),
true = KeyCount == length(ObjSizeList1_SO), true = KeyCount == length(ObjSizeList1_SO),
@ -346,33 +355,26 @@ breaking_folds(_Config) ->
end end
end, end,
{async, ObjFolderTo1K} = {async, ObjFolderTo1K} =
leveled_bookie:book_objectfold(Bookie1, leveled_bookie:book_objectfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{FoldThrowThousandFun(ObjFoldFun), []}, {FoldThrowThousandFun(ObjFoldFun), []},
false, false,
sqn_order), sqn_order
),
ObjSizeList2_SO = lists:reverse(CatchingFold(ObjFolderTo1K)), ObjSizeList2_SO = lists:reverse(CatchingFold(ObjFolderTo1K)),
io:format("Object fold with result size ~w~n", [length(ObjSizeList2_SO)]), io:format("Object fold with result size ~w~n", [length(ObjSizeList2_SO)]),
true = 1000 == length(ObjSizeList2_SO), true = 1000 == length(ObjSizeList2_SO),
ObjL2 = testutil:generate_objects(10, ObjL2 =
binary_uuid, testutil:generate_objects(
[], 10, binary_uuid, [], ObjectGen, IndexGen, <<"B2">>),
ObjectGen, ObjL3 =
IndexGen, testutil:generate_objects(
"B2"), 10, binary_uuid, [], ObjectGen, IndexGen, <<"B3">>),
ObjL3 = testutil:generate_objects(10, ObjL4 =
binary_uuid, testutil:generate_objects(
[], 10, binary_uuid, [], ObjectGen, IndexGen, <<"B4">>),
ObjectGen,
IndexGen,
"B3"),
ObjL4 = testutil:generate_objects(10,
binary_uuid,
[],
ObjectGen,
IndexGen,
"B4"),
testutil:riakload(Bookie1, ObjL2), testutil:riakload(Bookie1, ObjL2),
testutil:riakload(Bookie1, ObjL3), testutil:riakload(Bookie1, ObjL3),
testutil:riakload(Bookie1, ObjL4), testutil:riakload(Bookie1, ObjL4),
@ -396,20 +398,16 @@ breaking_folds(_Config) ->
end, end,
{async, StopAt3BucketFolder} = {async, StopAt3BucketFolder} =
leveled_bookie:book_bucketlist(Bookie1, leveled_bookie:book_bucketlist(
?RIAK_TAG, Bookie1, ?RIAK_TAG, {StopAt3Fun, []}, all),
{StopAt3Fun, []},
all),
BucketListSA3 = lists:reverse(CatchingFold(StopAt3BucketFolder)), BucketListSA3 = lists:reverse(CatchingFold(StopAt3BucketFolder)),
io:format("bucket list with result ~w~n", [BucketListSA3]), io:format("bucket list with result ~w~n", [BucketListSA3]),
true = [<<"B2">>, <<"B3">>] == BucketListSA3, true = [<<"B2">>, <<"B3">>] == BucketListSA3,
ok = leveled_bookie:book_close(Bookie1), ok = leveled_bookie:book_close(Bookie1),
testutil:reset_filestructure(). testutil:reset_filestructure().
single_object_with2i(_Config) -> single_object_with2i(_Config) ->
% Load a single object with an integer and a binary % Load a single object with an integer and a binary
% index and query for it % index and query for it
@ -429,36 +427,40 @@ single_object_with2i(_Config) ->
{async, IdxFolder1} = {async, IdxFolder1} =
leveled_bookie:book_indexfold( leveled_bookie:book_indexfold(
Bookie1, Bookie1,
"Bucket1", <<"Bucket1">>,
{fun testutil:foldkeysfun/3, []}, {fun testutil:foldkeysfun/3, []},
{list_to_binary("binary_bin"), {list_to_binary("binary_bin"),
<<99:32/integer>>, <<101:32/integer>>}, <<99:32/integer>>, <<101:32/integer>>},
{true, undefined}), {true, undefined}),
R1 = IdxFolder1(), R1 = IdxFolder1(),
io:format("R1 of ~w~n", [R1]), io:format("R1 of ~w~n", [R1]),
true = [{<<100:32/integer>>,"Key1"}] == R1, true = [{<<100:32/integer>>, <<"Key1">>}] == R1,
IdxQ2 = {index_query, IdxQ2 =
"Bucket1", {
index_query,
<<"Bucket1">>,
{fun testutil:foldkeysfun/3, []}, {fun testutil:foldkeysfun/3, []},
{list_to_binary("integer_int"), {list_to_binary("integer_int"), 99, 101},
99, 101}, {true, undefined}
{true, undefined}}, },
{async, IdxFolder2} = leveled_bookie:book_returnfolder(Bookie1, IdxQ2), {async, IdxFolder2} = leveled_bookie:book_returnfolder(Bookie1, IdxQ2),
R2 = IdxFolder2(), R2 = IdxFolder2(),
io:format("R2 of ~w~n", [R2]), io:format("R2 of ~w~n", [R2]),
true = [{100,"Key1"}] == R2, true = [{100, <<"Key1">>}] == R2,
IdxQ3 = {index_query, IdxQ3 =
{"Bucket1", "Key1"}, {
index_query,
{<<"Bucket1">>, <<"Key1">>},
{fun testutil:foldkeysfun/3, []}, {fun testutil:foldkeysfun/3, []},
{list_to_binary("integer_int"), {list_to_binary("integer_int"), 99, 101},
99, 101}, {true, undefined}
{true, undefined}}, },
{async, IdxFolder3} = leveled_bookie:book_returnfolder(Bookie1, IdxQ3), {async, IdxFolder3} = leveled_bookie:book_returnfolder(Bookie1, IdxQ3),
R3 = IdxFolder3(), R3 = IdxFolder3(),
io:format("R2 of ~w~n", [R3]), io:format("R2 of ~w~n", [R3]),
true = [{100,"Key1"}] == R3, true = [{100, <<"Key1">>}] == R3,
ok = leveled_bookie:book_close(Bookie1), ok = leveled_bookie:book_close(Bookie1),
testutil:reset_filestructure(). testutil:reset_filestructure().
@ -473,7 +475,7 @@ small_load_with2i(_Config) ->
{TestObject, TestSpec} = testutil:generate_testobject(), {TestObject, TestSpec} = testutil:generate_testobject(),
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec), ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
testutil:check_formissingobject(Bookie1, "Bucket1", "Key2"), testutil:check_formissingobject(Bookie1, <<"Bucket1">>, <<"Key2">>),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
ObjectGen = testutil:get_compressiblevalue_andinteger(), ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = testutil:get_randomindexes_generator(8), IndexGen = testutil:get_randomindexes_generator(8),
@ -486,58 +488,60 @@ small_load_with2i(_Config) ->
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
% Find all keys index, and then just the last key % Find all keys index, and then just the last key
IdxQ1 = {index_query, IdxQ1 =
"Bucket", {
index_query,
<<"Bucket">>,
{fun testutil:foldkeysfun/3, []}, {fun testutil:foldkeysfun/3, []},
{<<"idx1_bin">>, <<"#">>, <<"|">>}, {<<"idx1_bin">>, <<"#">>, <<"|">>},
{true, undefined}}, {true, undefined}
},
{async, IdxFolder} = leveled_bookie:book_returnfolder(Bookie1, IdxQ1), {async, IdxFolder} = leveled_bookie:book_returnfolder(Bookie1, IdxQ1),
KeyList1 = lists:usort(IdxFolder()), KeyList1 = lists:usort(IdxFolder()),
true = 10000 == length(KeyList1), true = 10000 == length(KeyList1),
{LastTerm, LastKey} = lists:last(KeyList1), {LastTerm, LastKey} = lists:last(KeyList1),
IdxQ2 = {index_query, IdxQ2 =
{"Bucket", LastKey}, {
index_query,
{<<"Bucket">>, LastKey},
{fun testutil:foldkeysfun/3, []}, {fun testutil:foldkeysfun/3, []},
{<<"idx1_bin">>, LastTerm, <<"|">>}, {<<"idx1_bin">>, LastTerm, <<"|">>},
{false, undefined}}, {false, undefined}
},
{async, IdxFolderLK} = leveled_bookie:book_returnfolder(Bookie1, IdxQ2), {async, IdxFolderLK} = leveled_bookie:book_returnfolder(Bookie1, IdxQ2),
KeyList2 = lists:usort(IdxFolderLK()), KeyList2 = lists:usort(IdxFolderLK()),
io:format("List should be last key ~w ~w~n", [LastKey, KeyList2]), io:format("List should be last key ~w ~w~n", [LastKey, KeyList2]),
true = 1 == length(KeyList2), true = 1 == length(KeyList2),
%% Delete the objects from the ChkList removing the indexes %% Delete the objects from the ChkList removing the indexes
lists:foreach(fun({_RN, Obj, Spc}) -> lists:foreach(
DSpc = lists:map(fun({add, F, T}) -> fun({_RN, Obj, Spc}) ->
{remove, F, T} DSpc =
end, lists:map(fun({add, F, T}) -> {remove, F, T} end, Spc),
Spc), {B, K} = {testutil:get_bucket(Obj), testutil:get_key(Obj)},
{B, K} =
{testutil:get_bucket(Obj), testutil:get_key(Obj)},
testutil:book_riakdelete(Bookie1, B, K, DSpc) testutil:book_riakdelete(Bookie1, B, K, DSpc)
end, end,
ChkList1), ChkList1
),
%% Get the Buckets Keys and Hashes for the whole bucket %% Get the Buckets Keys and Hashes for the whole bucket
FoldObjectsFun = fun(B, K, V, Acc) -> [{B, K, erlang:phash2(V)}|Acc] FoldObjectsFun =
end, fun(B, K, V, Acc) -> [{B, K, erlang:phash2(V)}|Acc] end,
{async, HTreeF1} = leveled_bookie:book_objectfold(Bookie1, {async, HTreeF1} =
?RIAK_TAG, leveled_bookie:book_objectfold(
{FoldObjectsFun, []}, Bookie1, ?RIAK_TAG, {FoldObjectsFun, []}, false),
false),
KeyHashList1 = HTreeF1(), KeyHashList1 = HTreeF1(),
{async, HTreeF2} = leveled_bookie:book_objectfold(Bookie1, {async, HTreeF2} =
?RIAK_TAG, leveled_bookie:book_objectfold(
"Bucket", Bookie1, ?RIAK_TAG, <<"Bucket">>, all, {FoldObjectsFun, []}, false
all, ),
{FoldObjectsFun, []},
false),
KeyHashList2 = HTreeF2(), KeyHashList2 = HTreeF2(),
{async, HTreeF3} = {async, HTreeF3} =
leveled_bookie:book_objectfold( leveled_bookie:book_objectfold(
Bookie1, Bookie1,
?RIAK_TAG, ?RIAK_TAG,
"Bucket", <<"Bucket">>,
{<<"idx1_bin">>, <<"#">>, <<"|">>}, {<<"idx1_bin">>, <<"#">>, <<"|">>},
{FoldObjectsFun, []}, {FoldObjectsFun, []},
false), false),
@ -546,12 +550,13 @@ small_load_with2i(_Config) ->
true = 9900 == length(KeyHashList2), true = 9900 == length(KeyHashList2),
true = 9900 == length(KeyHashList3), true = 9900 == length(KeyHashList3),
SumIntFun = fun(_B, _K, Obj, Acc) -> SumIntFun =
fun(_B, _K, Obj, Acc) ->
{I, _Bin} = testutil:get_value(Obj), {I, _Bin} = testutil:get_value(Obj),
Acc + I Acc + I
end, end,
BucketObjQ = BucketObjQ =
{foldobjects_bybucket, ?RIAK_TAG, "Bucket", all, {SumIntFun, 0}, true}, {foldobjects_bybucket, ?RIAK_TAG, <<"Bucket">>, all, {SumIntFun, 0}, true},
{async, Sum1} = leveled_bookie:book_returnfolder(Bookie1, BucketObjQ), {async, Sum1} = leveled_bookie:book_returnfolder(Bookie1, BucketObjQ),
Total1 = Sum1(), Total1 = Sum1(),
io:format("Total from summing all I is ~w~n", [Total1]), io:format("Total from summing all I is ~w~n", [Total1]),
@ -596,21 +601,18 @@ query_count(_Config) ->
BucketBin = list_to_binary("Bucket"), BucketBin = list_to_binary("Bucket"),
{TestObject, TestSpec} = {TestObject, TestSpec} =
testutil:generate_testobject( testutil:generate_testobject(
BucketBin, term_to_binary("Key1"), "Value1", [], [{"MDK1", "MDV1"}]), BucketBin, term_to_binary("Key1"), <<"Value1">>, [], [{<<"MDK1">>, <<"MDV1">>}]),
ok = testutil:book_riakput(Book1, TestObject, TestSpec), ok = testutil:book_riakput(Book1, TestObject, TestSpec),
testutil:check_forobject(Book1, TestObject), testutil:check_forobject(Book1, TestObject),
testutil:check_formissingobject(Book1, "Bucket1", "Key2"), testutil:check_formissingobject(Book1, <<"Bucket1">>, <<"Key2">>),
testutil:check_forobject(Book1, TestObject), testutil:check_forobject(Book1, TestObject),
lists:foreach( lists:foreach(
fun(_X) -> fun(_X) ->
V = testutil:get_compressiblevalue(), V = testutil:get_compressiblevalue(),
Indexes = testutil:get_randomindexes_generator(8), Indexes = testutil:get_randomindexes_generator(8),
SW = os:timestamp(), SW = os:timestamp(),
ObjL1 = testutil:generate_objects(10000, ObjL1 =
binary_uuid, testutil:generate_objects(10000, binary_uuid, [], V, Indexes),
[],
V,
Indexes),
testutil:riakload(Book1, ObjL1), testutil:riakload(Book1, ObjL1),
io:format( io:format(
"Put of 10000 objects with 8 index entries " "Put of 10000 objects with 8 index entries "
@ -681,7 +683,9 @@ query_count(_Config) ->
{true, undefined}}, {true, undefined}},
{async, {async,
Mia2KFolder2} = leveled_bookie:book_returnfolder(Book2, Query2), Mia2KFolder2} = leveled_bookie:book_returnfolder(Book2, Query2),
Mia2000Count2 = lists:foldl(fun({Term, _Key}, Acc) -> Mia2000Count2 =
lists:foldl(
fun({Term, _Key}, Acc) ->
case re:run(Term, RegMia) of case re:run(Term, RegMia) of
nomatch -> nomatch ->
Acc; Acc;
@ -731,7 +735,8 @@ query_count(_Config) ->
Spc9Del = lists:map(fun({add, IdxF, IdxT}) -> {remove, IdxF, IdxT} end, Spc9Del = lists:map(fun({add, IdxF, IdxT}) -> {remove, IdxF, IdxT} end,
Spc9), Spc9),
ok = testutil:book_riakput(Book2, Obj9, Spc9Del), ok = testutil:book_riakput(Book2, Obj9, Spc9Del),
lists:foreach(fun({IdxF, IdxT, X}) -> lists:foreach(
fun({IdxF, IdxT, X}) ->
Q = {index_query, Q = {index_query,
BucketBin, BucketBin,
{fun testutil:foldkeysfun/3, []}, {fun testutil:foldkeysfun/3, []},
@ -744,7 +749,8 @@ query_count(_Config) ->
Y = X - 1 Y = X - 1
end end
end, end,
R9), R9
),
ok = leveled_bookie:book_close(Book2), ok = leveled_bookie:book_close(Book2),
{ok, Book3} = {ok, Book3} =
leveled_bookie:book_start( leveled_bookie:book_start(
@ -800,13 +806,13 @@ query_count(_Config) ->
ObjList10A = ObjList10A =
testutil:generate_objects( testutil:generate_objects(
5000, binary_uuid, [], V9, Indexes9, "BucketA"), 5000, binary_uuid, [], V9, Indexes9, <<"BucketA">>),
ObjList10B = ObjList10B =
testutil:generate_objects( testutil:generate_objects(
5000, binary_uuid, [], V9, Indexes9, "BucketB"), 5000, binary_uuid, [], V9, Indexes9, <<"BucketB">>),
ObjList10C = ObjList10C =
testutil:generate_objects( testutil:generate_objects(
5000, binary_uuid, [], V9, Indexes9, "BucketC"), 5000, binary_uuid, [], V9, Indexes9, <<"BucketC">>),
testutil:riakload(Book4, ObjList10A), testutil:riakload(Book4, ObjList10A),
testutil:riakload(Book4, ObjList10B), testutil:riakload(Book4, ObjList10B),
testutil:riakload(Book4, ObjList10C), testutil:riakload(Book4, ObjList10C),
@ -819,10 +825,9 @@ query_count(_Config) ->
ok = leveled_bookie:book_close(Book4), ok = leveled_bookie:book_close(Book4),
{ok, Book5} = leveled_bookie:book_start(RootPath, {ok, Book5} =
2000, leveled_bookie:book_start(
50000000, RootPath, 2000, 50000000, testutil:sync_strategy()),
testutil:sync_strategy()),
{async, BLF3} = leveled_bookie:book_returnfolder(Book5, BucketListQuery), {async, BLF3} = leveled_bookie:book_returnfolder(Book5, BucketListQuery),
SW_QC = os:timestamp(), SW_QC = os:timestamp(),
BucketSet3 = BLF3(), BucketSet3 = BLF3(),
@ -866,33 +871,25 @@ multibucket_fold(_Config) ->
testutil:sync_strategy()), testutil:sync_strategy()),
ObjectGen = testutil:get_compressiblevalue_andinteger(), ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = fun() -> [] end, IndexGen = fun() -> [] end,
ObjL1 = testutil:generate_objects(13000, ObjL1 =
uuid, testutil:generate_objects(
[], 13000, uuid, [], ObjectGen, IndexGen, {<<"Type1">>, <<"Bucket1">>}
ObjectGen, ),
IndexGen,
{<<"Type1">>, <<"Bucket1">>}),
testutil:riakload(Bookie1, ObjL1), testutil:riakload(Bookie1, ObjL1),
ObjL2 = testutil:generate_objects(17000, ObjL2 =
uuid, testutil:generate_objects(
[], 17000, uuid, [], ObjectGen, IndexGen, <<"Bucket2">>
ObjectGen, ),
IndexGen,
<<"Bucket2">>),
testutil:riakload(Bookie1, ObjL2), testutil:riakload(Bookie1, ObjL2),
ObjL3 = testutil:generate_objects(7000, ObjL3 =
uuid, testutil:generate_objects(
[], 7000, uuid, [], ObjectGen, IndexGen, <<"Bucket3">>
ObjectGen, ),
IndexGen,
<<"Bucket3">>),
testutil:riakload(Bookie1, ObjL3), testutil:riakload(Bookie1, ObjL3),
ObjL4 = testutil:generate_objects(23000, ObjL4 =
uuid, testutil:generate_objects(
[], 23000, uuid, [], ObjectGen, IndexGen, {<<"Type2">>, <<"Bucket4">>}
ObjectGen, ),
IndexGen,
{<<"Type2">>, <<"Bucket4">>}),
testutil:riakload(Bookie1, ObjL4), testutil:riakload(Bookie1, ObjL4),
FF = fun(B, K, _PO, Acc) -> FF = fun(B, K, _PO, Acc) ->
@ -901,30 +898,30 @@ multibucket_fold(_Config) ->
FoldAccT = {FF, []}, FoldAccT = {FF, []},
{async, R1} = {async, R1} =
leveled_bookie:book_headfold(Bookie1, leveled_bookie:book_headfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{bucket_list, {bucket_list,
[{<<"Type1">>, <<"Bucket1">>}, [{<<"Type1">>, <<"Bucket1">>}, {<<"Type2">>, <<"Bucket4">>}]},
{<<"Type2">>, <<"Bucket4">>}]},
FoldAccT, FoldAccT,
false, false,
true, true,
false), false
),
O1 = length(R1()), O1 = length(R1()),
io:format("Result R1 of length ~w~n", [O1]), io:format("Result R1 of length ~w~n", [O1]),
{async, R2} = {async, R2} =
leveled_bookie:book_headfold(Bookie1, leveled_bookie:book_headfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
{bucket_list, {bucket_list, [<<"Bucket2">>, <<"Bucket3">>]},
[<<"Bucket2">>, {fun(_B, _K, _PO, Acc) -> Acc +1 end, 0},
<<"Bucket3">>]}, false,
{fun(_B, _K, _PO, Acc) -> true,
Acc +1 false
end, ),
0},
false, true, false),
O2 = R2(), O2 = R2(),
io:format("Result R2 of ~w~n", [O2]), io:format("Result R2 of ~w~n", [O2]),
@ -933,10 +930,8 @@ multibucket_fold(_Config) ->
FoldBucketsFun = fun(B, Acc) -> [B|Acc] end, FoldBucketsFun = fun(B, Acc) -> [B|Acc] end,
{async, Folder} = {async, Folder} =
leveled_bookie:book_bucketlist(Bookie1, leveled_bookie:book_bucketlist(
?RIAK_TAG, Bookie1, ?RIAK_TAG, {FoldBucketsFun, []}, all),
{FoldBucketsFun, []},
all),
BucketList = lists:reverse(Folder()), BucketList = lists:reverse(Folder()),
ExpectedBucketList = ExpectedBucketList =
[{<<"Type1">>, <<"Bucket1">>}, {<<"Type2">>, <<"Bucket4">>}, [{<<"Type1">>, <<"Bucket1">>}, {<<"Type2">>, <<"Bucket4">>},
@ -949,48 +944,47 @@ multibucket_fold(_Config) ->
rotating_objects(_Config) -> rotating_objects(_Config) ->
RootPath = testutil:reset_filestructure(), RootPath = testutil:reset_filestructure(),
ok = testutil:rotating_object_check(RootPath, "Bucket1", 10), ok = testutil:rotating_object_check(RootPath, <<"Bucket1">>, 10),
ok = testutil:rotating_object_check(RootPath, "Bucket2", 200), ok = testutil:rotating_object_check(RootPath, <<"Bucket2">>, 200),
ok = testutil:rotating_object_check(RootPath, "Bucket3", 800), ok = testutil:rotating_object_check(RootPath, <<"Bucket3">>, 800),
ok = testutil:rotating_object_check(RootPath, "Bucket4", 1600), ok = testutil:rotating_object_check(RootPath, <<"Bucket4">>, 1600),
ok = testutil:rotating_object_check(RootPath, "Bucket5", 3200), ok = testutil:rotating_object_check(RootPath, <<"Bucket5">>, 3200),
ok = testutil:rotating_object_check(RootPath, "Bucket6", 9600), ok = testutil:rotating_object_check(RootPath, <<"Bucket6">>, 9600),
testutil:reset_filestructure(). testutil:reset_filestructure().
foldobjects_bybucket_range(_Config) -> foldobjects_bybucket_range(_Config) ->
RootPath = testutil:reset_filestructure(), RootPath = testutil:reset_filestructure(),
{ok, Bookie1} = leveled_bookie:book_start(RootPath, {ok, Bookie1} =
2000, leveled_bookie:book_start(
50000000, RootPath, 2000, 50000000, testutil:sync_strategy()),
testutil:sync_strategy()),
ObjectGen = testutil:get_compressiblevalue_andinteger(), ObjectGen = testutil:get_compressiblevalue_andinteger(),
IndexGen = fun() -> [] end, IndexGen = fun() -> [] end,
ObjL1 = testutil:generate_objects(1300, ObjL1 =
{fixed_binary, 1}, testutil:generate_objects(
[], 1300, {fixed_binary, 1}, [], ObjectGen, IndexGen, <<"Bucket1">>),
ObjectGen,
IndexGen,
<<"Bucket1">>),
testutil:riakload(Bookie1, ObjL1), testutil:riakload(Bookie1, ObjL1),
FoldKeysFun = fun(_B, K,_V, Acc) -> FoldKeysFun = fun(_B, K,_V, Acc) -> [ K |Acc] end,
[ K |Acc]
end,
StartKey = testutil:fixed_bin_key(123), StartKey = testutil:fixed_bin_key(123),
EndKey = testutil:fixed_bin_key(779), EndKey = testutil:fixed_bin_key(779),
{async, Folder} = leveled_bookie:book_objectfold(Bookie1, {async, Folder} =
leveled_bookie:book_objectfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
<<"Bucket1">>, <<"Bucket1">>,
{StartKey, EndKey}, {FoldKeysFun, []}, {StartKey, EndKey},
{FoldKeysFun, []},
true true
), ),
ResLen = length(Folder()), ResLen = length(Folder()),
io:format("Length of Result of folder ~w~n", [ResLen]), io:format("Length of Result of folder ~w~n", [ResLen]),
true = 657 == ResLen, true = 657 == ResLen,
{async, AllFolder} = leveled_bookie:book_objectfold(Bookie1, {async, AllFolder} =
leveled_bookie:book_objectfold(
Bookie1,
?RIAK_TAG, ?RIAK_TAG,
<<"Bucket1">>, <<"Bucket1">>,
all, all,

28
test/end_to_end/perf_SUITE.erl
View file

@ -101,7 +101,7 @@ riak_load_tester(Bucket, KeyCount, ObjSize, ProfileList, PM, LC) ->
IndexGenFun = IndexGenFun =
fun(ListID) -> fun(ListID) ->
fun() -> fun() ->
RandInt = leveled_rand:uniform(IndexCount - 1), RandInt = rand:uniform(IndexCount - 1),
IntIndex = ["integer", integer_to_list(ListID), "_int"], IntIndex = ["integer", integer_to_list(ListID), "_int"],
BinIndex = ["binary", integer_to_list(ListID), "_bin"], BinIndex = ["binary", integer_to_list(ListID), "_bin"],
[{add, iolist_to_binary(IntIndex), RandInt}, [{add, iolist_to_binary(IntIndex), RandInt},
@ -434,35 +434,35 @@ rotation_withnocheck(Book, B, NumberOfObjects, ObjSize, IdxCnt) ->
Book, Book,
B, B,
NumberOfObjects, NumberOfObjects,
base64:encode(leveled_rand:rand_bytes(ObjSize)), base64:encode(crypto:strong_rand_bytes(ObjSize)),
IdxCnt IdxCnt
), ),
rotation_with_prefetch( rotation_with_prefetch(
Book, Book,
B, B,
NumberOfObjects, NumberOfObjects,
base64:encode(leveled_rand:rand_bytes(ObjSize)), base64:encode(crypto:strong_rand_bytes(ObjSize)),
IdxCnt IdxCnt
), ),
rotation_with_prefetch( rotation_with_prefetch(
Book, Book,
B, B,
NumberOfObjects, NumberOfObjects,
base64:encode(leveled_rand:rand_bytes(ObjSize)), base64:encode(crypto:strong_rand_bytes(ObjSize)),
IdxCnt IdxCnt
), ),
rotation_with_prefetch( rotation_with_prefetch(
Book, Book,
B, B,
NumberOfObjects, NumberOfObjects,
base64:encode(leveled_rand:rand_bytes(ObjSize)), base64:encode(crypto:strong_rand_bytes(ObjSize)),
IdxCnt IdxCnt
), ),
rotation_with_prefetch( rotation_with_prefetch(
Book, Book,
B, B,
NumberOfObjects, NumberOfObjects,
base64:encode(leveled_rand:rand_bytes(ObjSize)), base64:encode(crypto:strong_rand_bytes(ObjSize)),
IdxCnt IdxCnt
), ),
ok. ok.
@ -471,7 +471,7 @@ generate_chunk(CountPerList, ObjSize, IndexGenFun, Bucket, Chunk) ->
testutil:generate_objects( testutil:generate_objects(
CountPerList, CountPerList,
{fixed_binary, (Chunk - 1) * CountPerList + 1}, [], {fixed_binary, (Chunk - 1) * CountPerList + 1}, [],
base64:encode(leveled_rand:rand_bytes(ObjSize)), base64:encode(crypto:strong_rand_bytes(ObjSize)),
IndexGenFun(Chunk), IndexGenFun(Chunk),
Bucket Bucket
). ).
@ -480,7 +480,7 @@ load_chunk(Bookie, CountPerList, ObjSize, IndexGenFun, Bucket, Chunk) ->
ct:log(?INFO, "Generating and loading ObjList ~w", [Chunk]), ct:log(?INFO, "Generating and loading ObjList ~w", [Chunk]),
time_load_chunk( time_load_chunk(
Bookie, Bookie,
{Bucket, base64:encode(leveled_rand:rand_bytes(ObjSize)), IndexGenFun(Chunk)}, {Bucket, base64:encode(crypto:strong_rand_bytes(ObjSize)), IndexGenFun(Chunk)},
(Chunk - 1) * CountPerList + 1, (Chunk - 1) * CountPerList + 1,
Chunk * CountPerList, Chunk * CountPerList,
0, 0,
@ -577,9 +577,9 @@ random_fetches(FetchType, Bookie, Bucket, ObjCount, Fetches) ->
case I rem 5 of case I rem 5 of
1 -> 1 ->
testutil:fixed_bin_key( testutil:fixed_bin_key(
Twenty + leveled_rand:uniform(ObjCount - Twenty)); Twenty + rand:uniform(ObjCount - Twenty));
_ -> _ ->
testutil:fixed_bin_key(leveled_rand:uniform(Twenty)) testutil:fixed_bin_key(rand:uniform(Twenty))
end end
end, end,
{TC, ok} = {TC, ok} =
@ -616,18 +616,18 @@ random_fetches(FetchType, Bookie, Bucket, ObjCount, Fetches) ->
random_queries(Bookie, Bucket, IDs, IdxCnt, MaxRange, IndexesReturned) -> random_queries(Bookie, Bucket, IDs, IdxCnt, MaxRange, IndexesReturned) ->
QueryFun = QueryFun =
fun() -> fun() ->
ID = leveled_rand:uniform(IDs), ID = rand:uniform(IDs),
BinIndex = BinIndex =
iolist_to_binary(["binary", integer_to_list(ID), "_bin"]), iolist_to_binary(["binary", integer_to_list(ID), "_bin"]),
Twenty = IdxCnt div 5, Twenty = IdxCnt div 5,
RI = leveled_rand:uniform(MaxRange), RI = rand:uniform(MaxRange),
[Start, End] = [Start, End] =
case RI of case RI of
RI when RI < (MaxRange div 5) -> RI when RI < (MaxRange div 5) ->
R0 = leveled_rand:uniform(IdxCnt - (Twenty + RI)), R0 = rand:uniform(IdxCnt - (Twenty + RI)),
[R0 + Twenty, R0 + Twenty + RI]; [R0 + Twenty, R0 + Twenty + RI];
_ -> _ ->
R0 = leveled_rand:uniform(Twenty - RI), R0 = rand:uniform(Twenty - RI),
[R0, R0 + RI] [R0, R0 + RI]
end, end,
FoldKeysFun = fun(_B, _K, Cnt) -> Cnt + 1 end, FoldKeysFun = fun(_B, _K, Cnt) -> Cnt + 1 end,

42
test/end_to_end/riak_SUITE.erl
View file

@ -58,7 +58,7 @@ basic_riak_tester(Bucket, KeyCount) ->
IndexGenFun = IndexGenFun =
fun(ListID) -> fun(ListID) ->
fun() -> fun() ->
RandInt = leveled_rand:uniform(IndexCount), RandInt = rand:uniform(IndexCount),
ID = integer_to_list(ListID), ID = integer_to_list(ListID),
[{add, [{add,
list_to_binary("integer" ++ ID ++ "_int"), list_to_binary("integer" ++ ID ++ "_int"),
@ -75,7 +75,7 @@ basic_riak_tester(Bucket, KeyCount) ->
testutil:generate_objects( testutil:generate_objects(
CountPerList, CountPerList,
{fixed_binary, 1}, [], {fixed_binary, 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
IndexGenFun(1), IndexGenFun(1),
Bucket Bucket
), ),
@ -83,7 +83,7 @@ basic_riak_tester(Bucket, KeyCount) ->
testutil:generate_objects( testutil:generate_objects(
CountPerList, CountPerList,
{fixed_binary, CountPerList + 1}, [], {fixed_binary, CountPerList + 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
IndexGenFun(2), IndexGenFun(2),
Bucket Bucket
), ),
@ -92,7 +92,7 @@ basic_riak_tester(Bucket, KeyCount) ->
testutil:generate_objects( testutil:generate_objects(
CountPerList, CountPerList,
{fixed_binary, 2 * CountPerList + 1}, [], {fixed_binary, 2 * CountPerList + 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
IndexGenFun(3), IndexGenFun(3),
Bucket Bucket
), ),
@ -101,7 +101,7 @@ basic_riak_tester(Bucket, KeyCount) ->
testutil:generate_objects( testutil:generate_objects(
CountPerList, CountPerList,
{fixed_binary, 3 * CountPerList + 1}, [], {fixed_binary, 3 * CountPerList + 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
IndexGenFun(4), IndexGenFun(4),
Bucket Bucket
), ),
@ -110,7 +110,7 @@ basic_riak_tester(Bucket, KeyCount) ->
testutil:generate_objects( testutil:generate_objects(
CountPerList, CountPerList,
{fixed_binary, 4 * CountPerList + 1}, [], {fixed_binary, 4 * CountPerList + 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
IndexGenFun(5), IndexGenFun(5),
Bucket Bucket
), ),
@ -276,7 +276,7 @@ summarisable_sstindex(_Config) ->
ObjListToSort = ObjListToSort =
lists:map( lists:map(
fun(I) -> fun(I) ->
{leveled_rand:uniform(KeyCount * 10), {rand:uniform(KeyCount * 10),
testutil:set_object( testutil:set_object(
Bucket, KeyGen(I), integer_to_binary(I), IndexGen, [])} Bucket, KeyGen(I), integer_to_binary(I), IndexGen, [])}
end, end,
@ -344,7 +344,7 @@ summarisable_sstindex(_Config) ->
true = 200 == length(KeyRangeCheckFun(StartKey, EndKey)) true = 200 == length(KeyRangeCheckFun(StartKey, EndKey))
end, end,
lists:map( lists:map(
fun(_I) -> leveled_rand:uniform(KeyCount - 200) end, fun(_I) -> rand:uniform(KeyCount - 200) end,
lists:seq(1, 100))), lists:seq(1, 100))),
IdxObjKeyCount = 50000, IdxObjKeyCount = 50000,
@ -367,7 +367,7 @@ summarisable_sstindex(_Config) ->
IdxObjListToSort = IdxObjListToSort =
lists:map( lists:map(
fun(I) -> fun(I) ->
{leveled_rand:uniform(KeyCount * 10), {rand:uniform(KeyCount * 10),
testutil:set_object( testutil:set_object(
Bucket, Bucket,
KeyGen(I), KeyGen(I),
@ -419,7 +419,7 @@ summarisable_sstindex(_Config) ->
end, end,
lists:map( lists:map(
fun(_I) -> fun(_I) ->
leveled_rand:uniform(IdxObjKeyCount - 20) rand:uniform(IdxObjKeyCount - 20)
end, end,
lists:seq(1, 100))), lists:seq(1, 100))),
lists:foreach( lists:foreach(
@ -430,7 +430,7 @@ summarisable_sstindex(_Config) ->
end, end,
lists:map( lists:map(
fun(_I) -> fun(_I) ->
leveled_rand:uniform(IdxObjKeyCount - 10) rand:uniform(IdxObjKeyCount - 10)
end, end,
lists:seq(1, 100))), lists:seq(1, 100))),
@ -451,7 +451,7 @@ summarisable_sstindex(_Config) ->
true = 200 == length(KeyRangeCheckFun(StartKey, EndKey)) true = 200 == length(KeyRangeCheckFun(StartKey, EndKey))
end, end,
lists:map( lists:map(
fun(_I) -> leveled_rand:uniform(KeyCount - 200) end, fun(_I) -> rand:uniform(KeyCount - 200) end,
lists:seq(1, 100))), lists:seq(1, 100))),
ok = leveled_bookie:book_destroy(Bookie1). ok = leveled_bookie:book_destroy(Bookie1).
@ -475,7 +475,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
100000, 100000,
{fixed_binary, 1}, [], {fixed_binary, 1}, [],
leveled_rand:rand_bytes(32), crypto:strong_rand_bytes(32),
fun() -> [] end, fun() -> [] end,
<<"BaselineB">> <<"BaselineB">>
), ),
@ -485,7 +485,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
20000, 20000,
{fixed_binary, 1}, [], {fixed_binary, 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
fun() -> [] end, fun() -> [] end,
<<"B0">> <<"B0">>
), ),
@ -498,7 +498,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
15000, 15000,
{fixed_binary, 20001}, [], {fixed_binary, 20001}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
fun() -> [] end, fun() -> [] end,
<<"B0">> <<"B0">>
), ),
@ -511,7 +511,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
35000, 35000,
{fixed_binary, 35001}, [], {fixed_binary, 35001}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
fun() -> [] end, fun() -> [] end,
<<"B0">> <<"B0">>
), ),
@ -524,7 +524,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
30000, 30000,
{fixed_binary, 70001}, [], {fixed_binary, 70001}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
fun() -> [] end, fun() -> [] end,
<<"B0">> <<"B0">>
), ),
@ -537,7 +537,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
8000, 8000,
{fixed_binary, 1}, [], {fixed_binary, 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
fun() -> [] end, fun() -> [] end,
<<"B1">> <<"B1">>
), ),
@ -550,7 +550,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
7000, 7000,
{fixed_binary, 1}, [], {fixed_binary, 1}, [],
leveled_rand:rand_bytes(512), crypto:strong_rand_bytes(512),
fun() -> [] end, fun() -> [] end,
<<"B2">> <<"B2">>
), ),
@ -815,7 +815,7 @@ fetchclocks_modifiedbetween(_Config) ->
testutil:generate_objects( testutil:generate_objects(
200000, 200000,
{fixed_binary, 1}, [], {fixed_binary, 1}, [],
leveled_rand:rand_bytes(32), crypto:strong_rand_bytes(32),
fun() -> [] end, fun() -> [] end,
<<"B1.9">> <<"B1.9">>
), ),
@ -1637,7 +1637,7 @@ bigobject_memorycheck(_Config) ->
ObjPutFun = ObjPutFun =
fun(I) -> fun(I) ->
Key = base64:encode(<<I:32/integer>>), Key = base64:encode(<<I:32/integer>>),
Value = leveled_rand:rand_bytes(1024 * 1024), Value = crypto:strong_rand_bytes(1024 * 1024),
% a big object each time! % a big object each time!
{Obj, Spc} = testutil:set_object(Bucket, Key, Value, IndexGen, []), {Obj, Spc} = testutil:set_object(Bucket, Key, Value, IndexGen, []),
testutil:book_riakput(Bookie, Obj, Spc) testutil:book_riakput(Bookie, Obj, Spc)

100
test/end_to_end/testutil.erl
View file

@ -231,12 +231,14 @@ sync_strategy() ->
none. none.
book_riakput(Pid, RiakObject, IndexSpecs) -> book_riakput(Pid, RiakObject, IndexSpecs) ->
leveled_bookie:book_put(Pid, leveled_bookie:book_put(
Pid,
RiakObject#r_object.bucket, RiakObject#r_object.bucket,
RiakObject#r_object.key, RiakObject#r_object.key,
to_binary(v1, RiakObject), to_binary(v1, RiakObject),
IndexSpecs, IndexSpecs,
?RIAK_TAG). ?RIAK_TAG
).
book_tempriakput(Pid, RiakObject, IndexSpecs, TTL) -> book_tempriakput(Pid, RiakObject, IndexSpecs, TTL) ->
leveled_bookie:book_tempput( leveled_bookie:book_tempput(
@ -246,7 +248,8 @@ book_tempriakput(Pid, RiakObject, IndexSpecs, TTL) ->
to_binary(v1, RiakObject), to_binary(v1, RiakObject),
IndexSpecs, IndexSpecs,
?RIAK_TAG, ?RIAK_TAG,
TTL). TTL
).
book_riakdelete(Pid, Bucket, Key, IndexSpecs) -> book_riakdelete(Pid, Bucket, Key, IndexSpecs) ->
leveled_bookie:book_put(Pid, Bucket, Key, delete, IndexSpecs, ?RIAK_TAG). leveled_bookie:book_put(Pid, Bucket, Key, delete, IndexSpecs, ?RIAK_TAG).
@ -383,9 +386,8 @@ wait_for_compaction(Bookie) ->
check_bucket_stats(Bookie, Bucket) -> check_bucket_stats(Bookie, Bucket) ->
FoldSW1 = os:timestamp(), FoldSW1 = os:timestamp(),
io:format("Checking bucket size~n"), io:format("Checking bucket size~n"),
{async, Folder1} = leveled_bookie:book_returnfolder(Bookie, {async, Folder1} =
{riakbucket_stats, leveled_bookie:book_returnfolder(Bookie, {riakbucket_stats, Bucket}),
Bucket}),
{B1Size, B1Count} = Folder1(), {B1Size, B1Count} = Folder1(),
io:format("Bucket fold completed in ~w microseconds~n", io:format("Bucket fold completed in ~w microseconds~n",
[timer:now_diff(os:timestamp(), FoldSW1)]), [timer:now_diff(os:timestamp(), FoldSW1)]),
@ -399,7 +401,8 @@ check_forlist(Bookie, ChkList) ->
check_forlist(Bookie, ChkList, Log) -> check_forlist(Bookie, ChkList, Log) ->
SW = os:timestamp(), SW = os:timestamp(),
lists:foreach(fun({_RN, Obj, _Spc}) -> lists:foreach(
fun({_RN, Obj, _Spc}) ->
if if
Log == true -> Log == true ->
io:format("Fetching Key ~s~n", [Obj#r_object.key]); io:format("Fetching Key ~s~n", [Obj#r_object.key]);
@ -409,7 +412,8 @@ check_forlist(Bookie, ChkList, Log) ->
R = book_riakget(Bookie, R = book_riakget(Bookie,
Obj#r_object.bucket, Obj#r_object.bucket,
Obj#r_object.key), Obj#r_object.key),
true = case R of true =
case R of
{ok, Val} -> {ok, Val} ->
to_binary(v1, Obj) == Val; to_binary(v1, Obj) == Val;
not_found -> not_found ->
@ -419,8 +423,10 @@ check_forlist(Bookie, ChkList, Log) ->
end end
end, end,
ChkList), ChkList),
io:format("Fetch check took ~w microseconds checking list of length ~w~n", io:format(
[timer:now_diff(os:timestamp(), SW), length(ChkList)]). "Fetch check took ~w microseconds checking list of length ~w~n",
[timer:now_diff(os:timestamp(), SW), length(ChkList)]
).
checkhead_forlist(Bookie, ChkList) -> checkhead_forlist(Bookie, ChkList) ->
SW = os:timestamp(), SW = os:timestamp(),
@ -470,11 +476,14 @@ check_formissingobject(Bookie, Bucket, Key) ->
generate_testobject() -> generate_testobject() ->
{B1, K1, V1, Spec1, MD} = {"Bucket1", {B1, K1, V1, Spec1, MD} =
"Key1", {
"Value1", <<"Bucket1">>,
<<"Key1">>,
<<"Value1">>,
[], [],
[{"MDK1", "MDV1"}]}, [{<<"MDK1">>, <<"MDV1">>}]
},
generate_testobject(B1, K1, V1, Spec1, MD). generate_testobject(B1, K1, V1, Spec1, MD).
generate_testobject(B, K, V, Spec, MD) -> generate_testobject(B, K, V, Spec, MD) ->
@ -493,7 +502,7 @@ generate_compressibleobjects(Count, KeyNumber) ->
get_compressiblevalue_andinteger() -> get_compressiblevalue_andinteger() ->
{leveled_rand:uniform(1000), get_compressiblevalue()}. {rand:uniform(1000), get_compressiblevalue()}.
get_compressiblevalue() -> get_compressiblevalue() ->
S1 = "111111111111111", S1 = "111111111111111",
@ -510,7 +519,7 @@ get_compressiblevalue() ->
iolist_to_binary( iolist_to_binary(
lists:foldl( lists:foldl(
fun(_X, Acc) -> fun(_X, Acc) ->
{_, Str} = lists:keyfind(leveled_rand:uniform(8), 1, Selector), {_, Str} = lists:keyfind(rand:uniform(8), 1, Selector),
[Str|Acc] end, [Str|Acc] end,
[""], [""],
L L
@ -518,28 +527,39 @@ get_compressiblevalue() ->
). ).
generate_smallobjects(Count, KeyNumber) -> generate_smallobjects(Count, KeyNumber) ->
generate_objects(Count, KeyNumber, [], leveled_rand:rand_bytes(512)). generate_objects(Count, KeyNumber, [], crypto:strong_rand_bytes(512)).
generate_objects(Count, KeyNumber) -> generate_objects(Count, KeyNumber) ->
generate_objects(Count, KeyNumber, [], leveled_rand:rand_bytes(4096)). generate_objects(Count, KeyNumber, [], crypto:strong_rand_bytes(4096)).
generate_objects(Count, KeyNumber, ObjL, Value) -> generate_objects(Count, KeyNumber, ObjL, Value) ->
generate_objects(Count, KeyNumber, ObjL, Value, fun() -> [] end). generate_objects(Count, KeyNumber, ObjL, Value, fun() -> [] end).
generate_objects(Count, KeyNumber, ObjL, Value, IndexGen) -> generate_objects(Count, KeyNumber, ObjL, Value, IndexGen) ->
generate_objects(Count, KeyNumber, ObjL, Value, IndexGen, "Bucket"). generate_objects(Count, KeyNumber, ObjL, Value, IndexGen, <<"Bucket">>).
generate_objects(0, _KeyNumber, ObjL, _Value, _IndexGen, _Bucket) -> generate_objects(0, _KeyNumber, ObjL, _Value, _IndexGen, _Bucket) ->
lists:reverse(ObjL); lists:reverse(ObjL);
generate_objects(Count, binary_uuid, ObjL, Value, IndexGen, Bucket) -> generate_objects(
{Obj1, Spec1} = set_object(list_to_binary(Bucket), Count, binary_uuid, ObjL, Value, IndexGen, Bucket)
when is_list(Bucket) ->
generate_objects(
Count, binary_uuid, ObjL, Value, IndexGen, list_to_binary(Bucket)
);
generate_objects(
Count, binary_uuid, ObjL, Value, IndexGen, Bucket)
when is_binary(Bucket) ->
{Obj1, Spec1} =
set_object(
Bucket,
list_to_binary(leveled_util:generate_uuid()), list_to_binary(leveled_util:generate_uuid()),
Value, Value,
IndexGen), IndexGen
),
generate_objects(Count - 1, generate_objects(Count - 1,
binary_uuid, binary_uuid,
[{leveled_rand:uniform(), Obj1, Spec1}|ObjL], [{rand:uniform(), Obj1, Spec1}|ObjL],
Value, Value,
IndexGen, IndexGen,
Bucket); Bucket);
@ -550,19 +570,29 @@ generate_objects(Count, uuid, ObjL, Value, IndexGen, Bucket) ->
IndexGen), IndexGen),
generate_objects(Count - 1, generate_objects(Count - 1,
uuid, uuid,
[{leveled_rand:uniform(), Obj1, Spec1}|ObjL], [{rand:uniform(), Obj1, Spec1}|ObjL],
Value, Value,
IndexGen, IndexGen,
Bucket); Bucket);
generate_objects(Count, {binary, KeyNumber}, ObjL, Value, IndexGen, Bucket) -> generate_objects(
Count, {binary, KeyNumber}, ObjL, Value, IndexGen, Bucket)
when is_list(Bucket) ->
generate_objects(
Count, {binary, KeyNumber}, ObjL, Value, IndexGen, list_to_binary(Bucket)
);
generate_objects(
Count, {binary, KeyNumber}, ObjL, Value, IndexGen, Bucket)
when is_binary(Bucket) ->
{Obj1, Spec1} = {Obj1, Spec1} =
set_object(list_to_binary(Bucket), set_object(
Bucket,
list_to_binary(numbered_key(KeyNumber)), list_to_binary(numbered_key(KeyNumber)),
Value, Value,
IndexGen), IndexGen
),
generate_objects(Count - 1, generate_objects(Count - 1,
{binary, KeyNumber + 1}, {binary, KeyNumber + 1},
[{leveled_rand:uniform(), Obj1, Spec1}|ObjL], [{rand:uniform(), Obj1, Spec1}|ObjL],
Value, Value,
IndexGen, IndexGen,
Bucket); Bucket);
@ -574,7 +604,7 @@ generate_objects(Count, {fixed_binary, KeyNumber}, ObjL, Value, IndexGen, Bucket
IndexGen), IndexGen),
generate_objects(Count - 1, generate_objects(Count - 1,
{fixed_binary, KeyNumber + 1}, {fixed_binary, KeyNumber + 1},
[{leveled_rand:uniform(), Obj1, Spec1}|ObjL], [{rand:uniform(), Obj1, Spec1}|ObjL],
Value, Value,
IndexGen, IndexGen,
Bucket); Bucket);
@ -585,7 +615,7 @@ generate_objects(Count, KeyNumber, ObjL, Value, IndexGen, Bucket) ->
IndexGen), IndexGen),
generate_objects(Count - 1, generate_objects(Count - 1,
KeyNumber + 1, KeyNumber + 1,
[{leveled_rand:uniform(), Obj1, Spec1}|ObjL], [{rand:uniform(), Obj1, Spec1}|ObjL],
Value, Value,
IndexGen, IndexGen,
Bucket). Bucket).
@ -652,7 +682,7 @@ update_some_objects(Bookie, ObjList, SampleSize) ->
[C] = Obj#r_object.contents, [C] = Obj#r_object.contents,
MD = C#r_content.metadata, MD = C#r_content.metadata,
MD0 = dict:store(?MD_LASTMOD, os:timestamp(), MD), MD0 = dict:store(?MD_LASTMOD, os:timestamp(), MD),
C0 = C#r_content{value = leveled_rand:rand_bytes(512), C0 = C#r_content{value = crypto:strong_rand_bytes(512),
metadata = MD0}, metadata = MD0},
UpdObj = Obj#r_object{vclock = VC0, contents = [C0]}, UpdObj = Obj#r_object{vclock = VC0, contents = [C0]},
{R, UpdObj, Spec} {R, UpdObj, Spec}
@ -679,11 +709,11 @@ delete_some_objects(Bookie, ObjList, SampleSize) ->
generate_vclock() -> generate_vclock() ->
lists:map(fun(X) -> lists:map(fun(X) ->
{_, Actor} = lists:keyfind(leveled_rand:uniform(10), {_, Actor} = lists:keyfind(rand:uniform(10),
1, 1,
actor_list()), actor_list()),
{Actor, X} end, {Actor, X} end,
lists:seq(1, leveled_rand:uniform(8))). lists:seq(1, rand:uniform(8))).
update_vclock(VC) -> update_vclock(VC) ->
[{Actor, X}|Rest] = VC, [{Actor, X}|Rest] = VC,
@ -785,14 +815,14 @@ name_list() ->
get_randomname() -> get_randomname() ->
NameList = name_list(), NameList = name_list(),
N = leveled_rand:uniform(16), N = rand:uniform(16),
{N, Name} = lists:keyfind(N, 1, NameList), {N, Name} = lists:keyfind(N, 1, NameList),
Name. Name.
get_randomdate() -> get_randomdate() ->
LowTime = 60000000000, LowTime = 60000000000,
HighTime = 70000000000, HighTime = 70000000000,
RandPoint = LowTime + leveled_rand:uniform(HighTime - LowTime), RandPoint = LowTime + rand:uniform(HighTime - LowTime),
Date = calendar:gregorian_seconds_to_datetime(RandPoint), Date = calendar:gregorian_seconds_to_datetime(RandPoint),
{{Year, Month, Day}, {Hour, Minute, Second}} = Date, {{Year, Month, Day}, {Hour, Minute, Second}} = Date,
lists:flatten(io_lib:format("~4..0w~2..0w~2..0w~2..0w~2..0w~2..0w", lists:flatten(io_lib:format("~4..0w~2..0w~2..0w~2..0w~2..0w~2..0w",

134
test/end_to_end/tictac_SUITE.erl
View file

@ -41,11 +41,14 @@ many_put_compare(_Config) ->
{max_pencillercachesize, 16000}, {max_pencillercachesize, 16000},
{sync_strategy, riak_sync}], {sync_strategy, riak_sync}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1), {ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{B1, K1, V1, S1, MD} = {"Bucket", {B1, K1, V1, S1, MD} =
"Key1.1.4567.4321", {
"Value1", <<"Bucket">>,
<<"Key1.1.4567.4321">>,
<<"Value1">>,
[], [],
[{"MDK1", "MDV1"}]}, [{<<"MDK1">>, <<"MDV1">>}]
},
{TestObject, TestSpec} = testutil:generate_testobject(B1, K1, V1, S1, MD), {TestObject, TestSpec} = testutil:generate_testobject(B1, K1, V1, S1, MD),
ok = testutil:book_riakput(Bookie1, TestObject, TestSpec), ok = testutil:book_riakput(Bookie1, TestObject, TestSpec),
testutil:check_forobject(Bookie1, TestObject), testutil:check_forobject(Bookie1, TestObject),
@ -63,12 +66,15 @@ many_put_compare(_Config) ->
GenList = [2, 20002, 40002, 60002, 80002, GenList = [2, 20002, 40002, 60002, 80002,
100002, 120002, 140002, 160002, 180002], 100002, 120002, 140002, 160002, 180002],
CLs = testutil:load_objects(20000, CLs =
testutil:load_objects(
20000,
GenList, GenList,
Bookie2, Bookie2,
TestObject, TestObject,
fun testutil:generate_smallobjects/2, fun testutil:generate_smallobjects/2,
20000), 20000
),
% Start a new store, and load the same objects (except fot the original % Start a new store, and load the same objects (except fot the original
% test object) into this store % test object) into this store
@ -84,7 +90,7 @@ many_put_compare(_Config) ->
% state between stores is consistent % state between stores is consistent
TicTacQ = {tictactree_obj, TicTacQ = {tictactree_obj,
{o_rkv, "Bucket", null, null, true}, {o_rkv, <<"Bucket">>, null, null, true},
TreeSize, TreeSize,
fun(_B, _K) -> accumulate end}, fun(_B, _K) -> accumulate end},
{async, TreeAFolder} = leveled_bookie:book_returnfolder(Bookie2, TicTacQ), {async, TreeAFolder} = leveled_bookie:book_returnfolder(Bookie2, TicTacQ),
@ -113,10 +119,13 @@ many_put_compare(_Config) ->
true = length(AltList) > 10000, true = length(AltList) > 10000,
% check there are a significant number of differences from empty % check there are a significant number of differences from empty
WrongPartitionTicTacQ = {tictactree_obj, WrongPartitionTicTacQ =
{o_rkv, "Bucket", null, null, false}, {
tictactree_obj,
{o_rkv, <<"Bucket">>, null, null, false},
TreeSize, TreeSize,
fun(_B, _K) -> pass end}, fun(_B, _K) -> pass end
},
{async, TreeAFolder_WP} = {async, TreeAFolder_WP} =
leveled_bookie:book_returnfolder(Bookie2, WrongPartitionTicTacQ), leveled_bookie:book_returnfolder(Bookie2, WrongPartitionTicTacQ),
TreeAWP = TreeAFolder_WP(), TreeAWP = TreeAFolder_WP(),
@ -151,7 +160,7 @@ many_put_compare(_Config) ->
{async, TreeAObjFolder0} = {async, TreeAObjFolder0} =
leveled_bookie:book_headfold(Bookie2, leveled_bookie:book_headfold(Bookie2,
o_rkv, o_rkv,
{range, "Bucket", all}, {range, <<"Bucket">>, all},
FoldAccT, FoldAccT,
false, false,
true, true,
@ -170,7 +179,7 @@ many_put_compare(_Config) ->
leveled_bookie:book_headfold( leveled_bookie:book_headfold(
Bookie2, Bookie2,
?RIAK_TAG, ?RIAK_TAG,
{range, "Bucket", all}, {range, <<"Bucket">>, all},
{FoldObjectsFun, InitAccTree}, {FoldObjectsFun, InitAccTree},
true, true,
true, true,
@ -188,7 +197,7 @@ many_put_compare(_Config) ->
leveled_bookie:book_headfold( leveled_bookie:book_headfold(
Bookie2, Bookie2,
?RIAK_TAG, ?RIAK_TAG,
{range, "Bucket", all}, {range, <<"Bucket">>, all},
{FoldObjectsFun, leveled_tictac:new_tree(0, TreeSize, false)}, {FoldObjectsFun, leveled_tictac:new_tree(0, TreeSize, false)},
true, true,
true, true,
@ -218,27 +227,36 @@ many_put_compare(_Config) ->
end, end,
{async, TreeAAltObjFolder0} = {async, TreeAAltObjFolder0} =
leveled_bookie:book_headfold(Bookie2, leveled_bookie:book_headfold(
Bookie2,
?RIAK_TAG, ?RIAK_TAG,
{range, "Bucket", all}, {range, <<"Bucket">>, all},
{AltFoldObjectsFun, {AltFoldObjectsFun, InitAccTree},
InitAccTree}, false,
false, true, false), true,
false
),
SWB2Obj = os:timestamp(), SWB2Obj = os:timestamp(),
TreeAAltObj = TreeAAltObjFolder0(), TreeAAltObj = TreeAAltObjFolder0(),
io:format("Build tictac tree via object fold with no "++ io:format(
"Build tictac tree via object fold with no "
"presence check and 200K objects and alt hash in ~w~n", "presence check and 200K objects and alt hash in ~w~n",
[timer:now_diff(os:timestamp(), SWB2Obj)]), [timer:now_diff(os:timestamp(), SWB2Obj)]
),
{async, TreeBAltObjFolder0} = {async, TreeBAltObjFolder0} =
leveled_bookie:book_headfold(Bookie3, leveled_bookie:book_headfold(
Bookie3,
?RIAK_TAG, ?RIAK_TAG,
{range, "Bucket", all}, {range, <<"Bucket">>, all},
{AltFoldObjectsFun, {AltFoldObjectsFun, InitAccTree},
InitAccTree}, false,
false, true, false), true,
false
),
SWB3Obj = os:timestamp(), SWB3Obj = os:timestamp(),
TreeBAltObj = TreeBAltObjFolder0(), TreeBAltObj = TreeBAltObjFolder0(),
io:format("Build tictac tree via object fold with no "++ io:format(
"Build tictac tree via object fold with no "
"presence check and 200K objects and alt hash in ~w~n", "presence check and 200K objects and alt hash in ~w~n",
[timer:now_diff(os:timestamp(), SWB3Obj)]), [timer:now_diff(os:timestamp(), SWB3Obj)]),
DL_ExportFold = DL_ExportFold =
@ -261,7 +279,7 @@ many_put_compare(_Config) ->
end end
end end
end, end,
SegQuery = {keylist, o_rkv, "Bucket", {FoldKeysFun(SegList0), []}}, SegQuery = {keylist, o_rkv, <<"Bucket">>, {FoldKeysFun(SegList0), []}},
{async, SegKeyFinder} = {async, SegKeyFinder} =
leveled_bookie:book_returnfolder(Bookie2, SegQuery), leveled_bookie:book_returnfolder(Bookie2, SegQuery),
SWSKL0 = os:timestamp(), SWSKL0 = os:timestamp(),
@ -273,7 +291,7 @@ many_put_compare(_Config) ->
true = length(SegKeyList) >= 1, true = length(SegKeyList) >= 1,
true = length(SegKeyList) < 10, true = length(SegKeyList) < 10,
true = lists:member("Key1.1.4567.4321", SegKeyList), true = lists:member(<<"Key1.1.4567.4321">>, SegKeyList),
% Now remove the object which represents the difference between these % Now remove the object which represents the difference between these
% stores and confirm that the tictac trees will now match % stores and confirm that the tictac trees will now match
@ -630,17 +648,20 @@ tuplebuckets_headonly(_Config) ->
SW1 = os:timestamp(), SW1 = os:timestamp(),
{async, HeadRunner1} = {async, HeadRunner1} =
leveled_bookie:book_headfold(Bookie1, leveled_bookie:book_headfold(
Bookie1,
?HEAD_TAG, ?HEAD_TAG,
{bucket_list, BucketList}, {bucket_list, BucketList},
{FoldHeadFun, []}, {FoldHeadFun, []},
false, false, false, false,
false), false
),
ReturnedObjSpecL1 = lists:reverse(HeadRunner1()), ReturnedObjSpecL1 = lists:reverse(HeadRunner1()),
[FirstItem|_Rest] = ReturnedObjSpecL1, [FirstItem|_Rest] = ReturnedObjSpecL1,
LastItem = lists:last(ReturnedObjSpecL1), LastItem = lists:last(ReturnedObjSpecL1),
io:format("Returned ~w objects with first ~w and last ~w in ~w ms~n", io:format(
"Returned ~w objects with first ~w and last ~w in ~w ms~n",
[length(ReturnedObjSpecL1), [length(ReturnedObjSpecL1),
FirstItem, LastItem, FirstItem, LastItem,
timer:now_diff(os:timestamp(), SW1)/1000]), timer:now_diff(os:timestamp(), SW1)/1000]),
@ -654,12 +675,14 @@ tuplebuckets_headonly(_Config) ->
SW2 = os:timestamp(), SW2 = os:timestamp(),
{async, HeadRunner2} = {async, HeadRunner2} =
leveled_bookie:book_headfold(Bookie1, leveled_bookie:book_headfold(
Bookie1,
?HEAD_TAG, ?HEAD_TAG,
{bucket_list, BucketList}, {bucket_list, BucketList},
{FoldHeadFun, []}, {FoldHeadFun, []},
false, false, false, false,
SegList), SegList
),
ReturnedObjSpecL2 = lists:reverse(HeadRunner2()), ReturnedObjSpecL2 = lists:reverse(HeadRunner2()),
io:format("Returned ~w objects using seglist in ~w ms~n", io:format("Returned ~w objects using seglist in ~w ms~n",
@ -674,7 +697,6 @@ tuplebuckets_headonly(_Config) ->
leveled_bookie:book_destroy(Bookie1). leveled_bookie:book_destroy(Bookie1).
basic_headonly(_Config) -> basic_headonly(_Config) ->
ObjectCount = 200000, ObjectCount = 200000,
RemoveCount = 100, RemoveCount = 100,
@ -694,11 +716,14 @@ basic_headonly_test(ObjectCount, RemoveCount, HeadOnly) ->
{head_only, HeadOnly}, {head_only, HeadOnly},
{max_journalsize, 500000}], {max_journalsize, 500000}],
{ok, Bookie1} = leveled_bookie:book_start(StartOpts1), {ok, Bookie1} = leveled_bookie:book_start(StartOpts1),
{B1, K1, V1, S1, MD} = {"Bucket", {B1, K1, V1, S1, MD} =
"Key1.1.4567.4321", {
"Value1", <<"Bucket">>,
<<"Key1.1.4567.4321">>,
<<"Value1">>,
[], [],
[{"MDK1", "MDV1"}]}, [{<<"MDK1">>, <<"MDV1">>}]
},
{TestObject, TestSpec} = testutil:generate_testobject(B1, K1, V1, S1, MD), {TestObject, TestSpec} = testutil:generate_testobject(B1, K1, V1, S1, MD),
{unsupported_message, put} = {unsupported_message, put} =
testutil:book_riakput(Bookie1, TestObject, TestSpec), testutil:book_riakput(Bookie1, TestObject, TestSpec),
@ -818,23 +843,21 @@ basic_headonly_test(ObjectCount, RemoveCount, HeadOnly) ->
false = is_process_alive(AltSnapshot); false = is_process_alive(AltSnapshot);
no_lookup -> no_lookup ->
{unsupported_message, head} = {unsupported_message, head} =
leveled_bookie:book_head(Bookie1, leveled_bookie:book_head(
SegmentID0, Bookie1, SegmentID0, {Bucket0, Key0}, h),
{Bucket0, Key0},
h),
{unsupported_message, head} = {unsupported_message, head} =
leveled_bookie:book_headonly(Bookie1, leveled_bookie:book_headonly(
SegmentID0, Bookie1, SegmentID0, Bucket0, Key0),
Bucket0,
Key0),
io:format("Closing actual store ~w~n", [Bookie1]), io:format("Closing actual store ~w~n", [Bookie1]),
ok = leveled_bookie:book_close(Bookie1) ok = leveled_bookie:book_close(Bookie1)
end, end,
{ok, FinalJournals} = file:list_dir(JFP), {ok, FinalJournals} = file:list_dir(JFP),
io:format("Trim has reduced journal count from " ++ io:format(
"Trim has reduced journal count from "
"~w to ~w and ~w after restart~n", "~w to ~w and ~w after restart~n",
[length(FNs), length(FinalFNs), length(FinalJournals)]), [length(FNs), length(FinalFNs), length(FinalJournals)]
),
{ok, Bookie2} = leveled_bookie:book_start(StartOpts1), {ok, Bookie2} = leveled_bookie:book_start(StartOpts1),
@ -849,16 +872,12 @@ basic_headonly_test(ObjectCount, RemoveCount, HeadOnly) ->
% If we allow HEAD_TAG to be suubject to a lookup, then test this % If we allow HEAD_TAG to be suubject to a lookup, then test this
% here % here
{ok, Hash0} = {ok, Hash0} =
leveled_bookie:book_head(Bookie2, leveled_bookie:book_head(
SegmentID0, Bookie2, SegmentID0, {Bucket0, Key0}, h);
{Bucket0, Key0},
h);
no_lookup -> no_lookup ->
{unsupported_message, head} = {unsupported_message, head} =
leveled_bookie:book_head(Bookie2, leveled_bookie:book_head(
SegmentID0, Bookie2, SegmentID0, {Bucket0, Key0}, h)
{Bucket0, Key0},
h)
end, end,
RemoveSpecL0 = lists:sublist(ObjectSpecL, RemoveCount), RemoveSpecL0 = lists:sublist(ObjectSpecL, RemoveCount),
@ -873,12 +892,9 @@ basic_headonly_test(ObjectCount, RemoveCount, HeadOnly) ->
true = AccC3 == (ObjectCount - RemoveCount), true = AccC3 == (ObjectCount - RemoveCount),
false = AccH3 == AccH2, false = AccH3 == AccH2,
ok = leveled_bookie:book_close(Bookie2). ok = leveled_bookie:book_close(Bookie2).
load_objectspecs([], _SliceSize, _Bookie) -> load_objectspecs([], _SliceSize, _Bookie) ->
ok; ok;
load_objectspecs(ObjectSpecL, SliceSize, Bookie) load_objectspecs(ObjectSpecL, SliceSize, Bookie)