umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
leveled/src/leveled_sst.erl

2061 lines
82 KiB
Erlang
Raw Normal View History

Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%% -------- SST (Variant) ---------
%%
%% A FSM module intended to wrap a persisted, ordered view of Keys and Values
%%
%% The persisted view is built from a list (which may be created by merging
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%% multiple lists). The list is built first, then the view is created in bulk.
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%%
%% -------- Slots ---------
%%
%% The view is built from sublists referred to as slot. Each slot is up to 128
Comment update
2017-01-25 12:38:33 +00:00
%% keys and values in size. Three strategies have been benchmarked for the
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% slot: a skiplist, a gb-tree, four blocks of flat lists with an index.
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%%
%% Skiplist:
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% build and serialise slot - 3233 microseconds
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%% de-serialise and check * 128 - 14669 microseconds
%% flatten back to list - 164 microseconds
%%
%% GBTree:
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% build and serialise tree - 1433 microseconds
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%% de-serialise and check * 128 - 15263 microseconds
%% flatten back to list - 175 microseconds
%%
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% Indexed Blocks:
%% build and serialise slot 342 microseconds
Comment update
2017-01-25 12:38:33 +00:00
%% de-deserialise and check * 128 - 6746 microseconds
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% flatten back to list - 187 microseconds
%%
%% The negative side of using Indexed Blocks is the storage of the index. In
%% the original implementation this was stored on fadvised disk (the index in
%% this case was a rice-encoded view of which block the object is in). In this
%% implementation it is cached in memory -requiring 2-bytes per key to be kept
%% in memory.
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%%
%% -------- Blooms ---------
%%
Comment update
2017-01-25 12:38:33 +00:00
%% There is a bloom for each slot - based on two hashes and 8 bits per key.
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%%
Comment update
2017-01-25 12:38:33 +00:00
%% Hashing for blooms is a challenge, as the slot is a slice of an ordered
%% list of keys with a fixed format. It is likely that the keys may vary by
%% only one or two ascii characters, and there is a desire to avoid the
%% overhead of cryptographic hash functions that may be able to handle this.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%%
%% -------- Summary ---------
%%
%% Each file has a summary - which is the 128 keys at the top of each slot in
%% a skiplist, with some basic metadata about the slot stored as the value.
%%
Comment update
2017-01-25 12:38:33 +00:00
%% The summary is stored seperately to the slots (within the same file).
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%%
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% -------- CRC Checks ---------
%%
%% Every attempt to either read a summary or a slot off disk will also include
%% a CRC check. If the CRC check fails non-presence is assumed (the data
%% within is assumed to be entirely lost). The data can be recovered by either
%% using a recoverable strategy in transaction log compaction, and triggering
%% the transaction log replay; or by using a higher level for of anti-entropy
%% (i.e. make Riak responsible).
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-module(leveled_sst).
Some basic code layout work
2016-12-24 15:12:24 +00:00
-behaviour(gen_fsm).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-include("include/leveled.hrl").
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
-define(MAX_SLOTS, 256).
Midblock size - lookup No real reason for the midblock to be smaller in lookup slots - so give the blocks a more consistent size
2017-03-21 17:47:08 +00:00
-define(LOOK_SLOTSIZE, 128). % This is not configurable
-define(LOOK_BLOCKSIZE, {24, 32}).
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
-define(NOLOOK_SLOTSIZE, 256).
Midblock size - lookup No real reason for the midblock to be smaller in lookup slots - so give the blocks a more consistent size
2017-03-21 17:47:08 +00:00
-define(NOLOOK_BLOCKSIZE, {56, 32}).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-define(COMPRESSION_LEVEL, 1).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
-define(BINARY_SETTINGS, [{compressed, ?COMPRESSION_LEVEL}]).
Remove SST blooms
2017-01-03 09:12:41 +00:00
% -define(LEVEL_BLOOM_BITS, [{0, 8}, {1, 10}, {2, 8}, {default, 6}]).
Return to merge scan width of 16 This was reduced before the use of binary blocks was committed
2017-03-21 17:53:34 +00:00
-define(MERGE_SCANWIDTH, 16).
Some end-to-end testing
2016-12-24 17:48:31 +00:00
-define(DISCARD_EXT, ".discarded").
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
-define(DELETE_TIMEOUT, 10000).
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
-define(TREE_TYPE, idxt).
-define(TREE_SIZE, 4).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-include_lib("eunit/include/eunit.hrl").
Some basic code layout work
2016-12-24 15:12:24 +00:00
-export([init/1,
handle_sync_event/4,
handle_event/3,
handle_info/3,
terminate/3,
code_change/4,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
starting/2,
Some basic code layout work
2016-12-24 15:12:24 +00:00
starting/3,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
reader/3,
delete_pending/2,
delete_pending/3]).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
-export([sst_new/5,
sst_new/7,
sst_newlevelzero/6,
sst_open/2,
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_get/2,
sst_get/3,
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
sst_getkvrange/4,
sst_getslots/2,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_getmaxsequencenumber/1,
sst_setfordelete/2,
sst_clear/1,
sst_checkready/1,
sst_deleteconfirmed/1,
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_close/1]).
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
-export([expand_list_by_pointer/3]).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Confirm timings
2016-12-23 18:08:22 +00:00
-record(slot_index_value, {slot_id :: integer(),
start_position :: integer(),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
length :: integer(),
bloom :: binary()}).
Confirm timings
2016-12-23 18:08:22 +00:00
Some basic code layout work
2016-12-24 15:12:24 +00:00
-record(summary, {first_key :: tuple(),
last_key :: tuple(),
Cleanup dialyzer errrors in leveled_sst
2017-07-31 19:30:29 +02:00
index :: tuple() | undefined,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
size :: integer(),
Remove SST blooms
2017-01-03 09:12:41 +00:00
max_sqn :: integer()}).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
%% yield_blockquery is used to detemrine if the work necessary to process a
%% range query beyond the fetching the slot should be managed from within
%% this process, or should be handled by the calling process.
%% Handling within the calling process may lead to extra binary heap garbage
%% see Issue 52. Handling within the SST process may lead to contention and
%% extra copying. Files at the top of the tree yield, those lower down don't.
Some basic code layout work
2016-12-24 15:12:24 +00:00
-record(state, {summary,
Cleanup dialyzer errrors in leveled_sst
2017-07-31 19:30:29 +02:00
handle :: file:fd() | undefined,
sst_timings :: tuple() | undefined,
penciller :: pid() | undefined,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
root_path,
Some basic code layout work
2016-12-24 15:12:24 +00:00
filename,
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
yield_blockquery = false :: boolean(),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
blockindex_cache}).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%%%============================================================================
%%% API
%%%============================================================================
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_open(string(), string()) -> {ok, pid(), {tuple(), tuple()}}.
%% @doc
%% Open an SST file at a given path and filename. The first and last keys
%% are returned in response to the request - so that those keys can be used
%% in manifests to understand what range of keys are covered by the SST file.
%% All keys in the file should be between the first and last key in erlang
%% term order.
%%
%% The filename should include the file extension.
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
sst_open(RootPath, Filename) ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
case gen_fsm:sync_send_event(Pid,
{sst_open, RootPath, Filename},
infinity) of
Some basic code layout work
2016-12-24 15:12:24 +00:00
{ok, {SK, EK}} ->
{ok, Pid, {SK, EK}}
end.
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_new(string(), string(), integer(), list(), integer()) ->
{ok, pid(), {tuple(), tuple()}}.
%% @doc
%% Start a new SST file at the assigned level passing in a list of Key, Value
%% pairs. This should not be used for basement levels or unexpanded Key/Value
%% lists as merge_lists will not be called.
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
sst_new(RootPath, Filename, Level, KVList, MaxSQN) ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{[], [], SlotList, FK} = merge_lists(KVList),
Some basic code layout work
2016-12-24 15:12:24 +00:00
case gen_fsm:sync_send_event(Pid,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{sst_new,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
RootPath,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Filename,
Level,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{SlotList, FK},
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
MaxSQN},
Some basic code layout work
2016-12-24 15:12:24 +00:00
infinity) of
{ok, {SK, EK}} ->
{ok, Pid, {SK, EK}}
end.
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_new(string(), string(), list(), list(),
boolean(), integer(), integer()) ->
empty|{ok, pid(), {{list(), list()}, tuple(), tuple()}}.
%% @doc
%% Start a new SST file at the assigned level passing in a two lists of
%% {Key, Value} pairs to be merged. The merge_lists function will use the
%% IsBasement boolean to determine if expired keys or tombstones can be
%% deleted.
%%
%% The remainder of the lists is returned along with the StartKey and EndKey
%% so that the remainder cna be used in the next file in the merge. It might
%% be that the merge_lists returns nothin (for example when a basement file is
%% all tombstones) - and the atome empty is returned in this case so that the
%% file is not added to the manifest.
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
sst_new(RootPath, Filename, KVL1, KVL2, IsBasement, Level, MaxSQN) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{Rem1, Rem2, SlotList, FK} = merge_lists(KVL1, KVL2, {IsBasement, Level}),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case SlotList of
Fix empty file again No special definition of empty required, as now an empty list when empty
2017-03-14 00:17:09 +00:00
[] ->
Handle production of empty file
2016-12-29 05:09:47 +00:00
empty;
_ ->
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
case gen_fsm:sync_send_event(Pid,
{sst_new,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
RootPath,
Handle production of empty file
2016-12-29 05:09:47 +00:00
Filename,
Level,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{SlotList, FK},
Handle production of empty file
2016-12-29 05:09:47 +00:00
MaxSQN},
infinity) of
{ok, {SK, EK}} ->
{ok, Pid, {{Rem1, Rem2}, SK, EK}}
end
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
end.
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_newlevelzero(string(), string(),
integer(), fun(), pid()|undefined, integer()) ->
{ok, pid(), noreply}.
%% @doc
%% Start a new file at level zero. At this level the file size is not fixed -
%% it will be as big as the input. Also the KVList is not passed in, it is
%% fetched slot by slot using the FetchFun
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
sst_newlevelzero(RootPath, Filename, Slots, FetchFun, Penciller, MaxSQN) ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
gen_fsm:send_event(Pid,
{sst_newlevelzero,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
RootPath,
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Filename,
Slots,
FetchFun,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Penciller,
MaxSQN}),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{ok, Pid, noreply}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_get(pid(), tuple()) -> tuple()|not_present.
%% @doc
%% Return a Key, Value pair matching a Key or not_present if the Key is not in
%% the store. The magic_hash function is used to accelerate the seeking of
%% keys, sst_get/3 should be used directly if this has already been calculated
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_get(Pid, LedgerKey) ->
sst_get(Pid, LedgerKey, leveled_codec:magic_hash(LedgerKey)).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_get(pid(), tuple(), integer()) -> tuple()|not_present.
%% @doc
%% Return a Key, Value pair matching a Key or not_present if the Key is not in
%% the store (with the magic hash precalculated).
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_get(Pid, LedgerKey, Hash) ->
gen_fsm:sync_send_event(Pid, {get_kv, LedgerKey, Hash}, infinity).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_getkvrange(pid(), tuple()|all, tuple()|all, integer()) -> list().
%% @doc
%% Get a range of {Key, Value} pairs as a list between StartKey and EndKey
%% (inclusive). The ScanWidth is the maximum size of the range, a pointer
%% will be placed on the tail of the resulting list if results expand beyond
%% the Scan Width
%%
%% To make the range open-ended (either ta start, end or both) the all atom
%% can be use din place of the Key tuple.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
sst_getkvrange(Pid, StartKey, EndKey, ScanWidth) ->
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
case gen_fsm:sync_send_event(Pid,
Range queries - pass out as binaries Avoid converting to erlang temr wihtin the FSM and then requiring a copy outside of the FSM - pass out as a binary
2017-03-17 10:47:20 +00:00
{get_kvrange, StartKey, EndKey, ScanWidth},
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
infinity) of
{yield, SlotsToFetchBinList, SlotsToPoint} ->
FetchFun =
fun({SlotBin, SK, EK}, Acc) ->
Acc ++ binaryslot_trimmedlist(SlotBin, SK, EK)
end,
lists:foldl(FetchFun, [], SlotsToFetchBinList) ++ SlotsToPoint;
Reply ->
Reply
end.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_getslots(pid(), list()) -> list().
%% @doc
%% Get a list of slots by their ID. The slot will be converted from the binary
%% to term form outside of the FSM loop
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
sst_getslots(Pid, SlotList) ->
Pass out slots as a binary If we convetr firts to a list, then the list has to be copied - passing out as binaries means the bulk can be passed as references
2017-03-17 10:43:34 +00:00
SlotBins = gen_fsm:sync_send_event(Pid, {get_slots, SlotList}, infinity),
FetchFun =
fun({SlotBin, SK, EK}, Acc) ->
Acc ++ binaryslot_trimmedlist(SlotBin, SK, EK)
end,
lists:foldl(FetchFun, [], SlotBins).
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_getmaxsequencenumber(pid()) -> integer().
%% @doc
%% Get the maximume sequence number for this SST file
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_getmaxsequencenumber(Pid) ->
gen_fsm:sync_send_event(Pid, get_maxsequencenumber, infinity).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_setfordelete(pid(), pid()|false) -> ok.
%% @doc
%% If the SST is no longer in use in the active ledger it can be set for
%% delete. Once set for delete it will poll the Penciller pid to see if
%% it is yet safe to be deleted (i.e. because all snapshots which depend
%% on it have finished). No polling will be done if the Penciller pid
%% is 'false'
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_setfordelete(Pid, Penciller) ->
gen_fsm:sync_send_event(Pid, {set_for_delete, Penciller}, infinity).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_clear(pid()) -> ok.
%% @doc
%% For this file to be closed and deleted
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_clear(Pid) ->
gen_fsm:sync_send_event(Pid, {set_for_delete, false}, infinity),
gen_fsm:sync_send_event(Pid, close, 1000).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_deleteconfirmed(pid()) -> ok.
%% @doc
%% Allows a penciller to confirm to a SST file that it can be cleared, as it
%% is no longer in use
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_deleteconfirmed(Pid) ->
gen_fsm:send_event(Pid, close).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_checkready(pid()) -> {ok, string(), tuple(), tuple()}.
%% @doc
%% If a file has been set to be built, check that it has been built. Returns
%% the filename and the {startKey, EndKey} for the manifest.
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_checkready(Pid) ->
%% Only used in test
gen_fsm:sync_send_event(Pid, background_complete, 100).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_close(pid()) -> ok.
%% @doc
%% Close the file
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_close(Pid) ->
gen_fsm:sync_send_event(Pid, close, 2000).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_printtimings(pid()) -> ok.
%% @doc
%% The state of the FSM keeps track of timings of operations, and this can
%% forced to be printed.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
%% Used in unit tests to force the printing of timings
sst_printtimings(Pid) ->
gen_fsm:sync_send_event(Pid, print_timings, 1000).
Some basic code layout work
2016-12-24 15:12:24 +00:00
%%%============================================================================
%%% gen_server callbacks
%%%============================================================================
init([]) ->
{ok, starting, #state{}}.
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
starting({sst_open, RootPath, Filename}, _From, State) ->
UpdState = read_file(Filename, State#state{root_path=RootPath}),
Some basic code layout work
2016-12-24 15:12:24 +00:00
Summary = UpdState#state.summary,
{reply,
{ok, {Summary#summary.first_key, Summary#summary.last_key}},
reader,
UpdState};
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
starting({sst_new, RootPath, Filename, Level, {SlotList, FirstKey}, MaxSQN},
_From, State) ->
Add timer for SST creation
2016-12-29 02:55:28 +00:00
SW = os:timestamp(),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{Length,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SlotIndex,
BlockIndex,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
SlotsBin} = build_all_slots(SlotList),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
SummaryBin = build_table_summary(SlotIndex,
Level,
FirstKey,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Length,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
MaxSQN),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ActualFilename = write_file(RootPath, Filename, SummaryBin, SlotsBin),
Up level for yield to 2
2017-03-21 18:24:11 +00:00
YBQ = Level =< 2,
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
UpdState = read_file(ActualFilename,
State#state{root_path=RootPath,
yield_blockquery=YBQ}),
Some basic code layout work
2016-12-24 15:12:24 +00:00
Summary = UpdState#state.summary,
Add timer for SST creation
2016-12-29 02:55:28 +00:00
leveled_log:log_timer("SST08",
[ActualFilename, Level, Summary#summary.max_sqn],
SW),
Some basic code layout work
2016-12-24 15:12:24 +00:00
{reply,
{ok, {Summary#summary.first_key, Summary#summary.last_key}},
reader,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
UpdState#state{blockindex_cache = BlockIndex}}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
starting({sst_newlevelzero, RootPath, Filename,
Slots, FetchFun, Penciller, MaxSQN}, State) ->
Add timer for SST creation
2016-12-29 02:55:28 +00:00
SW = os:timestamp(),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
KVList = leveled_pmem:to_list(Slots, FetchFun),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{[], [], SlotList, FirstKey} = merge_lists(KVList),
{SlotCount,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SlotIndex,
BlockIndex,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
SlotsBin} = build_all_slots(SlotList),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
SummaryBin = build_table_summary(SlotIndex,
0,
FirstKey,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
SlotCount,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
MaxSQN),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ActualFilename = write_file(RootPath, Filename, SummaryBin, SlotsBin),
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
UpdState = read_file(ActualFilename,
State#state{root_path = RootPath,
yield_blockquery = true}),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Summary = UpdState#state.summary,
Add timer for SST creation
2016-12-29 02:55:28 +00:00
leveled_log:log_timer("SST08",
[ActualFilename, 0, Summary#summary.max_sqn],
SW),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
case Penciller of
undefined ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{next_state, reader, UpdState#state{blockindex_cache = BlockIndex}};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
_ ->
leveled_penciller:pcl_confirml0complete(Penciller,
UpdState#state.filename,
Summary#summary.first_key,
Summary#summary.last_key),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{next_state, reader, UpdState#state{blockindex_cache = BlockIndex}}
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
end.
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Some basic code layout work
2016-12-24 15:12:24 +00:00
reader({get_kv, LedgerKey, Hash}, _From, State) ->
SW = os:timestamp(),
Remove lastfetch cache It appears to have some benefit at lower levels, but overall has less benefit at higher levels. Probably not worth having unless it cna be controlled to go in at the basement only.
2017-01-03 15:26:44 +00:00
{Result, Stage, _SlotID, UpdState} = fetch(LedgerKey, Hash, State),
Some basic code layout work
2016-12-24 15:12:24 +00:00
UpdTimings = leveled_log:sst_timing(State#state.sst_timings, SW, Stage),
Remove lastfetch cache It appears to have some benefit at lower levels, but overall has less benefit at higher levels. Probably not worth having unless it cna be controlled to go in at the basement only.
2017-01-03 15:26:44 +00:00
{reply, Result, reader, UpdState#state{sst_timings = UpdTimings}};
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
reader({get_kvrange, StartKey, EndKey, ScanWidth}, _From, State) ->
Move sst get_kv range back inside process Moved outside to stop blocking, but also avoids copy. Move back out to see if it may be related to the binary memory leak
2017-03-20 23:22:46 +00:00
{SlotsToFetchBinList, SlotsToPoint} = fetch_range(StartKey,
EndKey,
ScanWidth,
State),
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
case State#state.yield_blockquery of
true ->
{reply,
{yield, SlotsToFetchBinList, SlotsToPoint},
reader,
State};
false ->
FetchFun =
fun({SlotBin, SK, EK}, Acc) ->
Acc ++ binaryslot_trimmedlist(SlotBin, SK, EK)
end,
{reply,
lists:foldl(FetchFun, [], SlotsToFetchBinList) ++ SlotsToPoint,
reader,
State}
end;
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
reader({get_slots, SlotList}, _From, State) ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
SlotBins = read_slots(State#state.handle, SlotList),
Pass out slots as a binary If we convetr firts to a list, then the list has to be copied - passing out as binaries means the bulk can be passed as references
2017-03-17 10:43:34 +00:00
{reply, SlotBins, reader, State};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
reader(get_maxsequencenumber, _From, State) ->
Summary = State#state.summary,
{reply, Summary#summary.max_sqn, reader, State};
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
reader(print_timings, _From, State) ->
Check query out of range It doesn't work - so protecting against it in fetch_range is pointless, will blow up in lookup_slots
2017-03-16 08:43:18 +00:00
io:format(user, "~nTimings of ~w~n", [State#state.sst_timings]),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
{reply, ok, reader, State#state{sst_timings = undefined}};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
reader({set_for_delete, Penciller}, _From, State) ->
leveled_log:log("SST06", [State#state.filename]),
{reply,
ok,
delete_pending,
State#state{penciller=Penciller},
?DELETE_TIMEOUT};
reader(background_complete, _From, State) ->
Summary = State#state.summary,
{reply,
{ok,
State#state.filename,
Summary#summary.first_key,
Summary#summary.last_key},
reader,
State};
Some end-to-end testing
2016-12-24 17:48:31 +00:00
reader(close, _From, State) ->
ok = file:close(State#state.handle),
{stop, normal, ok, State}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
delete_pending({get_kv, LedgerKey, Hash}, _From, State) ->
Remove lastfetch cache It appears to have some benefit at lower levels, but overall has less benefit at higher levels. Probably not worth having unless it cna be controlled to go in at the basement only.
2017-01-03 15:26:44 +00:00
{Result, _Stage, _SlotID, UpdState} = fetch(LedgerKey, Hash, State),
{reply, Result, delete_pending, UpdState, ?DELETE_TIMEOUT};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
delete_pending({get_kvrange, StartKey, EndKey, ScanWidth}, _From, State) ->
Fix to delete_pending state
2017-03-20 23:43:31 +00:00
{SlotsToFetchBinList, SlotsToPoint} = fetch_range(StartKey,
EndKey,
ScanWidth,
State),
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
% Always yield as about to clear and de-reference
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{reply,
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
{yield, SlotsToFetchBinList, SlotsToPoint},
State mixup in FSM
2016-12-29 02:40:09 +00:00
delete_pending,
State,
?DELETE_TIMEOUT};
delete_pending({get_slots, SlotList}, _From, State) ->
SlotBins = read_slots(State#state.handle, SlotList),
Pass out slots as a binary If we convetr firts to a list, then the list has to be copied - passing out as binaries means the bulk can be passed as references
2017-03-17 10:43:34 +00:00
{reply, SlotBins, delete_pending, State, ?DELETE_TIMEOUT};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
delete_pending(close, _From, State) ->
leveled_log:log("SST07", [State#state.filename]),
ok = file:close(State#state.handle),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:delete(filename:join(State#state.root_path,
State#state.filename)),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{stop, normal, ok, State}.
delete_pending(timeout, State) ->
ok = leveled_penciller:pcl_confirmdelete(State#state.penciller,
Fix-up message exchange on confirm delete
2017-01-17 11:18:58 +00:00
State#state.filename,
self()),
Timeout long-running snapshots Add logic to timeout long-running snapshots.
2017-04-05 09:16:01 +01:00
% If the next thing is another timeout - may be long-running snapshot, so
% back-off
{next_state, delete_pending, State, random:uniform(10) * ?DELETE_TIMEOUT};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
delete_pending(close, State) ->
leveled_log:log("SST07", [State#state.filename]),
ok = file:close(State#state.handle),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:delete(filename:join(State#state.root_path,
State#state.filename)),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{stop, normal, State}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
handle_sync_event(_Msg, _From, StateName, State) ->
{reply, undefined, StateName, State}.
handle_event(_Msg, StateName, State) ->
{next_state, StateName, State}.
handle_info(_Msg, StateName, State) ->
{next_state, StateName, State}.
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
terminate(normal, delete_pending, _State) ->
ok;
Some basic code layout work
2016-12-24 15:12:24 +00:00
terminate(Reason, _StateName, State) ->
leveled_log:log("SST04", [Reason, State#state.filename]).
code_change(_OldVsn, StateName, State, _Extra) ->
{ok, StateName, State}.
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%%%============================================================================
%%% Internal Functions
%%%============================================================================
Some basic code layout work
2016-12-24 15:12:24 +00:00
fetch(LedgerKey, Hash, State) ->
Summary = State#state.summary,
Remove SST blooms
2017-01-03 09:12:41 +00:00
Slot = lookup_slot(LedgerKey, Summary#summary.index),
SlotID = Slot#slot_index_value.slot_id,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
Bloom = Slot#slot_index_value.bloom,
case leveled_tinybloom:check_hash(Hash, Bloom) of
false ->
{not_present, tiny_bloom, SlotID, State};
true ->
CachedBlockIdx = array:get(SlotID - 1,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
State#state.blockindex_cache),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
case CachedBlockIdx of
none ->
Remove lastfetch cache It appears to have some benefit at lower levels, but overall has less benefit at higher levels. Probably not worth having unless it cna be controlled to go in at the basement only.
2017-01-03 15:26:44 +00:00
SlotBin = read_slot(State#state.handle, Slot),
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{Result,
BlockLengths,
BlockIdx} = binaryslot_get(SlotBin, LedgerKey, Hash),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
BlockIndexCache = array:set(SlotID - 1,
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
<<BlockLengths/binary,
BlockIdx/binary>>,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
State#state.blockindex_cache),
{Result,
slot_fetch,
Slot#slot_index_value.slot_id,
State#state{blockindex_cache = BlockIndexCache}};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<BlockLengths:24/binary, BlockIdx/binary>> ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
PosList = find_pos(BlockIdx,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
double_hash(Hash, LedgerKey),
[],
0),
case PosList of
[] ->
{not_present, slot_bloom, SlotID, State};
_ ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Result = check_blocks(PosList,
State#state.handle,
Slot,
BlockLengths,
LedgerKey),
{Result, slot_fetch, SlotID, State}
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
end
end
Some basic code layout work
2016-12-24 15:12:24 +00:00
end.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
fetch_range(StartKey, EndKey, ScanWidth, State) ->
Summary = State#state.summary,
Handle = State#state.handle,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
{Slots, RTrim} = lookup_slots(StartKey, EndKey, Summary#summary.index),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
Self = self(),
SL = length(Slots),
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
ExpandedSlots =
case SL of
1 ->
[Slot] = Slots,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
case RTrim of
true ->
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
[{pointer, Self, Slot, StartKey, EndKey}];
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
false ->
[{pointer, Self, Slot, StartKey, all}]
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
end;
N ->
{LSlot, MidSlots, RSlot} =
case N of
2 ->
[Slot1, Slot2] = Slots,
{Slot1, [], Slot2};
N ->
[Slot1|_Rest] = Slots,
SlotN = lists:last(Slots),
{Slot1, lists:sublist(Slots, 2, N - 2), SlotN}
end,
MidSlotPointers = lists:map(fun(S) ->
{pointer, Self, S, all, all}
end,
MidSlots),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
case RTrim of
true ->
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
[{pointer, Self, LSlot, StartKey, all}] ++
MidSlotPointers ++
[{pointer, Self, RSlot, all, EndKey}];
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
false ->
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
[{pointer, Self, LSlot, StartKey, all}] ++
MidSlotPointers ++
[{pointer, Self, RSlot, all, all}]
end
end,
{SlotsToFetch, SlotsToPoint} =
case ScanWidth of
SW when SW >= SL ->
{ExpandedSlots, []};
_ ->
lists:split(ScanWidth, ExpandedSlots)
end,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SlotsToFetchBinList = read_slots(Handle, SlotsToFetch),
Range queries - pass out as binaries Avoid converting to erlang temr wihtin the FSM and then requiring a copy outside of the FSM - pass out as a binary
2017-03-17 10:47:20 +00:00
{SlotsToFetchBinList, SlotsToPoint}.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
Some basic code layout work
2016-12-24 15:12:24 +00:00
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
write_file(RootPath, Filename, SummaryBin, SlotsBin) ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
SummaryLength = byte_size(SummaryBin),
SlotsLength = byte_size(SlotsBin),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
{PendingName, FinalName} = generate_filenames(Filename),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:write_file(filename:join(RootPath, PendingName),
Fix to ensure directory structure created
2016-12-29 02:31:10 +00:00
<<SlotsLength:32/integer,
SummaryLength:32/integer,
SlotsBin/binary,
SummaryBin/binary>>,
[raw]),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
case filelib:is_file(filename:join(RootPath, FinalName)) of
Some end-to-end testing
2016-12-24 17:48:31 +00:00
true ->
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
AltName = filename:join(RootPath, filename:basename(FinalName))
Some end-to-end testing
2016-12-24 17:48:31 +00:00
++ ?DISCARD_EXT,
leveled_log:log("SST05", [FinalName, AltName]),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:rename(filename:join(RootPath, FinalName), AltName);
Some end-to-end testing
2016-12-24 17:48:31 +00:00
false ->
ok
end,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
file:rename(filename:join(RootPath, PendingName),
filename:join(RootPath, FinalName)),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
FinalName.
Some basic code layout work
2016-12-24 15:12:24 +00:00
read_file(Filename, State) ->
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{Handle, SummaryBin} = open_reader(filename:join(State#state.root_path,
Filename)),
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
{Summary, SlotList} = read_table_summary(SummaryBin),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
BlockIndexCache = array:new([{size, Summary#summary.size},
{default, none}]),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
UpdState = State#state{blockindex_cache = BlockIndexCache},
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
SlotIndex = from_list(SlotList),
UpdSummary = Summary#summary{index = SlotIndex},
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
leveled_log:log("SST03", [Filename,
Summary#summary.size,
Summary#summary.max_sqn]),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
UpdState#state{summary = UpdSummary,
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
handle = Handle,
filename = Filename}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
open_reader(Filename) ->
{ok, Handle} = file:open(Filename, [binary, raw, read]),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
{ok, Lengths} = file:pread(Handle, 0, 8),
Some basic code layout work
2016-12-24 15:12:24 +00:00
<<SlotsLength:32/integer, SummaryLength:32/integer>> = Lengths,
Some end-to-end testing
2016-12-24 17:48:31 +00:00
{ok, SummaryBin} = file:pread(Handle, SlotsLength + 8, SummaryLength),
Some basic code layout work
2016-12-24 15:12:24 +00:00
{Handle, SummaryBin}.
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
build_table_summary(SlotIndex, _Level, FirstKey, SlotCount, MaxSQN) ->
[{LastKey, _LastV}|_Rest] = SlotIndex,
Some basic code layout work
2016-12-24 15:12:24 +00:00
Summary = #summary{first_key = FirstKey,
last_key = LastKey,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
size = SlotCount,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
max_sqn = MaxSQN},
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
SummBin = term_to_binary({Summary, lists:reverse(SlotIndex)},
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
?BINARY_SETTINGS),
Build the table summary The table summary will be a skiplist, and this and the slot binary will be CRC checked
2016-12-24 01:23:40 +00:00
SummCRC = erlang:crc32(SummBin),
<<SummCRC:32/integer, SummBin/binary>>.
read_table_summary(BinWithCheck) ->
<<SummCRC:32/integer, SummBin/binary>> = BinWithCheck,
CRCCheck = erlang:crc32(SummBin),
if
CRCCheck == SummCRC ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
% If not might it might be possible to rebuild from all the slots
Build the table summary The table summary will be a skiplist, and this and the slot binary will be CRC checked
2016-12-24 01:23:40 +00:00
binary_to_term(SummBin)
end.
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
build_all_slots(SlotList) ->
SlotCount = length(SlotList),
BuildResponse = build_all_slots(SlotList,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
8,
1,
[],
array:new([{size, SlotCount},
{default, none}]),
<<>>),
Tidy Up All Hashes As we're no longer generating a summayr bloom - no need to collect a big list of hashes whilst building the sst file
2017-01-03 18:20:28 +00:00
{SlotIndex, BlockIndex, SlotsBin} = BuildResponse,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{SlotCount, SlotIndex, BlockIndex, SlotsBin}.
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
build_all_slots([], _Pos, _SlotID,
SlotIdxAcc, BlockIdxAcc, SlotBinAcc) ->
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{SlotIdxAcc, BlockIdxAcc, SlotBinAcc};
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
build_all_slots([SlotD|Rest], Pos, SlotID,
SlotIdxAcc, BlockIdxAcc, SlotBinAcc) ->
{BlockIdx, SlotBin, HashList, LastKey} = SlotD,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Length = byte_size(SlotBin),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
Bloom = leveled_tinybloom:create_bloom(HashList),
Confirm timings
2016-12-23 18:08:22 +00:00
SlotIndexV = #slot_index_value{slot_id = SlotID,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
start_position = Pos,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
length = Length,
bloom = Bloom},
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
build_all_slots(Rest,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Pos + Length,
Confirm timings
2016-12-23 18:08:22 +00:00
SlotID + 1,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
[{LastKey, SlotIndexV}|SlotIdxAcc],
array:set(SlotID - 1, BlockIdx, BlockIdxAcc),
<<SlotBinAcc/binary, SlotBin/binary>>).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
Some end-to-end testing
2016-12-24 17:48:31 +00:00
generate_filenames(RootFilename) ->
Ext = filename:extension(RootFilename),
Components = filename:split(RootFilename),
case Ext of
[] ->
{filename:join(Components) ++ ".pnd",
filename:join(Components) ++ ".sst"};
Ext ->
DN = filename:dirname(RootFilename),
FP_NOEXT = filename:basename(RootFilename, Ext),
{filename:join(DN, FP_NOEXT) ++ ".pnd",
filename:join(DN, FP_NOEXT) ++ ".sst"}
end.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
%%%============================================================================
%%% SlotIndex Implementation
%%%============================================================================
%% The Slot Index is stored as a flat (sorted) list of {Key, Slot} where Key
%% is the last key within the slot.
%%
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
%% This implementation of the SlotIndex uses leveled_tree
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
from_list(SlotList) ->
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
leveled_tree:from_orderedlist(SlotList, ?TREE_TYPE, ?TREE_SIZE).
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
lookup_slot(Key, Tree) ->
StartKeyFun =
fun(_V) ->
all
end,
% The penciller should never ask for presence out of range - so will
% always return a slot (As we don't compare to StartKey)
{_LK, Slot} = leveled_tree:search(Key, Tree, StartKeyFun),
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
Slot.
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
lookup_slots(StartKey, EndKey, Tree) ->
StartKeyFun =
fun(_V) ->
all
end,
MapFun =
fun({_LK, Slot}) ->
Slot
end,
SlotList = leveled_tree:search_range(StartKey, EndKey, Tree, StartKeyFun),
{EK, _EndSlot} = lists:last(SlotList),
RTrim RTrim only worked in special case of key matching, that would never occur in real world range query. RTrim should really check for key passing. Returning empty list should not be possible - unless the query is outside of the range entirely (and such a query should never go to this SST).
2017-03-16 08:37:36 +00:00
{lists:map(MapFun, SlotList), not leveled_codec:endkey_passed(EK, EndKey)}.
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%%%============================================================================
%%% Slot Implementation
%%%============================================================================
%% Implementing a slot has gone through numerous iterations. One of the most
%% critical considerations has been the cost of the binary_to_term and
%% term_to_binary calls for different sizes of slots and different data types.
%%
Update SST comments
2017-03-04 20:47:46 +00:00
%% Microbenchmarking indicated that flat lists were the fastest at sst build
%% time. However, the lists need scanning at query time - and so give longer
%% lookups. Bigger slots did better at term_to_binary time. However
%% binary_to_term is an often repeated task, and this is better with smaller
%% slots.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%%
%% The outcome has been to divide the slot into four small blocks to minimise
%% the binary_to_term time. A binary index is provided for the slot for all
%% Keys that are directly fetchable (i.e. standard keys not index keys).
%%
%% The division and use of a list saves about 100 microseconds per fetch when
%% compared to using a 128-member gb:tree.
%%
%% The binary index is cacheable and doubles as a not_present filter, as it is
%% based on a 17-bit hash (so 0.0039 fpr).
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
generate_binary_slot(Lookup, KVL) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
HashFoldFun =
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
fun({K, V}, {PosBinAcc, NoHashCount, HashAcc}) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{_SQN, H1} = leveled_codec:strip_to_seqnhashonly({K, V}),
case is_integer(H1) of
true ->
PosH1 = double_hash(H1, K),
case NoHashCount of
0 ->
Tidy Up All Hashes As we're no longer generating a summayr bloom - no need to collect a big list of hashes whilst building the sst file
2017-01-03 18:20:28 +00:00
{<<1:1/integer,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
PosH1:15/integer,
PosBinAcc/binary>>,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
0,
[H1|HashAcc]};
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
N ->
% The No Hash Count is an integer between 0 and 127
% and so at read time should count NHC + 1
NHC = N - 1,
Tidy Up All Hashes As we're no longer generating a summayr bloom - no need to collect a big list of hashes whilst building the sst file
2017-01-03 18:20:28 +00:00
{<<1:1/integer,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
PosH1:15/integer,
0:1/integer,
NHC:7/integer,
PosBinAcc/binary>>,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
0,
HashAcc}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end;
false ->
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
{PosBinAcc, NoHashCount + 1, HashAcc}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end
end,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{HashL, PosBinIndex} =
case Lookup of
lookup ->
{PosBinIndex0,
NHC,
HashL0} = lists:foldr(HashFoldFun, {<<>>, 0, []}, KVL),
PosBinIndex1 =
case NHC of
0 ->
PosBinIndex0;
_ ->
N = NHC - 1,
<<0:1/integer, N:7/integer, PosBinIndex0/binary>>
end,
{HashL0, PosBinIndex1};
no_lookup ->
{[], <<0:1/integer, 127:7/integer>>}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{SideBlockSize, MidBlockSize} =
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case Lookup of
lookup ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?LOOK_BLOCKSIZE;
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
no_lookup ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?NOLOOK_BLOCKSIZE
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{B1, B2, B3, B4, B5} =
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
case length(KVL) of
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< SideBlockSize ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{term_to_binary(KVL, ?BINARY_SETTINGS),
<<0:0>>,
<<0:0>>,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<0:0>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< 2 * SideBlockSize ->
{KVLA, KVLB} = lists:split(SideBlockSize, KVL),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{term_to_binary(KVLA, ?BINARY_SETTINGS),
term_to_binary(KVLB, ?BINARY_SETTINGS),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<0:0>>,
<<0:0>>};
L when L =< (2 * SideBlockSize + MidBlockSize) ->
{KVLA, KVLB_Rest} = lists:split(SideBlockSize, KVL),
{KVLB, KVLC} = lists:split(SideBlockSize, KVLB_Rest),
{term_to_binary(KVLA, ?BINARY_SETTINGS),
term_to_binary(KVLB, ?BINARY_SETTINGS),
term_to_binary(KVLC, ?BINARY_SETTINGS),
<<0:0>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< (3 * SideBlockSize + MidBlockSize) ->
{KVLA, KVLB_Rest} = lists:split(SideBlockSize, KVL),
{KVLB, KVLC_Rest} = lists:split(SideBlockSize, KVLB_Rest),
{KVLC, KVLD} = lists:split(MidBlockSize, KVLC_Rest),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{term_to_binary(KVLA, ?BINARY_SETTINGS),
term_to_binary(KVLB, ?BINARY_SETTINGS),
term_to_binary(KVLC, ?BINARY_SETTINGS),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
term_to_binary(KVLD, ?BINARY_SETTINGS),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< (4 * SideBlockSize + MidBlockSize) ->
{KVLA, KVLB_Rest} = lists:split(SideBlockSize, KVL),
{KVLB, KVLC_Rest} = lists:split(SideBlockSize, KVLB_Rest),
{KVLC, KVLD_Rest} = lists:split(MidBlockSize, KVLC_Rest),
{KVLD, KVLE} = lists:split(SideBlockSize, KVLD_Rest),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{term_to_binary(KVLA, ?BINARY_SETTINGS),
term_to_binary(KVLB, ?BINARY_SETTINGS),
term_to_binary(KVLC, ?BINARY_SETTINGS),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
term_to_binary(KVLD, ?BINARY_SETTINGS),
term_to_binary(KVLE, ?BINARY_SETTINGS)}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
B1P = byte_size(PosBinIndex),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
B1L = byte_size(B1),
B2L = byte_size(B2),
B3L = byte_size(B3),
B4L = byte_size(B4),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B5L = byte_size(B5),
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Lengths = <<B1P:32/integer,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B4L:32/integer,
B5L:32/integer>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SlotBin = <<Lengths/binary,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
PosBinIndex/binary,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B1/binary, B2/binary, B3/binary, B4/binary, B5/binary>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
CRC32 = erlang:crc32(SlotBin),
FullBin = <<CRC32:32/integer, SlotBin/binary>>,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{LastKey, _LV} = lists:last(KVL),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{<<Lengths/binary, PosBinIndex/binary>>, FullBin, HashL, LastKey}.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
check_blocks([], _Handle, _Slot, _BlockLengths, _LedgerKey) ->
not_present;
check_blocks([Pos|Rest], Handle, Slot, BlockLengths, LedgerKey) ->
{BlockNumber, BlockPos} = revert_position(Pos),
BlockBin = read_block(Handle, Slot, BlockLengths, BlockNumber),
BlockL = binary_to_term(BlockBin),
{K, V} = lists:nth(BlockPos, BlockL),
case K of
LedgerKey ->
{K, V};
_ ->
check_blocks(Rest, Handle, Slot, BlockLengths, LedgerKey)
end.
read_block(Handle, Slot, BlockLengths, BlockID) ->
{BlockPos, Offset, Length} = block_offsetandlength(BlockLengths, BlockID),
{ok, BlockBin} = file:pread(Handle,
Slot#slot_index_value.start_position
+ BlockPos
+ Offset
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
+ 28,
% 4-byte CRC, 4 byte pos, 5x4 byte lengths
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Length),
BlockBin.
read_slot(Handle, Slot) ->
{ok, SlotBin} = file:pread(Handle,
Slot#slot_index_value.start_position,
Slot#slot_index_value.length),
SlotBin.
read_slots(Handle, SlotList) ->
PointerMapFun =
fun(Pointer) ->
{Slot, SK, EK} =
case Pointer of
{pointer, _Pid, Slot0, SK0, EK0} ->
{Slot0, SK0, EK0};
{pointer, Slot0, SK0, EK0} ->
{Slot0, SK0, EK0}
end,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{Slot#slot_index_value.start_position,
Slot#slot_index_value.length,
SK,
EK}
end,
LengthList = lists:map(PointerMapFun, SlotList),
StartPos = element(1, lists:nth(1, LengthList)),
EndPos = element(1, lists:last(LengthList))
+ element(2, lists:last(LengthList)),
{ok, MultiSlotBin} = file:pread(Handle, StartPos, EndPos - StartPos),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
BinSplitMapFun =
fun({SP, L, SK, EK}) ->
Start = SP - StartPos,
<<_Pre:Start/binary,
SlotBin:L/binary,
_Post/binary>> = MultiSlotBin,
{SlotBin, SK, EK}
end,
lists:map(BinSplitMapFun, LengthList).
binaryslot_get(FullBin, Key, Hash) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
case crc_check_slot(FullBin) of
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockLengths, Rest} ->
<<B1P:32/integer, _R/binary>> = BlockLengths,
<<PosBinIndex:B1P/binary, Blocks/binary>> = Rest,
PosList = find_pos(PosBinIndex,
double_hash(Hash, Key),
[],
0),
{fetch_value(PosList, BlockLengths, Blocks, Key),
BlockLengths,
PosBinIndex};
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_wonky ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{not_present,
none,
none}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end.
binaryslot_tolist(FullBin) ->
BlockFetchFun =
fun(Length, {Acc, Bin}) ->
case Length of
0 ->
{Acc, Bin};
_ ->
<<Block:Length/binary, Rest/binary>> = Bin,
{Acc ++ binary_to_term(Block), Rest}
end
end,
{Out, _Rem} =
case crc_check_slot(FullBin) of
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockLengths, RestBin} ->
<<B1P:32/integer,
B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B4L:32/integer,
B5L:32/integer>> = BlockLengths,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<_PosBinIndex:B1P/binary, Blocks/binary>> = RestBin,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
lists:foldl(BlockFetchFun,
{[], Blocks},
[B1L, B2L, B3L, B4L, B5L]);
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_wonky ->
{[], <<>>}
end,
Out.
binaryslot_trimmedlist(FullBin, all, all) ->
binaryslot_tolist(FullBin);
binaryslot_trimmedlist(FullBin, StartKey, EndKey) ->
LTrimFun = fun({K, _V}) -> K < StartKey end,
RTrimFun = fun({K, _V}) -> not leveled_codec:endkey_passed(EndKey, K) end,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
% It will be more effecient to check a subset of blocks. To work out
% the best subset we always look in the middle block of 5, and based on
% the first and last keys of that middle block when compared to the Start
% and EndKey of the query determines a subset of blocks
%
% This isn't perfectly efficient, esepcially if the query overlaps Block2
% and Block3 (as Block 1 will also be checked), but finessing this last
% scenario is hard to do in concise code
BlocksToCheck =
case crc_check_slot(FullBin) of
{BlockLengths, RestBin} ->
<<B1P:32/integer,
B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
B4L:32/integer,
B5L:32/integer>> = BlockLengths,
<<_PosBinIndex:B1P/binary,
Block1:B1L/binary, Block2:B2L/binary,
MidBlock:B3L/binary,
Block4:B4L/binary, Block5:B5L/binary>> = RestBin,
case B3L of
0 ->
[Block1, Block2];
_ ->
MidBlockList = binary_to_term(MidBlock),
{MidFirst, _} = lists:nth(1, MidBlockList),
{MidLast, _} = lists:last(MidBlockList),
Split = {StartKey > MidLast,
StartKey >= MidFirst,
leveled_codec:endkey_passed(EndKey,
MidFirst),
leveled_codec:endkey_passed(EndKey,
MidLast)},
case Split of
{true, _, _, _} ->
[Block4, Block5];
{false, true, false, true} ->
[MidBlockList];
{false, true, false, false} ->
[MidBlockList, Block4, Block5];
{false, false, true, true} ->
[Block1, Block2];
{false, false, false, true} ->
[Block1, Block2, MidBlockList];
_ ->
[Block1, Block2, MidBlockList, Block4, Block5]
end
end;
crc_wonky ->
[]
end,
BlockCheckFun =
fun(Block, {Acc, Continue}) ->
case {Block, Continue} of
{<<>>, _} ->
{Acc, false};
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
{_, true} ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
BlockList =
case is_binary(Block) of
true ->
binary_to_term(Block);
false ->
Block
end,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{LastKey, _LV} = lists:last(BlockList),
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
case StartKey > LastKey of
true ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc, true};
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
false ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{_LDrop, RKeep} = lists:splitwith(LTrimFun,
BlockList),
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
case leveled_codec:endkey_passed(EndKey, LastKey) of
true ->
{LKeep, _RDrop} = lists:splitwith(RTrimFun, RKeep),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc ++ LKeep, false};
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
false ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc ++ RKeep, true}
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
end
end;
{_ , false} ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc, false}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end
end,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc, _Continue} = lists:foldl(BlockCheckFun, {[], true}, BlocksToCheck),
Acc.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_check_slot(FullBin) ->
<<CRC32:32/integer, SlotBin/binary>> = FullBin,
case erlang:crc32(SlotBin) of
CRC32 ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<BlockLengths:24/binary, Rest/binary>> = SlotBin,
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockLengths, Rest};
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
_ ->
Log improvements Log on bad CRC, and also not seeing SST timing logs, so log these more frequently
2017-01-02 18:54:19 +00:00
leveled_log:log("SST09", []),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_wonky
end.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
block_offsetandlength(BlockLengths, BlockID) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<BlocksPos:32/integer, BlockLengths0:20/binary>> = BlockLengths,
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
case BlockID of
1 ->
<<B1L:32/integer, _BR/binary>> = BlockLengths0,
{BlocksPos, 0, B1L};
2 ->
<<B1L:32/integer, B2L:32/integer, _BR/binary>> = BlockLengths0,
{BlocksPos, B1L, B2L};
3 ->
<<B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
_BR/binary>> = BlockLengths0,
{BlocksPos, B1L + B2L, B3L};
4 ->
<<B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B4L:32/integer,
_BR/binary>> = BlockLengths0,
{BlocksPos, B1L + B2L + B3L, B4L};
5 ->
<<B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
B4L:32/integer,
B5L:32/integer,
_BR/binary>> = BlockLengths0,
{BlocksPos, B1L + B2L + B3L + B4L, B5L}
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
end.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
double_hash(Hash, Key) ->
H2 = erlang:phash2(Key),
(Hash bxor H2) band 32767.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
fetch_value([], _BlockLengths, _Blocks, _Key) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
not_present;
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
fetch_value([Pos|Rest], BlockLengths, Blocks, Key) ->
{BlockNumber, BlockPos} = revert_position(Pos),
{_BlockPos,
Offset,
Length} = block_offsetandlength(BlockLengths, BlockNumber),
<<_Pre:Offset/binary, Block:Length/binary, _Rest/binary>> = Blocks,
BlockL = binary_to_term(Block),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{K, V} = lists:nth(BlockPos, BlockL),
case K of
Key ->
{K, V};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
_ ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
fetch_value(Rest, BlockLengths, Blocks, Key)
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
revert_position(Pos) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{SideBlockSize, MidBlockSize} = ?LOOK_BLOCKSIZE,
case Pos < 2 * SideBlockSize of
true ->
{(Pos div SideBlockSize) + 1, (Pos rem SideBlockSize) + 1};
false ->
case Pos < (2 * SideBlockSize + MidBlockSize) of
true ->
{3, ((Pos - 2 * SideBlockSize) rem MidBlockSize) + 1};
false ->
TailPos = Pos - 2 * SideBlockSize - MidBlockSize,
{(TailPos div SideBlockSize) + 4,
(TailPos rem SideBlockSize) + 1}
end
end.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
find_pos(<<>>, _Hash, PosList, _Count) ->
PosList;
find_pos(<<1:1/integer, Hash:15/integer, T/binary>>, Hash, PosList, Count) ->
find_pos(T, Hash, PosList ++ [Count], Count + 1);
find_pos(<<1:1/integer, _Miss:15/integer, T/binary>>, Hash, PosList, Count) ->
find_pos(T, Hash, PosList, Count + 1);
find_pos(<<0:1/integer, NHC:7/integer, T/binary>>, Hash, PosList, Count) ->
find_pos(T, Hash, PosList, Count + NHC + 1).
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
%%%============================================================================
%%% Merge Functions
%%%============================================================================
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
%% The source lists are merged into lists of slots before the file is created
%% At Level zero, there will be a single source list - and this will always be
%% split into standard size slots
%%
%% At lower levels there will be two source lists and they will need to be
%% merged to ensure that the best conflicting answer survives and compactable
%% KV pairs are discarded.
%%
%% At lower levels slots can be larger if there are no lookup keys present in
%% the slot. This is to slow the growth of the manifest/number-of-files when
%% large numbers of index keys are present - as well as improving compression
%% ratios in the Ledger.
%%
%% The outcome of merge_lists/1 and merge_lists/3 should be an list of slots.
%% Each slot should be ordered by Key and be of the form {Flag, KVList}, where
%% Flag can either be lookup or no-lookup. The list of slots should also be
%% ordered by Key (i.e. the first key in the slot)
%%
%% For merging ...
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
%% Compare the keys at the head of the list, and either skip that "best" key or
%% identify as the next key.
%%
%% The logic needs to change if the file is in the basement level, as keys with
%% expired timestamps need not be written at this level
%%
%% The best key is considered to be the lowest key in erlang term order. If
%% there are matching keys then the highest sequence number must be chosen and
%% any lower sequence numbers should be compacted out of existence
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
merge_lists(KVList1) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
SlotCount = length(KVList1) div ?LOOK_SLOTSIZE,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{[],
[],
split_lists(KVList1, [], SlotCount),
element(1, lists:nth(1, KVList1))}.
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
split_lists([], SlotLists, 0) ->
lists:reverse(SlotLists);
split_lists(LastPuff, SlotLists, 0) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
SlotD = generate_binary_slot(lookup, LastPuff),
lists:reverse([SlotD|SlotLists]);
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
split_lists(KVList1, SlotLists, N) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Slot, KVListRem} = lists:split(?LOOK_SLOTSIZE, KVList1),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
SlotD = generate_binary_slot(lookup, Slot),
split_lists(KVListRem, [SlotD|SlotLists], N - 1).
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
merge_lists(KVList1, KVList2, LevelInfo) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
merge_lists(KVList1, KVList2, LevelInfo, [], null, 0).
merge_lists(KVList1, KVList2, _LI, SlotList, FirstKey, ?MAX_SLOTS) ->
{KVList1, KVList2, lists:reverse(SlotList), FirstKey};
merge_lists([], [], _LI, SlotList, FirstKey, _SlotCount) ->
{[], [], lists:reverse(SlotList), FirstKey};
merge_lists(KVList1, KVList2, LI, SlotList, FirstKey, SlotCount) ->
{KVRem1, KVRem2, Slot, FK0} =
form_slot(KVList1, KVList2, LI, no_lookup, 0, [], FirstKey),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case Slot of
Fix empty slot issue
2017-03-11 12:41:30 +00:00
{_, []} ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
merge_lists(KVRem1,
KVRem2,
LI,
SlotList,
FK0,
SlotCount);
{Lookup, KVL} ->
SlotD = generate_binary_slot(Lookup, KVL),
merge_lists(KVRem1,
KVRem2,
LI,
[SlotD|SlotList],
FK0,
SlotCount + 1)
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end.
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
form_slot([], [], _LI, Type, _Size, Slot, FK) ->
{[], [], {Type, lists:reverse(Slot)}, FK};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
form_slot(KVList1, KVList2, _LI, lookup, ?LOOK_SLOTSIZE, Slot, FK) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{KVList1, KVList2, {lookup, lists:reverse(Slot)}, FK};
form_slot(KVList1, KVList2, _LI, no_lookup, ?NOLOOK_SLOTSIZE, Slot, FK) ->
{KVList1, KVList2, {no_lookup, lists:reverse(Slot)}, FK};
form_slot(KVList1, KVList2, {IsBasement, TS}, lookup, Size, Slot, FK) ->
case {key_dominates(KVList1, KVList2, {IsBasement, TS}), FK} of
{{{next_key, TopKV}, Rem1, Rem2}, _} ->
form_slot(Rem1,
Rem2,
{IsBasement, TS},
lookup,
Size + 1,
[TopKV|Slot],
FK);
{{skipped_key, Rem1, Rem2}, _} ->
form_slot(Rem1, Rem2, {IsBasement, TS}, lookup, Size, Slot, FK)
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end;
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
form_slot(KVList1, KVList2, {IsBasement, TS}, no_lookup, Size, Slot, FK) ->
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case key_dominates(KVList1, KVList2, {IsBasement, TS}) of
{{next_key, {TopK, TopV}}, Rem1, Rem2} ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
FK0 =
case FK of
null ->
TopK;
_ ->
FK
end,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case leveled_codec:to_lookup(TopK) of
no_lookup ->
form_slot(Rem1,
Rem2,
{IsBasement, TS},
no_lookup,
Size + 1,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
[{TopK, TopV}|Slot],
FK0);
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
lookup ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
case Size >= ?LOOK_SLOTSIZE of
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
true ->
{KVList1,
KVList2,
Fix bad exit with no FK
2017-03-14 00:52:07 +00:00
{no_lookup, lists:reverse(Slot)},
FK};
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
false ->
form_slot(Rem1,
Rem2,
{IsBasement, TS},
lookup,
Size + 1,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
[{TopK, TopV}|Slot],
FK0)
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end
end;
{skipped_key, Rem1, Rem2} ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
form_slot(Rem1, Rem2, {IsBasement, TS}, no_lookup, Size, Slot, FK)
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
end.
key_dominates(KL1, KL2, Level) ->
key_dominates_expanded(maybe_expand_pointer(KL1),
maybe_expand_pointer(KL2),
Level).
key_dominates_expanded([H1|T1], [], Level) ->
case leveled_codec:maybe_reap_expiredkey(H1, Level) of
true ->
{skipped_key, T1, []};
false ->
{{next_key, H1}, T1, []}
end;
key_dominates_expanded([], [H2|T2], Level) ->
case leveled_codec:maybe_reap_expiredkey(H2, Level) of
true ->
{skipped_key, [], T2};
false ->
{{next_key, H2}, [], T2}
end;
key_dominates_expanded([H1|T1], [H2|T2], Level) ->
case leveled_codec:key_dominates(H1, H2) of
left_hand_first ->
case leveled_codec:maybe_reap_expiredkey(H1, Level) of
true ->
{skipped_key, T1, [H2|T2]};
false ->
{{next_key, H1}, T1, [H2|T2]}
end;
right_hand_first ->
case leveled_codec:maybe_reap_expiredkey(H2, Level) of
true ->
{skipped_key, [H1|T1], T2};
false ->
{{next_key, H2}, [H1|T1], T2}
end;
left_hand_dominant ->
{skipped_key, [H1|T1], T2};
right_hand_dominant ->
{skipped_key, T1, [H2|T2]}
end.
%% When a list is provided it may include a pointer to gain another batch of
%% entries from the same file, or a new batch of entries from another file
%%
%% This resultant list should include the Tail of any pointers added at the
%% end of the list
maybe_expand_pointer([]) ->
[];
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
maybe_expand_pointer([{pointer, SSTPid, Slot, StartKey, all}|Tail]) ->
expand_list_by_pointer({pointer, SSTPid, Slot, StartKey, all},
Tail,
?MERGE_SCANWIDTH);
Manifest now back to a simple list This has refactored code with the implementation of the manifest isolated in to a seperate module, and the pure async relationship between penciller and their clerk. However, the manifest is just a simple list at each level.
2017-01-17 10:12:15 +00:00
maybe_expand_pointer([{next, ManEntry, StartKey}|Tail]) ->
expand_list_by_pointer({next, ManEntry, StartKey, all},
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Tail,
?MERGE_SCANWIDTH);
maybe_expand_pointer(List) ->
List.
Handle mismatch on expanding pointer Remove the nasty legacy of hard-coding for a scan width of 1
2016-12-29 10:46:12 +00:00
expand_list_by_pointer({pointer, SSTPid, Slot, StartKey, EndKey}, Tail, Width) ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
FoldFun =
fun(X, {Pointers, Remainder}) ->
case length(Pointers) of
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
L when L < Width ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
case X of
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{pointer, SSTPid, S, SK, EK} ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{Pointers ++ [{pointer, S, SK, EK}], Remainder};
_ ->
{Pointers, Remainder ++ [X]}
end;
_ ->
{Pointers, Remainder ++ [X]}
end
end,
Handle mismatch on expanding pointer Remove the nasty legacy of hard-coding for a scan width of 1
2016-12-29 10:46:12 +00:00
InitAcc = {[{pointer, Slot, StartKey, EndKey}], []},
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{AccPointers, AccTail} = lists:foldl(FoldFun, InitAcc, Tail),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
ExpPointers = leveled_sst:sst_getslots(SSTPid, AccPointers),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
lists:append(ExpPointers, AccTail);
Manifest now back to a simple list This has refactored code with the implementation of the manifest isolated in to a seperate module, and the pure async relationship between penciller and their clerk. However, the manifest is just a simple list at each level.
2017-01-17 10:12:15 +00:00
expand_list_by_pointer({next, ManEntry, StartKey, EndKey}, Tail, Width) ->
SSTPid = ManEntry#manifest_entry.owner,
leveled_log:log("SST10", [SSTPid, is_process_alive(SSTPid)]),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
ExpPointer = leveled_sst:sst_getkvrange(SSTPid, StartKey, EndKey, Width),
ExpPointer ++ Tail.
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Some end-to-end testing
2016-12-24 17:48:31 +00:00
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%%%============================================================================
%%% Test
%%%============================================================================
-ifdef(TEST).
generate_randomkeys(Seqn, Count, BucketRangeLow, BucketRangeHigh) ->
generate_randomkeys(Seqn,
Count,
[],
BucketRangeLow,
BucketRangeHigh).
generate_randomkeys(_Seqn, 0, Acc, _BucketLow, _BucketHigh) ->
Acc;
generate_randomkeys(Seqn, Count, Acc, BucketLow, BRange) ->
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
BRand = random:uniform(BRange),
BNumber = string:right(integer_to_list(BucketLow + BRand), 4, $0),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
KNumber = string:right(integer_to_list(random:uniform(1000)), 6, $0),
Extract Last Modified Date from Riak Object As part of process to supporting a recent changes index for near-real-time anti-entropy
2017-06-27 16:25:09 +01:00
LK = leveled_codec:to_ledgerkey("Bucket" ++ BNumber, "Key" ++ KNumber, o),
Chunk = crypto:rand_bytes(64),
{_B, _K, MV, _H, _LMs} =
leveled_codec:generate_ledgerkv(LK, Seqn, Chunk, 64, infinity),
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
generate_randomkeys(Seqn + 1,
Count - 1,
Extract Last Modified Date from Riak Object As part of process to supporting a recent changes index for near-real-time anti-entropy
2017-06-27 16:25:09 +01:00
[{LK, MV}|Acc],
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
BucketLow,
BRange).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
generate_indexkeys(Count) ->
generate_indexkeys(Count, []).
generate_indexkeys(0, IndexList) ->
IndexList;
generate_indexkeys(Count, IndexList) ->
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
Changes = generate_indexkey(random:uniform(8000), Count),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
generate_indexkeys(Count - 1, IndexList ++ Changes).
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
generate_indexkey(Term, Count) ->
IndexSpecs = [{add, "t1_int", Term}],
Add temporary aae index Pending ct tests. The aae index should expire after limit_minutes and be on an index which is rounded to unit_minutes.
2017-06-30 10:03:36 +01:00
leveled_codec:idx_indexspecs(IndexSpecs,
"Bucket",
"Key" ++ integer_to_list(Count),
Count,
infinity).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Unit tets of lookup over-size issue A mistake meant resetting to lookup on a skipped key would cause issues if the skipped key ocurred under a no_lookup slot after the ?SLOT_SIZE had been reached. This caused the slot to switch to lookup, but beyond the maximum size
2017-03-11 00:03:55 +00:00
form_slot_test() ->
% If a skip key happens, mustn't switch to loookup by accident as could be
% over the expected size
SkippingKV = {{o, "B1", "K9999", null}, {9999, tomb, 1234567, {}}},
Slot = [{{o, "B1", "K5", null}, {5, active, 99234567, {}}}],
R1 = form_slot([SkippingKV], [],
{true, 99999999},
no_lookup,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?LOOK_SLOTSIZE + 1,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
Slot,
{o, "B1", "K5", null}),
?assertMatch({[], [], {no_lookup, Slot}, {o, "B1", "K5", null}}, R1).
Unit tets of lookup over-size issue A mistake meant resetting to lookup on a skipped key would cause issues if the skipped key ocurred under a no_lookup slot after the ?SLOT_SIZE had been reached. This caused the slot to switch to lookup, but beyond the maximum size
2017-03-11 00:03:55 +00:00
Fix empty slot issue
2017-03-11 12:41:30 +00:00
merge_tombstonelist_test() ->
% Merge lists wiht nothing but tombstones
SkippingKV1 = {{o, "B1", "K9995", null}, {9995, tomb, 1234567, {}}},
SkippingKV2 = {{o, "B1", "K9996", null}, {9996, tomb, 1234567, {}}},
SkippingKV3 = {{o, "B1", "K9997", null}, {9997, tomb, 1234567, {}}},
SkippingKV4 = {{o, "B1", "K9998", null}, {9998, tomb, 1234567, {}}},
SkippingKV5 = {{o, "B1", "K9999", null}, {9999, tomb, 1234567, {}}},
R = merge_lists([SkippingKV1, SkippingKV3, SkippingKV5],
[SkippingKV2, SkippingKV4],
{true, 9999999}),
Fix empty file again No special definition of empty required, as now an empty list when empty
2017-03-14 00:17:09 +00:00
?assertMatch({[], [], [], null}, R).
Fix empty slot issue
2017-03-11 12:41:30 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_test() ->
io:format(user, "~nIndexed list timing test:~n", []),
Five alternatives explored
2016-12-29 22:22:13 +00:00
N = 150,
KVL0 = lists:ukeysort(1, generate_randomkeys(1, N, 1, 4)),
KVL1 = lists:sublist(KVL0, 128),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SW0 = os:timestamp(),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{_PosBinIndex1, FullBin, _HL, _LK} = generate_binary_slot(lookup, KVL1),
Experiment Want to experiemnt with different datatypes for the slot - maybe use a raw list but with a mini hashtree index like the CDB file
2016-12-29 14:11:05 +00:00
io:format(user,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
"Indexed list created slot in ~w microseconds of size ~w~n",
[timer:now_diff(os:timestamp(), SW0), byte_size(FullBin)]),
Five alternatives explored
2016-12-29 22:22:13 +00:00
{TestK1, TestV1} = lists:nth(20, KVL1),
MH1 = leveled_codec:magic_hash(TestK1),
{TestK2, TestV2} = lists:nth(40, KVL1),
MH2 = leveled_codec:magic_hash(TestK2),
{TestK3, TestV3} = lists:nth(60, KVL1),
MH3 = leveled_codec:magic_hash(TestK3),
{TestK4, TestV4} = lists:nth(80, KVL1),
MH4 = leveled_codec:magic_hash(TestK4),
{TestK5, TestV5} = lists:nth(100, KVL1),
MH5 = leveled_codec:magic_hash(TestK5),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin, TestK1, MH1, {TestK1, TestV1}),
test_binary_slot(FullBin, TestK2, MH2, {TestK2, TestV2}),
test_binary_slot(FullBin, TestK3, MH3, {TestK3, TestV3}),
test_binary_slot(FullBin, TestK4, MH4, {TestK4, TestV4}),
test_binary_slot(FullBin, TestK5, MH5, {TestK5, TestV5}).
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_mixedkeys_test() ->
KVL0 = lists:ukeysort(1, generate_randomkeys(1, 50, 1, 4)),
KVL1 = lists:sublist(KVL0, 33),
Keys = lists:ukeysort(1, generate_indexkeys(60) ++ KVL1),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{_PosBinIndex1, FullBin, _HL, _LK} = generate_binary_slot(lookup, Keys),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{TestK1, TestV1} = lists:nth(4, KVL1),
MH1 = leveled_codec:magic_hash(TestK1),
{TestK2, TestV2} = lists:nth(8, KVL1),
MH2 = leveled_codec:magic_hash(TestK2),
{TestK3, TestV3} = lists:nth(12, KVL1),
MH3 = leveled_codec:magic_hash(TestK3),
{TestK4, TestV4} = lists:nth(16, KVL1),
MH4 = leveled_codec:magic_hash(TestK4),
{TestK5, TestV5} = lists:nth(20, KVL1),
MH5 = leveled_codec:magic_hash(TestK5),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin, TestK1, MH1, {TestK1, TestV1}),
test_binary_slot(FullBin, TestK2, MH2, {TestK2, TestV2}),
test_binary_slot(FullBin, TestK3, MH3, {TestK3, TestV3}),
test_binary_slot(FullBin, TestK4, MH4, {TestK4, TestV4}),
test_binary_slot(FullBin, TestK5, MH5, {TestK5, TestV5}).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
indexed_list_mixedkeys2_test() ->
KVL0 = lists:ukeysort(1, generate_randomkeys(1, 50, 1, 4)),
KVL1 = lists:sublist(KVL0, 33),
IdxKeys1 = lists:ukeysort(1, generate_indexkeys(30)),
IdxKeys2 = lists:ukeysort(1, generate_indexkeys(30)),
% this isn't actually ordered correctly
Keys = IdxKeys1 ++ KVL1 ++ IdxKeys2,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{_PosBinIndex1, FullBin, _HL, _LK} = generate_binary_slot(lookup, Keys),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
lists:foreach(fun({K, V}) ->
MH = leveled_codec:magic_hash(K),
test_binary_slot(FullBin, K, MH, {K, V})
end,
KVL1).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_allindexkeys_test() ->
Keys = lists:sublist(lists:ukeysort(1, generate_indexkeys(150)), 128),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{PosBinIndex1, FullBin, _HL, _LK} = generate_binary_slot(lookup, Keys),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?assertMatch(<<_BL:24/binary, 127:8/integer>>, PosBinIndex1),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
% SW = os:timestamp(),
BinToList = binaryslot_tolist(FullBin),
Five alternatives explored
2016-12-29 22:22:13 +00:00
% io:format(user,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
% "Indexed list flattened in ~w microseconds ~n",
% [timer:now_diff(os:timestamp(), SW)]),
?assertMatch(Keys, BinToList),
?assertMatch(Keys, binaryslot_trimmedlist(FullBin, all, all)).
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
indexed_list_allindexkeys_nolookup_test() ->
Keys = lists:sublist(lists:ukeysort(1, generate_indexkeys(1000)),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?NOLOOK_SLOTSIZE),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{PosBinIndex1, FullBin, _HL, _LK} = generate_binary_slot(no_lookup, Keys),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?assertMatch(<<_BL:24/binary, 127:8/integer>>, PosBinIndex1),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
% SW = os:timestamp(),
BinToList = binaryslot_tolist(FullBin),
% io:format(user,
% "Indexed list flattened in ~w microseconds ~n",
% [timer:now_diff(os:timestamp(), SW)]),
?assertMatch(Keys, BinToList),
?assertMatch(Keys, binaryslot_trimmedlist(FullBin, all, all)).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_allindexkeys_trimmed_test() ->
Keys = lists:sublist(lists:ukeysort(1, generate_indexkeys(150)), 128),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{PosBinIndex1, FullBin, _HL, _LK} = generate_binary_slot(lookup, Keys),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?assertMatch(<<_BL:24/binary, 127:8/integer>>, PosBinIndex1),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(Keys, binaryslot_trimmedlist(FullBin,
{i,
"Bucket",
{"t1_int", 0},
null},
{i,
"Bucket",
{"t1_int", 99999},
null})),
Ongoing timing tests
2016-12-29 14:14:09 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{SK1, _} = lists:nth(10, Keys),
{EK1, _} = lists:nth(100, Keys),
R1 = lists:sublist(Keys, 10, 91),
O1 = binaryslot_trimmedlist(FullBin, SK1, EK1),
?assertMatch(91, length(O1)),
?assertMatch(R1, O1),
{SK2, _} = lists:nth(10, Keys),
{EK2, _} = lists:nth(20, Keys),
R2 = lists:sublist(Keys, 10, 11),
O2 = binaryslot_trimmedlist(FullBin, SK2, EK2),
?assertMatch(11, length(O2)),
?assertMatch(R2, O2),
{SK3, _} = lists:nth(127, Keys),
{EK3, _} = lists:nth(128, Keys),
R3 = lists:sublist(Keys, 127, 2),
O3 = binaryslot_trimmedlist(FullBin, SK3, EK3),
?assertMatch(2, length(O3)),
?assertMatch(R3, O3).
indexed_list_mixedkeys_bitflip_test() ->
KVL0 = lists:ukeysort(1, generate_randomkeys(1, 50, 1, 4)),
KVL1 = lists:sublist(KVL0, 33),
Keys = lists:ukeysort(1, generate_indexkeys(60) ++ KVL1),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{_PosBinIndex1, FullBin, _HL, LK} = generate_binary_slot(lookup, Keys),
?assertMatch(LK, element(1, lists:last(Keys))),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
L = byte_size(FullBin),
Byte1 = random:uniform(L),
<<PreB1:Byte1/binary, A:8/integer, PostByte1/binary>> = FullBin,
FullBin0 =
case A of
0 ->
<<PreB1:Byte1/binary, 255:8/integer, PostByte1/binary>>;
_ ->
<<PreB1:Byte1/binary, 0:8/integer, PostByte1/binary>>
end,
{TestK1, _TestV1} = lists:nth(20, KVL1),
MH1 = leveled_codec:magic_hash(TestK1),
Ongoing timing tests
2016-12-29 14:14:09 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin0, TestK1, MH1, not_present),
ToList = binaryslot_tolist(FullBin0),
?assertMatch([], ToList),
Ongoing timing tests
2016-12-29 14:14:09 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{SK1, _} = lists:nth(10, Keys),
{EK1, _} = lists:nth(50, Keys),
O1 = binaryslot_trimmedlist(FullBin0, SK1, EK1),
?assertMatch(0, length(O1)),
?assertMatch([], O1).
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin, Key, Hash, ExpectedValue) ->
% SW = os:timestamp(),
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{ReturnedValue, _BLs, _Idx} = binaryslot_get(FullBin, Key, Hash),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(ExpectedValue, ReturnedValue).
% io:format(user, "Fetch success in ~w microseconds ~n",
% [timer:now_diff(os:timestamp(), SW)]).
Five alternatives explored
2016-12-29 22:22:13 +00:00
Experiment Want to experiemnt with different datatypes for the slot - maybe use a raw list but with a mini hashtree index like the CDB file
2016-12-29 14:11:05 +00:00
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
merge_test() ->
N = 3000,
KVL1 = lists:ukeysort(1, generate_randomkeys(N + 1, N, 1, 20)),
KVL2 = lists:ukeysort(1, generate_randomkeys(1, N, 1, 20)),
KVL3 = lists:ukeymerge(1, KVL1, KVL2),
SW0 = os:timestamp(),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{ok, P1, {FK1, LK1}} = sst_new("../test/", "level1_src", 1, KVL1, 6000),
{ok, P2, {FK2, LK2}} = sst_new("../test/", "level2_src", 2, KVL2, 3000),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
ExpFK1 = element(1, lists:nth(1, KVL1)),
ExpLK1 = element(1, lists:last(KVL1)),
ExpFK2 = element(1, lists:nth(1, KVL2)),
ExpLK2 = element(1, lists:last(KVL2)),
?assertMatch(ExpFK1, FK1),
?assertMatch(ExpFK2, FK2),
?assertMatch(ExpLK1, LK1),
?assertMatch(ExpLK2, LK2),
Change to use manifest_entry not straight PID in unit test
2017-01-17 10:14:40 +00:00
ML1 = [{next, #manifest_entry{owner = P1}, FK1}],
ML2 = [{next, #manifest_entry{owner = P2}, FK2}],
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
NewR = sst_new("../test/", "level2_merge", ML1, ML2, false, 2, N * 2),
{ok, P3, {{Rem1, Rem2}, FK3, LK3}} = NewR,
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
?assertMatch([], Rem1),
?assertMatch([], Rem2),
?assertMatch(true, FK3 == min(FK1, FK2)),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
io:format("LK1 ~w LK2 ~w LK3 ~w~n", [LK1, LK2, LK3]),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
?assertMatch(true, LK3 == max(LK1, LK2)),
io:format(user,
"Created and merged two files of size ~w in ~w microseconds~n",
[N, timer:now_diff(os:timestamp(), SW0)]),
SW1 = os:timestamp(),
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(P3, K))
end,
KVL3),
io:format(user,
"Checked presence of all ~w objects in ~w microseconds~n",
[length(KVL3), timer:now_diff(os:timestamp(), SW1)]),
ok = sst_close(P1),
ok = sst_close(P2),
ok = sst_close(P3),
ok = file:delete("../test/level1_src.sst"),
ok = file:delete("../test/level2_src.sst"),
ok = file:delete("../test/level2_merge.sst").
Share final timing test
2016-12-23 23:30:15 +00:00
Build the table summary The table summary will be a skiplist, and this and the slot binary will be CRC checked
2016-12-24 01:23:40 +00:00
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
simple_persisted_range_test() ->
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{RP, Filename} = {"../test/", "simple_test"},
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList0 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 16, 1, 20),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
KVList1 = lists:ukeysort(1, KVList0),
[{FirstKey, _FV}|_Rest] = KVList1,
{LastKey, _LV} = lists:last(KVList1),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{ok, Pid, {FirstKey, LastKey}} = sst_new(RP,
Filename,
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
1,
KVList1,
length(KVList1)),
{o, B, K, null} = LastKey,
SK1 = {o, B, K, 0},
EK1 = {o, B, K, 1},
FetchListA1 = sst_getkvrange(Pid, SK1, EK1, 1),
?assertMatch([], FetchListA1),
SK2 = element(1, lists:nth(127, KVList1)),
SK3 = element(1, lists:nth(128, KVList1)),
SK4 = element(1, lists:nth(129, KVList1)),
SK5 = element(1, lists:nth(130, KVList1)),
EK2 = element(1, lists:nth(255, KVList1)),
EK3 = element(1, lists:nth(256, KVList1)),
EK4 = element(1, lists:nth(257, KVList1)),
EK5 = element(1, lists:nth(258, KVList1)),
TestFun =
fun({SK, EK}) ->
FetchList = sst_getkvrange(Pid, SK, EK, 4),
?assertMatch(SK, element(1, lists:nth(1, FetchList))),
?assertMatch(EK, element(1, lists:last(FetchList)))
end,
TL2 = lists:map(fun(EK) -> {SK2, EK} end, [EK2, EK3, EK4, EK5]),
TL3 = lists:map(fun(EK) -> {SK3, EK} end, [EK2, EK3, EK4, EK5]),
TL4 = lists:map(fun(EK) -> {SK4, EK} end, [EK2, EK3, EK4, EK5]),
TL5 = lists:map(fun(EK) -> {SK5, EK} end, [EK2, EK3, EK4, EK5]),
lists:foreach(TestFun, TL2 ++ TL3 ++ TL4 ++ TL5).
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
additional_range_test() ->
% Test fetching ranges that fall into odd situations with regards to the
% summayr index
% - ranges which fall between entries in summary
% - ranges which go beyond the end of the range of the sst
% - ranges which match to an end key in the summary index
IK1 = lists:foldl(fun(X, Acc) ->
Acc ++ generate_indexkey(X, X)
end,
[],
lists:seq(1, ?NOLOOK_SLOTSIZE)),
Gap = 2,
IK2 = lists:foldl(fun(X, Acc) ->
Acc ++ generate_indexkey(X, X)
end,
[],
lists:seq(?NOLOOK_SLOTSIZE + Gap + 1,
2 * ?NOLOOK_SLOTSIZE + Gap)),
{ok,
P1,
{{Rem1, Rem2},
SK,
EK}} = sst_new("../test/", "range1_src", IK1, IK2, false, 1, 9999),
?assertMatch([], Rem1),
?assertMatch([], Rem2),
?assertMatch(SK, element(1, lists:nth(1, IK1))),
?assertMatch(EK, element(1, lists:last(IK2))),
% Basic test - checking scanwidth
R1 = sst_getkvrange(P1, SK, EK, 1),
?assertMatch(?NOLOOK_SLOTSIZE + 1, length(R1)),
QR1 = lists:sublist(R1, ?NOLOOK_SLOTSIZE),
?assertMatch(IK1, QR1),
R2 = sst_getkvrange(P1, SK, EK, 2),
?assertMatch(?NOLOOK_SLOTSIZE * 2, length(R2)),
QR2 = lists:sublist(R2, ?NOLOOK_SLOTSIZE),
QR3 = lists:sublist(R2, ?NOLOOK_SLOTSIZE + 1, 2 * ?NOLOOK_SLOTSIZE),
?assertMatch(IK1, QR2),
?assertMatch(IK2, QR3),
% Testing the gap
[GapSKV] = generate_indexkey(?NOLOOK_SLOTSIZE + 1, ?NOLOOK_SLOTSIZE + 1),
[GapEKV] = generate_indexkey(?NOLOOK_SLOTSIZE + 2, ?NOLOOK_SLOTSIZE + 2),
R3 = sst_getkvrange(P1, element(1, GapSKV), element(1, GapEKV), 1),
?assertMatch([], R3),
% Testing beyond the range
[PastEKV] = generate_indexkey(2 * ?NOLOOK_SLOTSIZE + Gap + 1,
2 * ?NOLOOK_SLOTSIZE + Gap + 1),
R4 = sst_getkvrange(P1, element(1, GapSKV), element(1, PastEKV), 2),
?assertMatch(IK2, R4),
Add unit test for more than one slot
2017-03-15 16:40:43 +00:00
R5 = sst_getkvrange(P1, SK, element(1, PastEKV), 2),
IKAll = IK1 ++ IK2,
?assertMatch(IKAll, R5),
RTrim RTrim only worked in special case of key matching, that would never occur in real world range query. RTrim should really check for key passing. Returning empty list should not be possible - unless the query is outside of the range entirely (and such a query should never go to this SST).
2017-03-16 08:37:36 +00:00
[MidREKV] = generate_indexkey(?NOLOOK_SLOTSIZE + Gap + 2,
?NOLOOK_SLOTSIZE + Gap + 2),
io:format(user, "Mid second range to past range test~n", []),
R6 = sst_getkvrange(P1, element(1, MidREKV), element(1, PastEKV), 2),
Exp6 = lists:sublist(IK2, 2, length(IK2)),
?assertMatch(Exp6, R6),
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
% Testing at a slot end
Slot1EK = element(1, lists:last(IK1)),
RTrim RTrim only worked in special case of key matching, that would never occur in real world range query. RTrim should really check for key passing. Returning empty list should not be possible - unless the query is outside of the range entirely (and such a query should never go to this SST).
2017-03-16 08:37:36 +00:00
R7 = sst_getkvrange(P1, SK, Slot1EK, 2),
?assertMatch(IK1, R7).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
Check query out of range It doesn't work - so protecting against it in fetch_range is pointless, will blow up in lookup_slots
2017-03-16 08:43:18 +00:00
% Testing beyond end (should never happen if manifest behaves)
% Test blows up anyway
% R8 = sst_getkvrange(P1, element(1, PastEKV), element(1, PastEKV), 2),
% ?assertMatch([], R8).
Additional unit test Need to test scenario where the key list the SST file created from is an exact multiple of the slot size
2017-03-19 23:42:24 +00:00
simple_persisted_slotsize_test() ->
{RP, Filename} = {"../test/", "simple_slotsize_test"},
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList0 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 2, 1, 20),
KVList1 = lists:sublist(lists:ukeysort(1, KVList0),
?LOOK_SLOTSIZE),
Additional unit test Need to test scenario where the key list the SST file created from is an exact multiple of the slot size
2017-03-19 23:42:24 +00:00
[{FirstKey, _FV}|_Rest] = KVList1,
{LastKey, _LV} = lists:last(KVList1),
{ok, Pid, {FirstKey, LastKey}} = sst_new(RP,
Filename,
1,
KVList1,
length(KVList1)),
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(Pid, K))
end,
KVList1),
ok = sst_close(Pid),
ok = file:delete(filename:join(RP, Filename ++ ".sst")).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
Some end-to-end testing
2016-12-24 17:48:31 +00:00
simple_persisted_test() ->
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{RP, Filename} = {"../test/", "simple_test"},
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList0 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 32, 1, 20),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
KVList1 = lists:ukeysort(1, KVList0),
[{FirstKey, _FV}|_Rest] = KVList1,
{LastKey, _LV} = lists:last(KVList1),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{ok, Pid, {FirstKey, LastKey}} = sst_new(RP,
Filename,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
1,
KVList1,
length(KVList1)),
Fix broken SST cache
2017-01-03 13:03:59 +00:00
SW0 = os:timestamp(),
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(Pid, K))
end,
KVList1),
io:format(user,
"Checking for ~w keys (once) in file with cache hit took ~w "
++ "microseconds~n",
[length(KVList1), timer:now_diff(os:timestamp(), SW0)]),
Test for missing keys
2016-12-24 17:59:07 +00:00
SW1 = os:timestamp(),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(Pid, K)),
?assertMatch({K, V}, sst_get(Pid, K))
end,
KVList1),
io:format(user,
"Checking for ~w keys (twice) in file with cache hit took ~w "
++ "microseconds~n",
Test for missing keys
2016-12-24 17:59:07 +00:00
[length(KVList1), timer:now_diff(os:timestamp(), SW1)]),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
ok = sst_printtimings(Pid),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList2 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 32, 1, 20),
Test for missing keys
2016-12-24 17:59:07 +00:00
MapFun =
fun({K, V}, Acc) ->
In = lists:keymember(K, 1, KVList1),
case {K > FirstKey, LastKey > K, In} of
{true, true, false} ->
Add hash to missing key test
2016-12-24 18:03:34 +00:00
[{K, leveled_codec:magic_hash(K), V}|Acc];
Test for missing keys
2016-12-24 17:59:07 +00:00
_ ->
Acc
end
end,
KVList3 = lists:foldl(MapFun, [], KVList2),
SW2 = os:timestamp(),
Add hash to missing key test
2016-12-24 18:03:34 +00:00
lists:foreach(fun({K, H, _V}) ->
?assertMatch(not_present, sst_get(Pid, K, H))
Test for missing keys
2016-12-24 17:59:07 +00:00
end,
KVList3),
io:format(user,
"Checking for ~w missing keys took ~w microseconds~n",
[length(KVList3), timer:now_diff(os:timestamp(), SW2)]),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
ok = sst_printtimings(Pid),
FetchList1 = sst_getkvrange(Pid, all, all, 2),
FoldFun = fun(X, Acc) ->
case X of
{pointer, P, S, SK, EK} ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Acc ++ sst_getslots(P, [{pointer, P, S, SK, EK}]);
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
_ ->
Acc ++ [X]
end end,
FetchedList1 = lists:foldl(FoldFun, [], FetchList1),
?assertMatch(KVList1, FetchedList1),
{TenthKey, _v10} = lists:nth(10, KVList1),
{Three000Key, _v300} = lists:nth(300, KVList1),
SubKVList1 = lists:sublist(KVList1, 10, 291),
SubKVList1L = length(SubKVList1),
FetchList2 = sst_getkvrange(Pid, TenthKey, Three000Key, 2),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
?assertMatch(pointer, element(1, lists:last(FetchList2))),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
FetchedList2 = lists:foldl(FoldFun, [], FetchList2),
?assertMatch(SubKVList1L, length(FetchedList2)),
?assertMatch(SubKVList1, FetchedList2),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{Eight000Key, _v800} = lists:nth(800, KVList1),
SubKVListA1 = lists:sublist(KVList1, 10, 791),
SubKVListA1L = length(SubKVListA1),
FetchListA2 = sst_getkvrange(Pid, TenthKey, Eight000Key, 2),
?assertMatch(pointer, element(1, lists:last(FetchListA2))),
FetchedListA2 = lists:foldl(FoldFun, [], FetchListA2),
?assertMatch(SubKVListA1L, length(FetchedListA2)),
?assertMatch(SubKVListA1, FetchedListA2),
FetchListB2 = sst_getkvrange(Pid, TenthKey, Eight000Key, 4),
?assertMatch(pointer, element(1, lists:last(FetchListB2))),
FetchedListB2 = lists:foldl(FoldFun, [], FetchListB2),
?assertMatch(SubKVListA1L, length(FetchedListB2)),
?assertMatch(SubKVListA1, FetchedListB2),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
FetchListB3 = sst_getkvrange(Pid,
Eight000Key,
{o, null, null, null},
4),
FetchedListB3 = lists:foldl(FoldFun, [], FetchListB3),
SubKVListA3 = lists:nthtail(800 - 1, KVList1),
SubKVListA3L = length(SubKVListA3),
io:format("Length expected ~w~n", [SubKVListA3L]),
?assertMatch(SubKVListA3L, length(FetchedListB3)),
?assertMatch(SubKVListA3, FetchedListB3),
Fixed issues Two issues - when the key range falls in-between two marks in the summary, we didn't pick up any mark. then when trimming both right and left, the left trim was being discarded.
2016-12-29 04:37:49 +00:00
io:format("Eight hundredth key ~w~n", [Eight000Key]),
FetchListB4 = sst_getkvrange(Pid,
Eight000Key,
Eight000Key,
4),
FetchedListB4 = lists:foldl(FoldFun, [], FetchListB4),
?assertMatch([{Eight000Key, _v800}], FetchedListB4),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
ok = sst_close(Pid),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:delete(filename:join(RP, Filename ++ ".sst")).
Some end-to-end testing
2016-12-24 17:48:31 +00:00
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
key_dominates_test() ->
KV1 = {{o, "Bucket", "Key1", null}, {5, {active, infinity}, 0, []}},
KV2 = {{o, "Bucket", "Key3", null}, {6, {active, infinity}, 0, []}},
KV3 = {{o, "Bucket", "Key2", null}, {3, {active, infinity}, 0, []}},
KV4 = {{o, "Bucket", "Key4", null}, {7, {active, infinity}, 0, []}},
KV5 = {{o, "Bucket", "Key1", null}, {4, {active, infinity}, 0, []}},
KV6 = {{o, "Bucket", "Key1", null}, {99, {tomb, 999}, 0, []}},
KV7 = {{o, "Bucket", "Key1", null}, {99, tomb, 0, []}},
KL1 = [KV1, KV2],
KL2 = [KV3, KV4],
?assertMatch({{next_key, KV1}, [KV2], KL2},
key_dominates(KL1, KL2, {undefined, 1})),
?assertMatch({{next_key, KV1}, KL2, [KV2]},
key_dominates(KL2, KL1, {undefined, 1})),
?assertMatch({skipped_key, KL2, KL1},
key_dominates([KV5|KL2], KL1, {undefined, 1})),
?assertMatch({{next_key, KV1}, [KV2], []},
key_dominates(KL1, [], {undefined, 1})),
?assertMatch({skipped_key, [KV6|KL2], [KV2]},
key_dominates([KV6|KL2], KL1, {undefined, 1})),
?assertMatch({{next_key, KV6}, KL2, [KV2]},
key_dominates([KV6|KL2], [KV2], {undefined, 1})),
?assertMatch({skipped_key, [KV6|KL2], [KV2]},
key_dominates([KV6|KL2], KL1, {true, 1})),
?assertMatch({skipped_key, [KV6|KL2], [KV2]},
key_dominates([KV6|KL2], KL1, {true, 1000})),
?assertMatch({{next_key, KV6}, KL2, [KV2]},
key_dominates([KV6|KL2], [KV2], {true, 1})),
?assertMatch({skipped_key, KL2, [KV2]},
key_dominates([KV6|KL2], [KV2], {true, 1000})),
?assertMatch({skipped_key, [], []},
key_dominates([KV6], [], {true, 1000})),
?assertMatch({skipped_key, [], []},
key_dominates([], [KV6], {true, 1000})),
?assertMatch({{next_key, KV6}, [], []},
key_dominates([KV6], [], {true, 1})),
?assertMatch({{next_key, KV6}, [], []},
key_dominates([], [KV6], {true, 1})),
?assertMatch({skipped_key, [], []},
key_dominates([KV7], [], {true, 1})),
?assertMatch({skipped_key, [], []},
key_dominates([], [KV7], {true, 1})),
?assertMatch({skipped_key, [KV7|KL2], [KV2]},
key_dominates([KV7|KL2], KL1, {undefined, 1})),
?assertMatch({{next_key, KV7}, KL2, [KV2]},
key_dominates([KV7|KL2], [KV2], {undefined, 1})),
?assertMatch({skipped_key, [KV7|KL2], [KV2]},
key_dominates([KV7|KL2], KL1, {true, 1})),
?assertMatch({skipped_key, KL2, [KV2]},
key_dominates([KV7|KL2], [KV2], {true, 1})).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
nonsense_coverage_test() ->
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
ok = gen_fsm:send_all_state_event(Pid, nonsense),
?assertMatch({next_state, reader, #state{}}, handle_info(nonsense,
reader,
#state{})),
?assertMatch({ok, reader, #state{}}, code_change(nonsense,
reader,
#state{},
Coverage checks
2017-02-26 21:37:47 +00:00
nonsense)),
?assertMatch({reply, undefined, reader, #state{}},
handle_sync_event("hello", self(), reader, #state{})).
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Cleanup dialyzer errrors in leveled_sst
2017-07-31 19:30:29 +02:00
-endif.
Reference in a new issue Copy permalink