umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
leveled/src/leveled_sst.erl

2538 lines
100 KiB
Erlang
Raw Normal View History

Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%% -------- SST (Variant) ---------
%%
%% A FSM module intended to wrap a persisted, ordered view of Keys and Values
%%
%% The persisted view is built from a list (which may be created by merging
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%% multiple lists). The list is built first, then the view is created in bulk.
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%%
%% -------- Slots ---------
%%
%% The view is built from sublists referred to as slot. Each slot is up to 128
Comment update
2017-01-25 12:38:33 +00:00
%% keys and values in size. Three strategies have been benchmarked for the
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% slot: a skiplist, a gb-tree, four blocks of flat lists with an index.
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%%
%% Skiplist:
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% build and serialise slot - 3233 microseconds
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%% de-serialise and check * 128 - 14669 microseconds
%% flatten back to list - 164 microseconds
%%
%% GBTree:
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% build and serialise tree - 1433 microseconds
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
%% de-serialise and check * 128 - 15263 microseconds
%% flatten back to list - 175 microseconds
%%
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% Indexed Blocks:
%% build and serialise slot 342 microseconds
Comment update
2017-01-25 12:38:33 +00:00
%% de-deserialise and check * 128 - 6746 microseconds
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% flatten back to list - 187 microseconds
%%
%% The negative side of using Indexed Blocks is the storage of the index. In
%% the original implementation this was stored on fadvised disk (the index in
%% this case was a rice-encoded view of which block the object is in). In this
%% implementation it is cached in memory -requiring 2-bytes per key to be kept
%% in memory.
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%%
%% -------- Blooms ---------
%%
Comment update
2017-01-25 12:38:33 +00:00
%% There is a bloom for each slot - based on two hashes and 8 bits per key.
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%%
Comment update
2017-01-25 12:38:33 +00:00
%% Hashing for blooms is a challenge, as the slot is a slice of an ordered
%% list of keys with a fixed format. It is likely that the keys may vary by
%% only one or two ascii characters, and there is a desire to avoid the
%% overhead of cryptographic hash functions that may be able to handle this.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%%
%% -------- Summary ---------
%%
%% Each file has a summary - which is the 128 keys at the top of each slot in
%% a skiplist, with some basic metadata about the slot stored as the value.
%%
Comment update
2017-01-25 12:38:33 +00:00
%% The summary is stored seperately to the slots (within the same file).
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
%%
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%% -------- CRC Checks ---------
%%
%% Every attempt to either read a summary or a slot off disk will also include
%% a CRC check. If the CRC check fails non-presence is assumed (the data
%% within is assumed to be entirely lost). The data can be recovered by either
%% using a recoverable strategy in transaction log compaction, and triggering
%% the transaction log replay; or by using a higher level for of anti-entropy
%% (i.e. make Riak responsible).
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-module(leveled_sst).
Some basic code layout work
2016-12-24 15:12:24 +00:00
-behaviour(gen_fsm).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-include("include/leveled.hrl").
Revert to previous standard slot size But maintain configurability of slot size to maximum
2017-10-25 08:59:34 +01:00
-define(MAX_SLOTS, 256).
-define(LOOK_SLOTSIZE, 128). % Maximum of 128
-define(LOOK_BLOCKSIZE, {24, 32}). % 4x + y = ?LOOK_SLOTSIZE
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
-define(NOLOOK_SLOTSIZE, 256).
Revert to previous standard slot size But maintain configurability of slot size to maximum
2017-10-25 08:59:34 +01:00
-define(NOLOOK_BLOCKSIZE, {56, 32}). % 4x + y = ?NOLOOK_SLOTSIZE
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-define(COMPRESSION_LEVEL, 1).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
-define(BINARY_SETTINGS, [{compressed, ?COMPRESSION_LEVEL}]).
Return to merge scan width of 16 This was reduced before the use of binary blocks was committed
2017-03-21 17:53:34 +00:00
-define(MERGE_SCANWIDTH, 16).
Some end-to-end testing
2016-12-24 17:48:31 +00:00
-define(DISCARD_EXT, ".discarded").
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
-define(DELETE_TIMEOUT, 10000).
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
-define(TREE_TYPE, idxt).
-define(TREE_SIZE, 4).
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
-define(TIMING_SAMPLECOUNTDOWN, 10000).
-define(TIMING_SAMPLESIZE, 100).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
-include_lib("eunit/include/eunit.hrl").
Some basic code layout work
2016-12-24 15:12:24 +00:00
-export([init/1,
handle_sync_event/4,
handle_event/3,
handle_info/3,
terminate/3,
code_change/4,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
starting/2,
Some basic code layout work
2016-12-24 15:12:24 +00:00
starting/3,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
reader/3,
delete_pending/2,
delete_pending/3]).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
-export([sst_new/6,
sst_new/8,
sst_newlevelzero/7,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
sst_open/2,
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_get/2,
sst_get/3,
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
sst_getkvrange/4,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
sst_getfilteredrange/5,
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
sst_getslots/2,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
sst_getfilteredslots/3,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_getmaxsequencenumber/1,
sst_setfordelete/2,
sst_clear/1,
sst_checkready/1,
sst_deleteconfirmed/1,
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_close/1]).
Remove get_slotid Had been used in some debug logging - now not called
2017-11-01 17:05:35 +00:00
-export([expand_list_by_pointer/4]).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Confirm timings
2016-12-23 18:08:22 +00:00
-record(slot_index_value, {slot_id :: integer(),
start_position :: integer(),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
length :: integer(),
bloom :: binary()}).
Confirm timings
2016-12-23 18:08:22 +00:00
Some basic code layout work
2016-12-24 15:12:24 +00:00
-record(summary, {first_key :: tuple(),
last_key :: tuple(),
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
index :: tuple() | undefined,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
size :: integer(),
Remove SST blooms
2017-01-03 09:12:41 +00:00
max_sqn :: integer()}).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
-type press_methods() :: lz4|native.
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
%% yield_blockquery is used to detemrine if the work necessary to process a
%% range query beyond the fetching the slot should be managed from within
%% this process, or should be handled by the calling process.
%% Handling within the calling process may lead to extra binary heap garbage
%% see Issue 52. Handling within the SST process may lead to contention and
%% extra copying. Files at the top of the tree yield, those lower down don't.
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
-record(state,
{summary,
handle :: file:fd() | undefined,
penciller :: pid() | undefined,
root_path,
filename,
yield_blockquery = false :: boolean(),
blockindex_cache,
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
compression_method = native :: press_methods(),
timings = no_timing :: sst_timings(),
timings_countdown = 0 :: integer()}).
-record(sst_timings,
{sample_count = 0 :: integer(),
index_query_time = 0 :: integer(),
tiny_bloom_time = 0 :: integer(),
slot_index_time = 0 :: integer(),
slot_fetch_time = 0 :: integer(),
noncached_block_time = 0 :: integer(),
tiny_bloom_count = 0 :: integer(),
slot_index_count = 0 :: integer(),
slot_fetch_count = 0 :: integer(),
noncached_block_count = 0 :: integer()}).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
-type sst_state() :: #state{}.
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
-type sst_timings() :: no_timing|#sst_timings{}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%%%============================================================================
%%% API
%%%============================================================================
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_open(string(), string()) -> {ok, pid(), {tuple(), tuple()}}.
%% @doc
%% Open an SST file at a given path and filename. The first and last keys
%% are returned in response to the request - so that those keys can be used
%% in manifests to understand what range of keys are covered by the SST file.
%% All keys in the file should be between the first and last key in erlang
%% term order.
%%
%% The filename should include the file extension.
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
sst_open(RootPath, Filename) ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
case gen_fsm:sync_send_event(Pid,
{sst_open, RootPath, Filename},
infinity) of
Some basic code layout work
2016-12-24 15:12:24 +00:00
{ok, {SK, EK}} ->
{ok, Pid, {SK, EK}}
end.
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
-spec sst_new(string(), string(), integer(),
list(), integer(), press_methods()) ->
spec help for SST file
2017-05-18 12:29:56 +01:00
{ok, pid(), {tuple(), tuple()}}.
%% @doc
%% Start a new SST file at the assigned level passing in a list of Key, Value
%% pairs. This should not be used for basement levels or unexpanded Key/Value
%% lists as merge_lists will not be called.
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
sst_new(RootPath, Filename, Level, KVList, MaxSQN, PressMethod) ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{[], [], SlotList, FK} = merge_lists(KVList, PressMethod),
Some basic code layout work
2016-12-24 15:12:24 +00:00
case gen_fsm:sync_send_event(Pid,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{sst_new,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
RootPath,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Filename,
Level,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{SlotList, FK},
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
MaxSQN,
PressMethod},
Some basic code layout work
2016-12-24 15:12:24 +00:00
infinity) of
{ok, {SK, EK}} ->
{ok, Pid, {SK, EK}}
end.
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_new(string(), string(), list(), list(),
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
boolean(), integer(), integer(), press_methods()) ->
spec help for SST file
2017-05-18 12:29:56 +01:00
empty|{ok, pid(), {{list(), list()}, tuple(), tuple()}}.
%% @doc
%% Start a new SST file at the assigned level passing in a two lists of
%% {Key, Value} pairs to be merged. The merge_lists function will use the
%% IsBasement boolean to determine if expired keys or tombstones can be
%% deleted.
%%
%% The remainder of the lists is returned along with the StartKey and EndKey
%% so that the remainder cna be used in the next file in the merge. It might
%% be that the merge_lists returns nothin (for example when a basement file is
%% all tombstones) - and the atome empty is returned in this case so that the
%% file is not added to the manifest.
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
sst_new(RootPath, Filename,
KVL1, KVL2, IsBasement, Level,
MaxSQN, PressMethod) ->
{Rem1, Rem2, SlotList, FK} =
merge_lists(KVL1, KVL2, {IsBasement, Level}, PressMethod),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case SlotList of
Fix empty file again No special definition of empty required, as now an empty list when empty
2017-03-14 00:17:09 +00:00
[] ->
Handle production of empty file
2016-12-29 05:09:47 +00:00
empty;
_ ->
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
case gen_fsm:sync_send_event(Pid,
{sst_new,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
RootPath,
Handle production of empty file
2016-12-29 05:09:47 +00:00
Filename,
Level,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{SlotList, FK},
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
MaxSQN,
PressMethod},
Handle production of empty file
2016-12-29 05:09:47 +00:00
infinity) of
{ok, {SK, EK}} ->
{ok, Pid, {{Rem1, Rem2}, SK, EK}}
end
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
end.
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_newlevelzero(string(), string(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
integer(), fun(), pid()|undefined, integer(),
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
press_methods()) ->
spec help for SST file
2017-05-18 12:29:56 +01:00
{ok, pid(), noreply}.
%% @doc
%% Start a new file at level zero. At this level the file size is not fixed -
%% it will be as big as the input. Also the KVList is not passed in, it is
%% fetched slot by slot using the FetchFun
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
sst_newlevelzero(RootPath, Filename,
Slots, FetchFun, Penciller,
MaxSQN, PressMethod) ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
gen_fsm:send_event(Pid,
{sst_newlevelzero,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
RootPath,
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Filename,
Slots,
FetchFun,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Penciller,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
MaxSQN,
PressMethod}),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{ok, Pid, noreply}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_get(pid(), tuple()) -> tuple()|not_present.
%% @doc
%% Return a Key, Value pair matching a Key or not_present if the Key is not in
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
%% the store. The segment_hash function is used to accelerate the seeking of
spec help for SST file
2017-05-18 12:29:56 +01:00
%% keys, sst_get/3 should be used directly if this has already been calculated
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_get(Pid, LedgerKey) ->
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
sst_get(Pid, LedgerKey, leveled_codec:segment_hash(LedgerKey)).
Some basic code layout work
2016-12-24 15:12:24 +00:00
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
-spec sst_get(pid(), tuple(), {integer(), integer()}) -> tuple()|not_present.
spec help for SST file
2017-05-18 12:29:56 +01:00
%% @doc
%% Return a Key, Value pair matching a Key or not_present if the Key is not in
%% the store (with the magic hash precalculated).
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_get(Pid, LedgerKey, Hash) ->
gen_fsm:sync_send_event(Pid, {get_kv, LedgerKey, Hash}, infinity).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_getkvrange(pid(), tuple()|all, tuple()|all, integer()) -> list().
%% @doc
%% Get a range of {Key, Value} pairs as a list between StartKey and EndKey
%% (inclusive). The ScanWidth is the maximum size of the range, a pointer
%% will be placed on the tail of the resulting list if results expand beyond
%% the Scan Width
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
sst_getkvrange(Pid, StartKey, EndKey, ScanWidth) ->
sst_getfilteredrange(Pid, StartKey, EndKey, ScanWidth, false).
-spec sst_getfilteredrange(pid(), tuple()|all, tuple()|all, integer(),
list()|false) -> list().
%% @doc
%% Get a range of {Key, Value} pairs as a list between StartKey and EndKey
%% (inclusive). The ScanWidth is the maximum size of the range, a pointer
%% will be placed on the tail of the resulting list if results expand beyond
%% the Scan Width
spec help for SST file
2017-05-18 12:29:56 +01:00
%%
%% To make the range open-ended (either ta start, end or both) the all atom
%% can be use din place of the Key tuple.
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
sst_getfilteredrange(Pid, StartKey, EndKey, ScanWidth, SegList) ->
Add tests for different tree sizes Note that accelerating segment_list queries will not work for tree sizes smaller than small. How to flag this up? Should smaller tree sizes just be removed from leveled_tictac?
2017-11-01 11:51:51 +00:00
SegList0 = tune_seglist(SegList),
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
case gen_fsm:sync_send_event(Pid,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
{get_kvrange,
StartKey, EndKey,
ScanWidth, SegList0},
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
infinity) of
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{yield, SlotsToFetchBinList, SlotsToPoint, PressMethod} ->
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
binaryslot_reader(SlotsToFetchBinList, PressMethod)
++ SlotsToPoint;
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
Reply ->
Reply
end.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_getslots(pid(), list()) -> list().
%% @doc
%% Get a list of slots by their ID. The slot will be converted from the binary
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
%% to term form outside of the FSM loop, this is to stop the copying of the
%% converted term to the calling process.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
sst_getslots(Pid, SlotList) ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
sst_getfilteredslots(Pid, SlotList, false).
-spec sst_getfilteredslots(pid(), list(), false|list()) -> list().
%% @doc
%% Get a list of slots by their ID. The slot will be converted from the binary
%% to term form outside of the FSM loop
%%
%% A list of 16-bit integer Segment IDs can be passed to filter the keys
%% returned (not precisely - with false results returned in addition). Use
%% false as a SegList to not filter
sst_getfilteredslots(Pid, SlotList, SegList) ->
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
SegL0 = tune_seglist(SegList),
{SlotBins, PressMethod} =
gen_fsm:sync_send_event(Pid, {get_slots, SlotList, SegL0}, infinity),
binaryslot_reader(SlotBins, PressMethod).
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_getmaxsequencenumber(pid()) -> integer().
%% @doc
%% Get the maximume sequence number for this SST file
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_getmaxsequencenumber(Pid) ->
gen_fsm:sync_send_event(Pid, get_maxsequencenumber, infinity).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_setfordelete(pid(), pid()|false) -> ok.
%% @doc
%% If the SST is no longer in use in the active ledger it can be set for
%% delete. Once set for delete it will poll the Penciller pid to see if
%% it is yet safe to be deleted (i.e. because all snapshots which depend
%% on it have finished). No polling will be done if the Penciller pid
%% is 'false'
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_setfordelete(Pid, Penciller) ->
gen_fsm:sync_send_event(Pid, {set_for_delete, Penciller}, infinity).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_clear(pid()) -> ok.
%% @doc
%% For this file to be closed and deleted
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_clear(Pid) ->
gen_fsm:sync_send_event(Pid, {set_for_delete, false}, infinity),
gen_fsm:sync_send_event(Pid, close, 1000).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_deleteconfirmed(pid()) -> ok.
%% @doc
%% Allows a penciller to confirm to a SST file that it can be cleared, as it
%% is no longer in use
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_deleteconfirmed(Pid) ->
gen_fsm:send_event(Pid, close).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_checkready(pid()) -> {ok, string(), tuple(), tuple()}.
%% @doc
%% If a file has been set to be built, check that it has been built. Returns
%% the filename and the {startKey, EndKey} for the manifest.
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
sst_checkready(Pid) ->
%% Only used in test
gen_fsm:sync_send_event(Pid, background_complete, 100).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_close(pid()) -> ok.
%% @doc
%% Close the file
Some basic code layout work
2016-12-24 15:12:24 +00:00
sst_close(Pid) ->
gen_fsm:sync_send_event(Pid, close, 2000).
spec help for SST file
2017-05-18 12:29:56 +01:00
-spec sst_printtimings(pid()) -> ok.
%% @doc
%% The state of the FSM keeps track of timings of operations, and this can
%% forced to be printed.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
%% Used in unit tests to force the printing of timings
sst_printtimings(Pid) ->
gen_fsm:sync_send_event(Pid, print_timings, 1000).
Some basic code layout work
2016-12-24 15:12:24 +00:00
%%%============================================================================
%%% gen_server callbacks
%%%============================================================================
init([]) ->
{ok, starting, #state{}}.
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
starting({sst_open, RootPath, Filename}, _From, State) ->
UpdState = read_file(Filename, State#state{root_path=RootPath}),
Some basic code layout work
2016-12-24 15:12:24 +00:00
Summary = UpdState#state.summary,
{reply,
{ok, {Summary#summary.first_key, Summary#summary.last_key}},
reader,
UpdState};
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
starting({sst_new,
RootPath, Filename, Level,
{SlotList, FirstKey}, MaxSQN,
PressMethod}, _From, State) ->
Add timer for SST creation
2016-12-29 02:55:28 +00:00
SW = os:timestamp(),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{Length,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SlotIndex,
BlockIndex,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotsBin} = build_all_slots(SlotList, PressMethod),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
SummaryBin = build_table_summary(SlotIndex,
Level,
FirstKey,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Length,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
MaxSQN),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
ActualFilename =
write_file(RootPath, Filename, SummaryBin, SlotsBin, PressMethod),
Up level for yield to 2
2017-03-21 18:24:11 +00:00
YBQ = Level =< 2,
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
UpdState = read_file(ActualFilename,
State#state{root_path=RootPath,
yield_blockquery=YBQ}),
Some basic code layout work
2016-12-24 15:12:24 +00:00
Summary = UpdState#state.summary,
Add timer for SST creation
2016-12-29 02:55:28 +00:00
leveled_log:log_timer("SST08",
[ActualFilename, Level, Summary#summary.max_sqn],
SW),
Some basic code layout work
2016-12-24 15:12:24 +00:00
{reply,
{ok, {Summary#summary.first_key, Summary#summary.last_key}},
reader,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
UpdState#state{blockindex_cache = BlockIndex}}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
starting({sst_newlevelzero, RootPath, Filename,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
Slots, FetchFun, Penciller, MaxSQN,
PressMethod}, State) ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
SW0 = os:timestamp(),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
KVList = leveled_pmem:to_list(Slots, FetchFun),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
Time0 = timer:now_diff(os:timestamp(), SW0),
SW1 = os:timestamp(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{[], [], SlotList, FirstKey} = merge_lists(KVList, PressMethod),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
Time1 = timer:now_diff(os:timestamp(), SW1),
SW2 = os:timestamp(),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{SlotCount,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SlotIndex,
BlockIndex,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotsBin} = build_all_slots(SlotList, PressMethod),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
Time2 = timer:now_diff(os:timestamp(), SW2),
SW3 = os:timestamp(),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
SummaryBin = build_table_summary(SlotIndex,
0,
FirstKey,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
SlotCount,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
MaxSQN),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
Time3 = timer:now_diff(os:timestamp(), SW3),
SW4 = os:timestamp(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
ActualFilename =
write_file(RootPath, Filename, SummaryBin, SlotsBin, PressMethod),
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
UpdState = read_file(ActualFilename,
State#state{root_path = RootPath,
yield_blockquery = true}),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Summary = UpdState#state.summary,
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
Time4 = timer:now_diff(os:timestamp(), SW4),
Add timer for SST creation
2016-12-29 02:55:28 +00:00
leveled_log:log_timer("SST08",
[ActualFilename, 0, Summary#summary.max_sqn],
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
SW0),
leveled_log:log("SST11", [Time0, Time1, Time2, Time3, Time4]),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
case Penciller of
undefined ->
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{next_state,
reader,
UpdState#state{blockindex_cache = BlockIndex}};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
_ ->
leveled_penciller:pcl_confirml0complete(Penciller,
UpdState#state.filename,
Summary#summary.first_key,
Summary#summary.last_key),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{next_state,
reader,
UpdState#state{blockindex_cache = BlockIndex}}
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
end.
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Some basic code layout work
2016-12-24 15:12:24 +00:00
reader({get_kv, LedgerKey, Hash}, _From, State) ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
% Get a KV value and potentially take sample timings
{Result, UpdState, UpdTimings} =
fetch(LedgerKey, Hash, State, State#state.timings),
{UpdTimings0, CountDown} =
update_statetimings(UpdTimings, State#state.timings_countdown),
{reply, Result, reader, UpdState#state{timings = UpdTimings0,
timings_countdown = CountDown}};
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
reader({get_kvrange, StartKey, EndKey, ScanWidth, SlotList}, _From, State) ->
Move sst get_kv range back inside process Moved outside to stop blocking, but also avoids copy. Move back out to see if it may be related to the binary memory leak
2017-03-20 23:22:46 +00:00
{SlotsToFetchBinList, SlotsToPoint} = fetch_range(StartKey,
EndKey,
ScanWidth,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
SlotList,
Move sst get_kv range back inside process Moved outside to stop blocking, but also avoids copy. Move back out to see if it may be related to the binary memory leak
2017-03-20 23:22:46 +00:00
State),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
PressMethod = State#state.compression_method,
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
case State#state.yield_blockquery of
true ->
{reply,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{yield,
SlotsToFetchBinList,
SlotsToPoint,
PressMethod},
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
reader,
State};
false ->
{reply,
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
binaryslot_reader(SlotsToFetchBinList, PressMethod)
++ SlotsToPoint,
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
reader,
State}
end;
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
reader({get_slots, SlotList, SegList}, _From, State) ->
SlotBins =
read_slots(State#state.handle,
SlotList,
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
{SegList, State#state.blockindex_cache},
State#state.compression_method),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{reply, {SlotBins, State#state.compression_method}, reader, State};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
reader(get_maxsequencenumber, _From, State) ->
Summary = State#state.summary,
{reply, Summary#summary.max_sqn, reader, State};
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
reader(print_timings, _From, State) ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
log_timings(State#state.timings),
{reply, ok, reader, State};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
reader({set_for_delete, Penciller}, _From, State) ->
leveled_log:log("SST06", [State#state.filename]),
{reply,
ok,
delete_pending,
State#state{penciller=Penciller},
?DELETE_TIMEOUT};
reader(background_complete, _From, State) ->
Summary = State#state.summary,
{reply,
{ok,
State#state.filename,
Summary#summary.first_key,
Summary#summary.last_key},
reader,
State};
Some end-to-end testing
2016-12-24 17:48:31 +00:00
reader(close, _From, State) ->
ok = file:close(State#state.handle),
{stop, normal, ok, State}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
delete_pending({get_kv, LedgerKey, Hash}, _From, State) ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{Result, UpdState, _Ts} = fetch(LedgerKey, Hash, State, no_timing),
Remove lastfetch cache It appears to have some benefit at lower levels, but overall has less benefit at higher levels. Probably not worth having unless it cna be controlled to go in at the basement only.
2017-01-03 15:26:44 +00:00
{reply, Result, delete_pending, UpdState, ?DELETE_TIMEOUT};
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
delete_pending({get_kvrange, StartKey, EndKey, ScanWidth, SlotList},
_From, State) ->
Fix to delete_pending state
2017-03-20 23:43:31 +00:00
{SlotsToFetchBinList, SlotsToPoint} = fetch_range(StartKey,
EndKey,
ScanWidth,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
SlotList,
Fix to delete_pending state
2017-03-20 23:43:31 +00:00
State),
Yield on query selectively Still not clear if yielding is the cause of memory problems, but taking it away universally has impacted throughput. At the very least we should continue to yield on high-contention files (those at higher levels), where the processes are more likely to be quickly terminated anyway allowing GC to be invoked.
2017-03-21 11:03:29 +00:00
% Always yield as about to clear and de-reference
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
PressMethod = State#state.compression_method,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{reply,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{yield, SlotsToFetchBinList, SlotsToPoint, PressMethod},
State mixup in FSM
2016-12-29 02:40:09 +00:00
delete_pending,
State,
?DELETE_TIMEOUT};
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
delete_pending({get_slots, SlotList, SegList}, _From, State) ->
SlotBins =
read_slots(State#state.handle,
SlotList,
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
{SegList, State#state.blockindex_cache},
State#state.compression_method),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{reply,
{SlotBins, State#state.compression_method},
delete_pending,
State,
?DELETE_TIMEOUT};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
delete_pending(close, _From, State) ->
leveled_log:log("SST07", [State#state.filename]),
ok = file:close(State#state.handle),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:delete(filename:join(State#state.root_path,
State#state.filename)),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{stop, normal, ok, State}.
delete_pending(timeout, State) ->
ok = leveled_penciller:pcl_confirmdelete(State#state.penciller,
Fix-up message exchange on confirm delete
2017-01-17 11:18:58 +00:00
State#state.filename,
self()),
Timeout long-running snapshots Add logic to timeout long-running snapshots.
2017-04-05 09:16:01 +01:00
% If the next thing is another timeout - may be long-running snapshot, so
% back-off
Add compatibility for old and new random / rand functions
2017-07-31 20:20:39 +02:00
{next_state, delete_pending, State, leveled_rand:uniform(10) * ?DELETE_TIMEOUT};
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
delete_pending(close, State) ->
leveled_log:log("SST07", [State#state.filename]),
ok = file:close(State#state.handle),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:delete(filename:join(State#state.root_path,
State#state.filename)),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{stop, normal, State}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
handle_sync_event(_Msg, _From, StateName, State) ->
{reply, undefined, StateName, State}.
handle_event(_Msg, StateName, State) ->
{next_state, StateName, State}.
handle_info(_Msg, StateName, State) ->
{next_state, StateName, State}.
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
terminate(normal, delete_pending, _State) ->
ok;
Some basic code layout work
2016-12-24 15:12:24 +00:00
terminate(Reason, _StateName, State) ->
leveled_log:log("SST04", [Reason, State#state.filename]).
code_change(_OldVsn, StateName, State, _Extra) ->
{ok, StateName, State}.
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%%%============================================================================
%%% Internal Functions
%%%============================================================================
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
-spec fetch(tuple(),
{integer(), integer()}|integer(),
sst_state(), sst_timings())
-> {not_present|tuple(), sst_state(), sst_timings()}.
%% @doc
%%
%% Fetch a key from the store, potentially taking timings. Result should be
%% not_present if the key is not in the store.
fetch(LedgerKey, Hash, State, Timings0) ->
SW0 = os:timestamp(),
Some basic code layout work
2016-12-24 15:12:24 +00:00
Summary = State#state.summary,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
PressMethod = State#state.compression_method,
Remove SST blooms
2017-01-03 09:12:41 +00:00
Slot = lookup_slot(LedgerKey, Summary#summary.index),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{SW1, Timings1} = update_timings(SW0, Timings0, index_query, true),
Remove SST blooms
2017-01-03 09:12:41 +00:00
SlotID = Slot#slot_index_value.slot_id,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
Bloom = Slot#slot_index_value.bloom,
case leveled_tinybloom:check_hash(Hash, Bloom) of
false ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{_SW2, Timings2} =
update_timings(SW1, Timings1, tiny_bloom, false),
{not_present, State, Timings2};
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
true ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{SW2, Timings2} =
update_timings(SW1, Timings1, tiny_bloom, true),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
CachedBlockIdx = array:get(SlotID - 1,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
State#state.blockindex_cache),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
case CachedBlockIdx of
none ->
Remove lastfetch cache It appears to have some benefit at lower levels, but overall has less benefit at higher levels. Probably not worth having unless it cna be controlled to go in at the basement only.
2017-01-03 15:26:44 +00:00
SlotBin = read_slot(State#state.handle, Slot),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{Result, BlockLengths, BlockIdx} =
binaryslot_get(SlotBin, LedgerKey, Hash, PressMethod),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
BlockIndexCache = array:set(SlotID - 1,
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
<<BlockLengths/binary,
BlockIdx/binary>>,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
State#state.blockindex_cache),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{_SW3, Timings3} =
update_timings(SW2, Timings2, noncached_block, false),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
{Result,
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
State#state{blockindex_cache = BlockIndexCache},
Timings3};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<BlockLengths:24/binary, BlockIdx/binary>> ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
PosList = find_pos(BlockIdx,
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
extra_hash(Hash),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
[],
0),
case PosList of
[] ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{_SW3, Timings3} =
update_timings(SW2,
Timings2,
slot_index,
false),
{not_present, State, Timings3};
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
_ ->
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{SW3, Timings3} =
update_timings(SW2,
Timings2,
slot_index,
true),
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
StartPos = Slot#slot_index_value.start_position,
Result =
check_blocks(PosList,
State#state.handle,
StartPos,
BlockLengths,
LedgerKey,
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
PressMethod,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
not_present),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
{_SW4, Timings4} =
update_timings(SW3,
Timings3,
slot_fetch,
false),
{Result, State, Timings4}
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
end
end
Some basic code layout work
2016-12-24 15:12:24 +00:00
end.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
-spec fetch_range(tuple(), tuple(), integer(), false|list(integer()), sst_state())
-> {list(), list()}.
%% @doc
%% Fetch the contents of the SST file for a given key range. This will
%% pre-fetch some results, and append pointers for additional results.
%%
%% A filter can be provided based on the Segment ID (usable for hashable
%% objects not no_lookup entries) to accelerate the query if the 5-arity
%% version is used
fetch_range(StartKey, EndKey, ScanWidth, SegList, State) ->
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
Summary = State#state.summary,
Handle = State#state.handle,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
{Slots, RTrim} = lookup_slots(StartKey, EndKey, Summary#summary.index),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
Self = self(),
SL = length(Slots),
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
ExpandedSlots =
case SL of
1 ->
[Slot] = Slots,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
case RTrim of
true ->
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
[{pointer, Self, Slot, StartKey, EndKey}];
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
false ->
[{pointer, Self, Slot, StartKey, all}]
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
end;
N ->
{LSlot, MidSlots, RSlot} =
case N of
2 ->
[Slot1, Slot2] = Slots,
{Slot1, [], Slot2};
N ->
[Slot1|_Rest] = Slots,
SlotN = lists:last(Slots),
{Slot1, lists:sublist(Slots, 2, N - 2), SlotN}
end,
MidSlotPointers = lists:map(fun(S) ->
{pointer, Self, S, all, all}
end,
MidSlots),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
case RTrim of
true ->
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
[{pointer, Self, LSlot, StartKey, all}] ++
MidSlotPointers ++
[{pointer, Self, RSlot, all, EndKey}];
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
false ->
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
[{pointer, Self, LSlot, StartKey, all}] ++
MidSlotPointers ++
[{pointer, Self, RSlot, all, all}]
end
end,
{SlotsToFetch, SlotsToPoint} =
case ScanWidth of
SW when SW >= SL ->
{ExpandedSlots, []};
_ ->
lists:split(ScanWidth, ExpandedSlots)
end,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
SlotsToFetchBinList =
read_slots(Handle,
SlotsToFetch,
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
{SegList, State#state.blockindex_cache},
State#state.compression_method),
Range queries - pass out as binaries Avoid converting to erlang temr wihtin the FSM and then requiring a copy outside of the FSM - pass out as a binary
2017-03-17 10:47:20 +00:00
{SlotsToFetchBinList, SlotsToPoint}.
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
Some basic code layout work
2016-12-24 15:12:24 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
write_file(RootPath, Filename, SummaryBin, SlotsBin, PressMethod) ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
SummaryLength = byte_size(SummaryBin),
SlotsLength = byte_size(SlotsBin),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
{PendingName, FinalName} = generate_filenames(Filename),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
FileVersion = gen_fileversion(PressMethod),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:write_file(filename:join(RootPath, PendingName),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
<<FileVersion:8/integer,
SlotsLength:32/integer,
Fix to ensure directory structure created
2016-12-29 02:31:10 +00:00
SummaryLength:32/integer,
SlotsBin/binary,
SummaryBin/binary>>,
[raw]),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
case filelib:is_file(filename:join(RootPath, FinalName)) of
Some end-to-end testing
2016-12-24 17:48:31 +00:00
true ->
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
AltName = filename:join(RootPath, filename:basename(FinalName))
Some end-to-end testing
2016-12-24 17:48:31 +00:00
++ ?DISCARD_EXT,
leveled_log:log("SST05", [FinalName, AltName]),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:rename(filename:join(RootPath, FinalName), AltName);
Some end-to-end testing
2016-12-24 17:48:31 +00:00
false ->
ok
end,
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
file:rename(filename:join(RootPath, PendingName),
filename:join(RootPath, FinalName)),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
FinalName.
Some basic code layout work
2016-12-24 15:12:24 +00:00
read_file(Filename, State) ->
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{Handle, FileVersion, SummaryBin} =
open_reader(filename:join(State#state.root_path, Filename)),
UpdState0 = imp_fileversion(FileVersion, State),
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
{Summary, SlotList} = read_table_summary(SummaryBin),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
BlockIndexCache = array:new([{size, Summary#summary.size},
{default, none}]),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
UpdState1 = UpdState0#state{blockindex_cache = BlockIndexCache},
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
SlotIndex = from_list(SlotList),
UpdSummary = Summary#summary{index = SlotIndex},
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
leveled_log:log("SST03", [Filename,
Summary#summary.size,
Summary#summary.max_sqn]),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
UpdState1#state{summary = UpdSummary,
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
handle = Handle,
filename = Filename}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
gen_fileversion(PressMethod) ->
Bit1 =
case PressMethod of
lz4 -> 1;
native -> 0
end,
Bit1.
imp_fileversion(VersionInt, State) ->
UpdState =
case VersionInt band 1 of
0 ->
State#state{compression_method = native};
1 ->
State#state{compression_method = lz4}
end,
UpdState.
Some basic code layout work
2016-12-24 15:12:24 +00:00
open_reader(Filename) ->
{ok, Handle} = file:open(Filename, [binary, raw, read]),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{ok, Lengths} = file:pread(Handle, 0, 9),
<<FileVersion:8/integer,
SlotsLength:32/integer,
SummaryLength:32/integer>> = Lengths,
{ok, SummaryBin} = file:pread(Handle, SlotsLength + 9, SummaryLength),
{Handle, FileVersion, SummaryBin}.
Some basic code layout work
2016-12-24 15:12:24 +00:00
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
build_table_summary(SlotIndex, _Level, FirstKey, SlotCount, MaxSQN) ->
[{LastKey, _LastV}|_Rest] = SlotIndex,
Some basic code layout work
2016-12-24 15:12:24 +00:00
Summary = #summary{first_key = FirstKey,
last_key = LastKey,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
size = SlotCount,
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
max_sqn = MaxSQN},
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
SummBin = term_to_binary({Summary, lists:reverse(SlotIndex)},
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
?BINARY_SETTINGS),
Build the table summary The table summary will be a skiplist, and this and the slot binary will be CRC checked
2016-12-24 01:23:40 +00:00
SummCRC = erlang:crc32(SummBin),
<<SummCRC:32/integer, SummBin/binary>>.
read_table_summary(BinWithCheck) ->
<<SummCRC:32/integer, SummBin/binary>> = BinWithCheck,
CRCCheck = erlang:crc32(SummBin),
if
CRCCheck == SummCRC ->
Some basic code layout work
2016-12-24 15:12:24 +00:00
% If not might it might be possible to rebuild from all the slots
Build the table summary The table summary will be a skiplist, and this and the slot binary will be CRC checked
2016-12-24 01:23:40 +00:00
binary_to_term(SummBin)
end.
Settled on sizes Also removed length check due to warning in Erlang guidance about non-constant time nature of this command. Intend to remove lengths from elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
build_all_slots(SlotList, PressMethod) ->
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
SlotCount = length(SlotList),
BuildResponse = build_all_slots(SlotList,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
9,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
1,
[],
array:new([{size, SlotCount},
{default, none}]),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
<<>>,
PressMethod),
Tidy Up All Hashes As we're no longer generating a summayr bloom - no need to collect a big list of hashes whilst building the sst file
2017-01-03 18:20:28 +00:00
{SlotIndex, BlockIndex, SlotsBin} = BuildResponse,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{SlotCount, SlotIndex, BlockIndex, SlotsBin}.
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
build_all_slots([], _Pos, _SlotID,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotIdxAcc, BlockIdxAcc, SlotBinAcc,
_PressMethod) ->
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{SlotIdxAcc, BlockIdxAcc, SlotBinAcc};
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
build_all_slots([SlotD|Rest], Pos, SlotID,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotIdxAcc, BlockIdxAcc, SlotBinAcc,
PressMethod) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{BlockIdx, SlotBin, HashList, LastKey} = SlotD,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Length = byte_size(SlotBin),
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
Bloom = leveled_tinybloom:create_bloom(HashList),
Confirm timings
2016-12-23 18:08:22 +00:00
SlotIndexV = #slot_index_value{slot_id = SlotID,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
start_position = Pos,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
length = Length,
bloom = Bloom},
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
build_all_slots(Rest,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Pos + Length,
Confirm timings
2016-12-23 18:08:22 +00:00
SlotID + 1,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
[{LastKey, SlotIndexV}|SlotIdxAcc],
array:set(SlotID - 1, BlockIdx, BlockIdxAcc),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
<<SlotBinAcc/binary, SlotBin/binary>>,
PressMethod).
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
Some end-to-end testing
2016-12-24 17:48:31 +00:00
generate_filenames(RootFilename) ->
Ext = filename:extension(RootFilename),
Components = filename:split(RootFilename),
case Ext of
[] ->
{filename:join(Components) ++ ".pnd",
filename:join(Components) ++ ".sst"};
Ext ->
DN = filename:dirname(RootFilename),
FP_NOEXT = filename:basename(RootFilename, Ext),
{filename:join(DN, FP_NOEXT) ++ ".pnd",
filename:join(DN, FP_NOEXT) ++ ".sst"}
end.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
-spec serialise_block(any(), press_methods()) -> binary().
Split out block serialisation To allow for alternate compression scenarios to be more easily tested
2017-11-03 11:04:31 +00:00
%% @doc
%% Convert term to binary
%% Function split out to make it easier to experiment with different
%% compression methods. Also, perhaps standardise applictaion of CRC
%% checks
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
serialise_block(Term, lz4) ->
Add LZ4
2017-11-03 14:18:49 +00:00
{ok, Bin} = lz4:pack(term_to_binary(Term)),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
Bin;
serialise_block(Term, native) ->
term_to_binary(Term, ?BINARY_SETTINGS).
Split out block serialisation To allow for alternate compression scenarios to be more easily tested
2017-11-03 11:04:31 +00:00
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
-spec deserialise_block(binary(), press_methods()) -> any().
Split out block serialisation To allow for alternate compression scenarios to be more easily tested
2017-11-03 11:04:31 +00:00
%% @doc
%% Convert binary to term
%% Function split out to make it easier to experiment with different
%% compression methods. Also, perhaps standardise applictaion of CRC
%% checks
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
deserialise_block(Bin, lz4) ->
Add LZ4
2017-11-03 14:18:49 +00:00
{ok, Bin0} = lz4:unpack(Bin),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
binary_to_term(Bin0);
deserialise_block(Bin, native) ->
binary_to_term(Bin).
Split out block serialisation To allow for alternate compression scenarios to be more easily tested
2017-11-03 11:04:31 +00:00
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
%%%============================================================================
%%% SlotIndex Implementation
%%%============================================================================
%% The Slot Index is stored as a flat (sorted) list of {Key, Slot} where Key
%% is the last key within the slot.
%%
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
%% This implementation of the SlotIndex uses leveled_tree
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
from_list(SlotList) ->
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
leveled_tree:from_orderedlist(SlotList, ?TREE_TYPE, ?TREE_SIZE).
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
lookup_slot(Key, Tree) ->
StartKeyFun =
fun(_V) ->
all
end,
% The penciller should never ask for presence out of range - so will
% always return a slot (As we don't compare to StartKey)
{_LK, Slot} = leveled_tree:search(Key, Tree, StartKeyFun),
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
Slot.
Add use of leveled_tree to sst summary
2017-01-23 22:58:51 +00:00
lookup_slots(StartKey, EndKey, Tree) ->
StartKeyFun =
fun(_V) ->
all
end,
MapFun =
fun({_LK, Slot}) ->
Slot
end,
SlotList = leveled_tree:search_range(StartKey, EndKey, Tree, StartKeyFun),
{EK, _EndSlot} = lists:last(SlotList),
RTrim RTrim only worked in special case of key matching, that would never occur in real world range query. RTrim should really check for key passing. Returning empty list should not be possible - unless the query is outside of the range entirely (and such a query should never go to this SST).
2017-03-16 08:37:36 +00:00
{lists:map(MapFun, SlotList), not leveled_codec:endkey_passed(EK, EndKey)}.
Switch summary index Simplify the summayr index implementation
2017-01-04 14:26:11 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%%%============================================================================
%%% Slot Implementation
%%%============================================================================
%% Implementing a slot has gone through numerous iterations. One of the most
%% critical considerations has been the cost of the binary_to_term and
%% term_to_binary calls for different sizes of slots and different data types.
%%
Update SST comments
2017-03-04 20:47:46 +00:00
%% Microbenchmarking indicated that flat lists were the fastest at sst build
%% time. However, the lists need scanning at query time - and so give longer
%% lookups. Bigger slots did better at term_to_binary time. However
%% binary_to_term is an often repeated task, and this is better with smaller
%% slots.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
%%
%% The outcome has been to divide the slot into four small blocks to minimise
%% the binary_to_term time. A binary index is provided for the slot for all
%% Keys that are directly fetchable (i.e. standard keys not index keys).
%%
%% The division and use of a list saves about 100 microseconds per fetch when
%% compared to using a 128-member gb:tree.
%%
%% The binary index is cacheable and doubles as a not_present filter, as it is
%% based on a 17-bit hash (so 0.0039 fpr).
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
generate_binary_slot(Lookup, KVL, PressMethod) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
HashFoldFun =
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
fun({K, V}, {PosBinAcc, NoHashCount, HashAcc}) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{_SQN, H1} = leveled_codec:strip_to_seqnhashonly({K, V}),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
PosH1 = extra_hash(H1),
case is_integer(PosH1) of
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
true ->
case NoHashCount of
0 ->
Tidy Up All Hashes As we're no longer generating a summayr bloom - no need to collect a big list of hashes whilst building the sst file
2017-01-03 18:20:28 +00:00
{<<1:1/integer,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
PosH1:15/integer,
PosBinAcc/binary>>,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
0,
[H1|HashAcc]};
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
N ->
% The No Hash Count is an integer between 0 and 127
% and so at read time should count NHC + 1
NHC = N - 1,
Tidy Up All Hashes As we're no longer generating a summayr bloom - no need to collect a big list of hashes whilst building the sst file
2017-01-03 18:20:28 +00:00
{<<1:1/integer,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
PosH1:15/integer,
0:1/integer,
NHC:7/integer,
PosBinAcc/binary>>,
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
0,
HashAcc}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end;
false ->
Re-introduce tinybloom to SST This had been removed due to the CPU cost of adding - however then the tinybloom wa simplemented by directly manipulating bits through binary comprehension - rather than applying bor band bsl bsr operations. With these operations the cost of producing and checking the bloom is <10% by comparison.
2017-01-24 21:51:12 +00:00
{PosBinAcc, NoHashCount + 1, HashAcc}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end
end,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{HashL, PosBinIndex} =
case Lookup of
lookup ->
{PosBinIndex0,
NHC,
HashL0} = lists:foldr(HashFoldFun, {<<>>, 0, []}, KVL),
PosBinIndex1 =
case NHC of
0 ->
PosBinIndex0;
_ ->
N = NHC - 1,
<<0:1/integer, N:7/integer, PosBinIndex0/binary>>
end,
{HashL0, PosBinIndex1};
no_lookup ->
{[], <<0:1/integer, 127:7/integer>>}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{SideBlockSize, MidBlockSize} =
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case Lookup of
lookup ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?LOOK_BLOCKSIZE;
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
no_lookup ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?NOLOOK_BLOCKSIZE
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{B1, B2, B3, B4, B5} =
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
case length(KVL) of
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< SideBlockSize ->
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{serialise_block(KVL, PressMethod),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>,
<<0:0>>,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<0:0>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< 2 * SideBlockSize ->
{KVLA, KVLB} = lists:split(SideBlockSize, KVL),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{serialise_block(KVLA, PressMethod),
serialise_block(KVLB, PressMethod),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<0:0>>,
<<0:0>>};
L when L =< (2 * SideBlockSize + MidBlockSize) ->
{KVLA, KVLB_Rest} = lists:split(SideBlockSize, KVL),
{KVLB, KVLC} = lists:split(SideBlockSize, KVLB_Rest),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{serialise_block(KVLA, PressMethod),
serialise_block(KVLB, PressMethod),
serialise_block(KVLC, PressMethod),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<0:0>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< (3 * SideBlockSize + MidBlockSize) ->
{KVLA, KVLB_Rest} = lists:split(SideBlockSize, KVL),
{KVLB, KVLC_Rest} = lists:split(SideBlockSize, KVLB_Rest),
{KVLC, KVLD} = lists:split(MidBlockSize, KVLC_Rest),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{serialise_block(KVLA, PressMethod),
serialise_block(KVLB, PressMethod),
serialise_block(KVLC, PressMethod),
serialise_block(KVLD, PressMethod),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<0:0>>};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
L when L =< (4 * SideBlockSize + MidBlockSize) ->
{KVLA, KVLB_Rest} = lists:split(SideBlockSize, KVL),
{KVLB, KVLC_Rest} = lists:split(SideBlockSize, KVLB_Rest),
{KVLC, KVLD_Rest} = lists:split(MidBlockSize, KVLC_Rest),
{KVLD, KVLE} = lists:split(SideBlockSize, KVLD_Rest),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{serialise_block(KVLA, PressMethod),
serialise_block(KVLB, PressMethod),
serialise_block(KVLC, PressMethod),
serialise_block(KVLD, PressMethod),
serialise_block(KVLE, PressMethod)}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
B1P = byte_size(PosBinIndex),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
B1L = byte_size(B1),
B2L = byte_size(B2),
B3L = byte_size(B3),
B4L = byte_size(B4),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B5L = byte_size(B5),
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Lengths = <<B1P:32/integer,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B4L:32/integer,
B5L:32/integer>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SlotBin = <<Lengths/binary,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
PosBinIndex/binary,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B1/binary, B2/binary, B3/binary, B4/binary, B5/binary>>,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
CRC32 = erlang:crc32(SlotBin),
FullBin = <<CRC32:32/integer, SlotBin/binary>>,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{LastKey, _LV} = lists:last(KVL),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{<<Lengths/binary, PosBinIndex/binary>>, FullBin, HashL, LastKey}.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
% Acc should start as not_present if LedgerKey is a key, and a list if
% LedgerKey is false
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
check_blocks([], _Handle, _StartPos, _BlockLengths,
_LedgerKeyToCheck, _PressMethod, Acc) ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
Acc;
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
check_blocks([Pos|Rest], Handle, StartPos,
BlockLengths, LedgerKeyToCheck, PressMethod, Acc) ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockNumber, BlockPos} = revert_position(Pos),
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
BlockBin =
read_block(Handle,
StartPos,
BlockLengths,
BlockNumber),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
BlockL = deserialise_block(BlockBin, PressMethod),
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{K, V} = lists:nth(BlockPos, BlockL),
case K of
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
LedgerKeyToCheck ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{K, V};
_ ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
case LedgerKeyToCheck of
false ->
Acc ++ [{K, V}];
_ ->
check_blocks(Rest, Handle, StartPos, BlockLengths,
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
LedgerKeyToCheck, PressMethod, Acc)
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
end
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
end.
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
read_block(Handle, StartPos, BlockLengths, BlockID) ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockPos, Offset, Length} = block_offsetandlength(BlockLengths, BlockID),
{ok, BlockBin} = file:pread(Handle,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
StartPos
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
+ BlockPos
+ Offset
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
+ 28,
% 4-byte CRC, 4 byte pos, 5x4 byte lengths
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Length),
BlockBin.
read_slot(Handle, Slot) ->
{ok, SlotBin} = file:pread(Handle,
Slot#slot_index_value.start_position,
Slot#slot_index_value.length),
SlotBin.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
pointer_mapfun(Pointer) ->
{Slot, SK, EK} =
case Pointer of
{pointer, _Pid, Slot0, SK0, EK0} ->
{Slot0, SK0, EK0};
{pointer, Slot0, SK0, EK0} ->
{Slot0, SK0, EK0}
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
end,
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
{Slot#slot_index_value.start_position,
Slot#slot_index_value.length,
Slot#slot_index_value.slot_id,
SK,
EK}.
-spec binarysplit_mapfun(binary(), integer()) -> fun().
%% @doc
%% Return a function that can pull individual slot binaries from a binary
%% covering multiple slots
binarysplit_mapfun(MultiSlotBin, StartPos) ->
fun({SP, L, _ID, SK, EK}) ->
Start = SP - StartPos,
<<_Pre:Start/binary, SlotBin:L/binary, _Post/binary>> = MultiSlotBin,
{SlotBin, SK, EK}
end.
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
-spec read_slots(file:io_device(), list(),
{false|list(), any()}, press_methods()) -> list().
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
%% @doc
%% The reading of sots will return a list of either 2-tuples containing
%% {K, V} pairs - or 3-tuples containing {Binary, SK, EK}. The 3 tuples
%% can be exploded into lists of {K, V} pairs using the binaryslot_reader/2
%% function
%%
%% Reading slots is generally unfiltered, but in the sepcial case when
%% querting across slots when only matching segment IDs are required the
%% BlockIndexCache can be used
%%
%% Note that false positives will be passed through. It is important that
%% any key comparison between levels should allow for a non-matching key to
%% be considered as superior to a matching key - as otherwise a matching key
%% may be intermittently removed from the result set
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
read_slots(Handle, SlotList, {false, _BlockIndexCache}, _PressMethod) ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
% No list of segments passed
LengthList = lists:map(fun pointer_mapfun/1, SlotList),
{MultiSlotBin, StartPos} = read_length_list(Handle, LengthList),
lists:map(binarysplit_mapfun(MultiSlotBin, StartPos), LengthList);
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
read_slots(Handle, SlotList, {SegList, BlockIndexCache}, PressMethod) ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
% List of segments passed so only {K, V} pairs matching those segments
% should be returned. This required the {K, V} pair to have been added
% with the appropriate hash - if the pair were added with no_lookup as
% the hash value this will fial unexpectedly.
BinMapFun =
fun(Pointer, Acc) ->
{SP, _L, ID, _SK, _EK} = pointer_mapfun(Pointer),
case array:get(ID - 1, BlockIndexCache) of
none ->
% If there is an attempt to use the seg list query and the
% index block cache isn't cached for any part this may be
% slower as each slot will be read in turn
LengthDetails = pointer_mapfun(Pointer),
{MultiSlotBin, StartPos} =
read_length_list(Handle, [LengthDetails]),
MapFun = binarysplit_mapfun(MultiSlotBin, StartPos),
Acc ++ [MapFun(LengthDetails)];
<<BlockLengths:24/binary, BlockIdx/binary>> ->
% If there is a BlockIndex cached then we cna use it to
% check to see if any of the expected segments are
% present without lifting the slot off disk. Also the
% fact that we know position can be used to filter out
% other keys
case find_pos(BlockIdx, SegList, [], 0) of
[] ->
Acc;
PL ->
Acc ++ check_blocks(PL, Handle, SP, BlockLengths,
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
false, PressMethod, [])
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
end
end
end,
lists:foldl(BinMapFun, [], SlotList).
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
-spec binaryslot_reader(list({tuple(), tuple()}|{binary(), tuple(), tuple()}),
native|lz4) -> list({tuple(), tuple()}).
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
%% @doc
%% Read the binary slots converting them to {K, V} pairs if they were not
%% already {K, V} pairs
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
binaryslot_reader(SlotBinsToFetch, PressMethod) ->
binaryslot_reader(SlotBinsToFetch, PressMethod, []).
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
binaryslot_reader([], _PressMethod, Acc) ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
Acc;
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
binaryslot_reader([{SlotBin, SK, EK}|Tail], PressMethod, Acc) ->
binaryslot_reader(Tail,
PressMethod,
Acc ++ binaryslot_trimmedlist(SlotBin,
SK, EK,
PressMethod));
binaryslot_reader([{K, V}|Tail], PressMethod, Acc) ->
binaryslot_reader(Tail, PressMethod, Acc ++ [{K, V}]).
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
read_length_list(Handle, LengthList) ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
StartPos = element(1, lists:nth(1, LengthList)),
EndPos = element(1, lists:last(LengthList))
+ element(2, lists:last(LengthList)),
{ok, MultiSlotBin} = file:pread(Handle, StartPos, EndPos - StartPos),
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
{MultiSlotBin, StartPos}.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
binaryslot_get(FullBin, Key, Hash, PressMethod) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
case crc_check_slot(FullBin) of
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockLengths, Rest} ->
<<B1P:32/integer, _R/binary>> = BlockLengths,
<<PosBinIndex:B1P/binary, Blocks/binary>> = Rest,
PosList = find_pos(PosBinIndex,
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
extra_hash(Hash),
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
[],
0),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{fetch_value(PosList, BlockLengths, Blocks, Key, PressMethod),
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
BlockLengths,
PosBinIndex};
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_wonky ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{not_present,
none,
none}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end.
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
binaryslot_tolist(FullBin, PressMethod) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
BlockFetchFun =
fun(Length, {Acc, Bin}) ->
case Length of
0 ->
{Acc, Bin};
_ ->
<<Block:Length/binary, Rest/binary>> = Bin,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{Acc ++ deserialise_block(Block, PressMethod), Rest}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end
end,
{Out, _Rem} =
case crc_check_slot(FullBin) of
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockLengths, RestBin} ->
<<B1P:32/integer,
B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B4L:32/integer,
B5L:32/integer>> = BlockLengths,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
<<_PosBinIndex:B1P/binary, Blocks/binary>> = RestBin,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
lists:foldl(BlockFetchFun,
{[], Blocks},
[B1L, B2L, B3L, B4L, B5L]);
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_wonky ->
{[], <<>>}
end,
Out.
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
binaryslot_trimmedlist(FullBin, all, all, PressMethod) ->
binaryslot_tolist(FullBin, PressMethod);
binaryslot_trimmedlist(FullBin, StartKey, EndKey, PressMethod) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
LTrimFun = fun({K, _V}) -> K < StartKey end,
RTrimFun = fun({K, _V}) -> not leveled_codec:endkey_passed(EndKey, K) end,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
% It will be more effecient to check a subset of blocks. To work out
% the best subset we always look in the middle block of 5, and based on
% the first and last keys of that middle block when compared to the Start
% and EndKey of the query determines a subset of blocks
%
% This isn't perfectly efficient, esepcially if the query overlaps Block2
% and Block3 (as Block 1 will also be checked), but finessing this last
% scenario is hard to do in concise code
BlocksToCheck =
case crc_check_slot(FullBin) of
{BlockLengths, RestBin} ->
<<B1P:32/integer,
B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
B4L:32/integer,
B5L:32/integer>> = BlockLengths,
<<_PosBinIndex:B1P/binary,
Block1:B1L/binary, Block2:B2L/binary,
MidBlock:B3L/binary,
Block4:B4L/binary, Block5:B5L/binary>> = RestBin,
case B3L of
0 ->
[Block1, Block2];
_ ->
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
MidBlockList =
deserialise_block(MidBlock, PressMethod),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{MidFirst, _} = lists:nth(1, MidBlockList),
{MidLast, _} = lists:last(MidBlockList),
Split = {StartKey > MidLast,
StartKey >= MidFirst,
leveled_codec:endkey_passed(EndKey,
MidFirst),
leveled_codec:endkey_passed(EndKey,
MidLast)},
case Split of
{true, _, _, _} ->
[Block4, Block5];
{false, true, false, true} ->
[MidBlockList];
{false, true, false, false} ->
[MidBlockList, Block4, Block5];
{false, false, true, true} ->
[Block1, Block2];
{false, false, false, true} ->
[Block1, Block2, MidBlockList];
_ ->
[Block1, Block2, MidBlockList, Block4, Block5]
end
end;
crc_wonky ->
[]
end,
BlockCheckFun =
fun(Block, {Acc, Continue}) ->
case {Block, Continue} of
{<<>>, _} ->
{Acc, false};
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
{_, true} ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
BlockList =
case is_binary(Block) of
true ->
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
deserialise_block(Block, PressMethod);
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
false ->
Block
end,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{LastKey, _LV} = lists:last(BlockList),
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
case StartKey > LastKey of
true ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc, true};
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
false ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{_LDrop, RKeep} = lists:splitwith(LTrimFun,
BlockList),
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
case leveled_codec:endkey_passed(EndKey, LastKey) of
true ->
{LKeep, _RDrop} = lists:splitwith(RTrimFun, RKeep),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc ++ LKeep, false};
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
false ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc ++ RKeep, true}
Curtail trimmed slot crazyness There was complicated and confusing code that achieved nothing for effiency when trimming slots. the expensive part (binary_to_term) was still needed on every block, and it was hard to get code coverage and make sense of what it was really trying to achieve. This is now much simpler - and may set us up for potential further indexing help.
2017-03-19 21:47:22 +00:00
end
end;
{_ , false} ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc, false}
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end
end,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Acc, _Continue} = lists:foldl(BlockCheckFun, {[], true}, BlocksToCheck),
Acc.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_check_slot(FullBin) ->
<<CRC32:32/integer, SlotBin/binary>> = FullBin,
case erlang:crc32(SlotBin) of
CRC32 ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<BlockLengths:24/binary, Rest/binary>> = SlotBin,
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockLengths, Rest};
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
_ ->
Log improvements Log on bad CRC, and also not seeing SST timing logs, so log these more frequently
2017-01-02 18:54:19 +00:00
leveled_log:log("SST09", []),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
crc_wonky
end.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
block_offsetandlength(BlockLengths, BlockID) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
<<BlocksPos:32/integer, BlockLengths0:20/binary>> = BlockLengths,
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
case BlockID of
1 ->
<<B1L:32/integer, _BR/binary>> = BlockLengths0,
{BlocksPos, 0, B1L};
2 ->
<<B1L:32/integer, B2L:32/integer, _BR/binary>> = BlockLengths0,
{BlocksPos, B1L, B2L};
3 ->
<<B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
_BR/binary>> = BlockLengths0,
{BlocksPos, B1L + B2L, B3L};
4 ->
<<B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
B4L:32/integer,
_BR/binary>> = BlockLengths0,
{BlocksPos, B1L + B2L + B3L, B4L};
5 ->
<<B1L:32/integer,
B2L:32/integer,
B3L:32/integer,
B4L:32/integer,
B5L:32/integer,
_BR/binary>> = BlockLengths0,
{BlocksPos, B1L + B2L + B3L + B4L, B5L}
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
end.
Switch segment and extra hash More entropy by using the position index with the segment hash - so this would be a better filter to apply. Also could increase the key count now, as extra hash can be larger. As an aside - a leveled_iclerk unit test failure appeared - the range was just wrong. Don't know why this strated happening
2017-10-24 14:32:04 +01:00
extra_hash({SegHash, _ExtraHash}) when is_integer(SegHash) ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
tune_hash(SegHash);
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
extra_hash(NotHash) ->
NotHash.
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
tune_hash(SegHash) ->
SegHash band 32767.
Add tests for different tree sizes Note that accelerating segment_list queries will not work for tree sizes smaller than small. How to flag this up? Should smaller tree sizes just be removed from leveled_tictac?
2017-11-01 11:51:51 +00:00
tune_seglist(SegList) ->
case is_list(SegList) of
true ->
lists:map(fun tune_hash/1, SegList);
false ->
SegList
end.
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
fetch_value([], _BlockLengths, _Blocks, _Key, _PressMethod) ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
not_present;
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
fetch_value([Pos|Rest], BlockLengths, Blocks, Key, PressMethod) ->
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
{BlockNumber, BlockPos} = revert_position(Pos),
{_BlockPos,
Offset,
Length} = block_offsetandlength(BlockLengths, BlockNumber),
<<_Pre:Offset/binary, Block:Length/binary, _Rest/binary>> = Blocks,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
BlockL = deserialise_block(Block, PressMethod),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{K, V} = lists:nth(BlockPos, BlockL),
case K of
Key ->
{K, V};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
_ ->
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
fetch_value(Rest, BlockLengths, Blocks, Key, PressMethod)
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
end.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
revert_position(Pos) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{SideBlockSize, MidBlockSize} = ?LOOK_BLOCKSIZE,
case Pos < 2 * SideBlockSize of
true ->
{(Pos div SideBlockSize) + 1, (Pos rem SideBlockSize) + 1};
false ->
case Pos < (2 * SideBlockSize + MidBlockSize) of
true ->
{3, ((Pos - 2 * SideBlockSize) rem MidBlockSize) + 1};
false ->
TailPos = Pos - 2 * SideBlockSize - MidBlockSize,
{(TailPos div SideBlockSize) + 4,
(TailPos rem SideBlockSize) + 1}
end
end.
Fetch specific block only Rely on CRC check in zlib. Still need to catch on failure
2017-03-07 20:19:11 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
find_pos(<<>>, _Hash, PosList, _Count) ->
PosList;
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
find_pos(<<1:1/integer, PotentialHit:15/integer, T/binary>>,
HashList, PosList, Count) when is_list(HashList) ->
case lists:member(PotentialHit, HashList) of
true ->
find_pos(T, HashList, PosList ++ [Count], Count + 1);
false ->
find_pos(T, HashList, PosList, Count + 1)
end;
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
find_pos(<<1:1/integer, Hash:15/integer, T/binary>>, Hash, PosList, Count) ->
find_pos(T, Hash, PosList ++ [Count], Count + 1);
find_pos(<<1:1/integer, _Miss:15/integer, T/binary>>, Hash, PosList, Count) ->
find_pos(T, Hash, PosList, Count + 1);
find_pos(<<0:1/integer, NHC:7/integer, T/binary>>, Hash, PosList, Count) ->
find_pos(T, Hash, PosList, Count + NHC + 1).
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
%%%============================================================================
%%% Merge Functions
%%%============================================================================
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
%% The source lists are merged into lists of slots before the file is created
%% At Level zero, there will be a single source list - and this will always be
%% split into standard size slots
%%
%% At lower levels there will be two source lists and they will need to be
%% merged to ensure that the best conflicting answer survives and compactable
%% KV pairs are discarded.
%%
%% At lower levels slots can be larger if there are no lookup keys present in
%% the slot. This is to slow the growth of the manifest/number-of-files when
%% large numbers of index keys are present - as well as improving compression
%% ratios in the Ledger.
%%
%% The outcome of merge_lists/1 and merge_lists/3 should be an list of slots.
%% Each slot should be ordered by Key and be of the form {Flag, KVList}, where
%% Flag can either be lookup or no-lookup. The list of slots should also be
%% ordered by Key (i.e. the first key in the slot)
%%
%% For merging ...
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
%% Compare the keys at the head of the list, and either skip that "best" key or
%% identify as the next key.
%%
%% The logic needs to change if the file is in the basement level, as keys with
%% expired timestamps need not be written at this level
%%
%% The best key is considered to be the lowest key in erlang term order. If
%% there are matching keys then the highest sequence number must be chosen and
%% any lower sequence numbers should be compacted out of existence
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
merge_lists(KVList1, PressMethod) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
SlotCount = length(KVList1) div ?LOOK_SLOTSIZE,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{[],
[],
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
split_lists(KVList1, [], SlotCount, PressMethod),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
element(1, lists:nth(1, KVList1))}.
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
split_lists([], SlotLists, 0, _PressMethod) ->
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
lists:reverse(SlotLists);
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
split_lists(LastPuff, SlotLists, 0, PressMethod) ->
SlotD = generate_binary_slot(lookup, LastPuff, PressMethod),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
lists:reverse([SlotD|SlotLists]);
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
split_lists(KVList1, SlotLists, N, PressMethod) ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
{Slot, KVListRem} = lists:split(?LOOK_SLOTSIZE, KVList1),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotD = generate_binary_slot(lookup, Slot, PressMethod),
split_lists(KVListRem, [SlotD|SlotLists], N - 1, PressMethod).
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
merge_lists(KVList1, KVList2, LevelInfo, PressMethod) ->
merge_lists(KVList1, KVList2, LevelInfo, [], null, 0, PressMethod).
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
merge_lists(KVList1, KVList2, _LI, SlotList, FirstKey, ?MAX_SLOTS,
_PressMethod) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{KVList1, KVList2, lists:reverse(SlotList), FirstKey};
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
merge_lists([], [], _LI, SlotList, FirstKey, _SlotCount, _PressMethod) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{[], [], lists:reverse(SlotList), FirstKey};
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
merge_lists(KVList1, KVList2, LI, SlotList, FirstKey, SlotCount,
PressMethod) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{KVRem1, KVRem2, Slot, FK0} =
form_slot(KVList1, KVList2, LI, no_lookup, 0, [], FirstKey),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case Slot of
Fix empty slot issue
2017-03-11 12:41:30 +00:00
{_, []} ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
merge_lists(KVRem1,
KVRem2,
LI,
SlotList,
FK0,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotCount,
PressMethod);
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{Lookup, KVL} ->
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotD = generate_binary_slot(Lookup, KVL, PressMethod),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
merge_lists(KVRem1,
KVRem2,
LI,
[SlotD|SlotList],
FK0,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
SlotCount + 1,
PressMethod)
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end.
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
form_slot([], [], _LI, Type, _Size, Slot, FK) ->
{[], [], {Type, lists:reverse(Slot)}, FK};
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
form_slot(KVList1, KVList2, _LI, lookup, ?LOOK_SLOTSIZE, Slot, FK) ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
{KVList1, KVList2, {lookup, lists:reverse(Slot)}, FK};
form_slot(KVList1, KVList2, _LI, no_lookup, ?NOLOOK_SLOTSIZE, Slot, FK) ->
{KVList1, KVList2, {no_lookup, lists:reverse(Slot)}, FK};
form_slot(KVList1, KVList2, {IsBasement, TS}, lookup, Size, Slot, FK) ->
case {key_dominates(KVList1, KVList2, {IsBasement, TS}), FK} of
{{{next_key, TopKV}, Rem1, Rem2}, _} ->
form_slot(Rem1,
Rem2,
{IsBasement, TS},
lookup,
Size + 1,
[TopKV|Slot],
FK);
{{skipped_key, Rem1, Rem2}, _} ->
form_slot(Rem1, Rem2, {IsBasement, TS}, lookup, Size, Slot, FK)
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end;
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
form_slot(KVList1, KVList2, {IsBasement, TS}, no_lookup, Size, Slot, FK) ->
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case key_dominates(KVList1, KVList2, {IsBasement, TS}) of
{{next_key, {TopK, TopV}}, Rem1, Rem2} ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
FK0 =
case FK of
null ->
TopK;
_ ->
FK
end,
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
case leveled_codec:to_lookup(TopK) of
no_lookup ->
form_slot(Rem1,
Rem2,
{IsBasement, TS},
no_lookup,
Size + 1,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
[{TopK, TopV}|Slot],
FK0);
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
lookup ->
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
case Size >= ?LOOK_SLOTSIZE of
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
true ->
{KVList1,
KVList2,
Fix bad exit with no FK
2017-03-14 00:52:07 +00:00
{no_lookup, lists:reverse(Slot)},
FK};
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
false ->
form_slot(Rem1,
Rem2,
{IsBasement, TS},
lookup,
Size + 1,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
[{TopK, TopV}|Slot],
FK0)
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
end
end;
{skipped_key, Rem1, Rem2} ->
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
form_slot(Rem1, Rem2, {IsBasement, TS}, no_lookup, Size, Slot, FK)
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
end.
key_dominates(KL1, KL2, Level) ->
key_dominates_expanded(maybe_expand_pointer(KL1),
maybe_expand_pointer(KL2),
Level).
key_dominates_expanded([H1|T1], [], Level) ->
case leveled_codec:maybe_reap_expiredkey(H1, Level) of
true ->
{skipped_key, T1, []};
false ->
{{next_key, H1}, T1, []}
end;
key_dominates_expanded([], [H2|T2], Level) ->
case leveled_codec:maybe_reap_expiredkey(H2, Level) of
true ->
{skipped_key, [], T2};
false ->
{{next_key, H2}, [], T2}
end;
key_dominates_expanded([H1|T1], [H2|T2], Level) ->
case leveled_codec:key_dominates(H1, H2) of
left_hand_first ->
case leveled_codec:maybe_reap_expiredkey(H1, Level) of
true ->
{skipped_key, T1, [H2|T2]};
false ->
{{next_key, H1}, T1, [H2|T2]}
end;
right_hand_first ->
case leveled_codec:maybe_reap_expiredkey(H2, Level) of
true ->
{skipped_key, [H1|T1], T2};
false ->
{{next_key, H2}, [H1|T1], T2}
end;
left_hand_dominant ->
{skipped_key, [H1|T1], T2};
right_hand_dominant ->
{skipped_key, T1, [H2|T2]}
end.
%% When a list is provided it may include a pointer to gain another batch of
%% entries from the same file, or a new batch of entries from another file
%%
%% This resultant list should include the Tail of any pointers added at the
%% end of the list
maybe_expand_pointer([]) ->
[];
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
maybe_expand_pointer([{pointer, SSTPid, Slot, StartKey, all}|Tail]) ->
expand_list_by_pointer({pointer, SSTPid, Slot, StartKey, all},
Tail,
?MERGE_SCANWIDTH);
Manifest now back to a simple list This has refactored code with the implementation of the manifest isolated in to a seperate module, and the pure async relationship between penciller and their clerk. However, the manifest is just a simple list at each level.
2017-01-17 10:12:15 +00:00
maybe_expand_pointer([{next, ManEntry, StartKey}|Tail]) ->
expand_list_by_pointer({next, ManEntry, StartKey, all},
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Tail,
?MERGE_SCANWIDTH);
maybe_expand_pointer(List) ->
List.
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
expand_list_by_pointer(Pointer, Tail, Width) ->
expand_list_by_pointer(Pointer, Tail, Width, false).
expand_list_by_pointer({pointer, SSTPid, Slot, StartKey, EndKey},
Tail, Width, SegList) ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
FoldFun =
fun(X, {Pointers, Remainder}) ->
case length(Pointers) of
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
L when L < Width ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
case X of
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
{pointer, SSTPid, S, SK, EK} ->
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{Pointers ++ [{pointer, S, SK, EK}], Remainder};
_ ->
{Pointers, Remainder ++ [X]}
end;
_ ->
{Pointers, Remainder ++ [X]}
end
end,
Handle mismatch on expanding pointer Remove the nasty legacy of hard-coding for a scan width of 1
2016-12-29 10:46:12 +00:00
InitAcc = {[{pointer, Slot, StartKey, EndKey}], []},
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{AccPointers, AccTail} = lists:foldl(FoldFun, InitAcc, Tail),
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
ExpPointers =
leveled_sst:sst_getfilteredslots(SSTPid, AccPointers, SegList),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
lists:append(ExpPointers, AccTail);
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
expand_list_by_pointer({next, ManEntry, StartKey, EndKey},
Tail, Width, SegList) ->
Manifest now back to a simple list This has refactored code with the implementation of the manifest isolated in to a seperate module, and the pure async relationship between penciller and their clerk. However, the manifest is just a simple list at each level.
2017-01-17 10:12:15 +00:00
SSTPid = ManEntry#manifest_entry.owner,
leveled_log:log("SST10", [SSTPid, is_process_alive(SSTPid)]),
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
ExpPointer =
leveled_sst:sst_getfilteredrange(SSTPid,
StartKey, EndKey,
Width, SegList),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
ExpPointer ++ Tail.
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
%%%============================================================================
%%% Timing Functions
%%%============================================================================
-spec update_statetimings(sst_timings(), integer())
-> {sst_timings(), integer()}.
%% @doc
%%
%% The timings state is either in countdown to the next set of samples of
%% we are actively collecting a sample. Active collection take place
%% when the countdown is 0. Once the sample has reached the expected count
%% then there is a log of that sample, and the countdown is restarted.
%%
%% Outside of sample windows the timings object should be set to the atom
%% no_timing. no_timing is a valid state for the cdb_timings type.
update_statetimings(no_timing, 0) ->
{#sst_timings{}, 0};
update_statetimings(Timings, 0) ->
case Timings#sst_timings.sample_count of
SC when SC >= ?TIMING_SAMPLESIZE ->
log_timings(Timings),
{no_timing, leveled_rand:uniform(2 * ?TIMING_SAMPLECOUNTDOWN)};
_SC ->
{Timings, 0}
end;
update_statetimings(no_timing, N) ->
{no_timing, N - 1}.
log_timings(no_timing) ->
ok;
log_timings(Timings) ->
leveled_log:log("SST12", [Timings#sst_timings.sample_count,
Timings#sst_timings.index_query_time,
Timings#sst_timings.tiny_bloom_time,
Timings#sst_timings.slot_index_time,
Timings#sst_timings.slot_fetch_time,
Timings#sst_timings.noncached_block_time,
Timings#sst_timings.tiny_bloom_count,
Timings#sst_timings.slot_index_count,
Timings#sst_timings.slot_fetch_count,
Timings#sst_timings.noncached_block_count]).
update_timings(_SW, no_timing, _Stage, _Continue) ->
{no_timing, no_timing};
update_timings(SW, Timings, Stage, Continue) ->
Timer = timer:now_diff(os:timestamp(), SW),
Timings0 =
case Stage of
index_query ->
IQT = Timings#sst_timings.index_query_time,
Timings#sst_timings{index_query_time = IQT + Timer};
tiny_bloom ->
TBT = Timings#sst_timings.tiny_bloom_time,
Timings#sst_timings{tiny_bloom_time = TBT + Timer};
slot_index ->
SIT = Timings#sst_timings.slot_index_time,
Timings#sst_timings{slot_index_time = SIT + Timer};
slot_fetch ->
SFT = Timings#sst_timings.slot_fetch_time,
Timings#sst_timings{slot_fetch_time = SFT + Timer};
noncached_block ->
NCT = Timings#sst_timings.noncached_block_time,
Timings#sst_timings{noncached_block_time = NCT + Timer}
end,
case Continue of
true ->
{os:timestamp(), Timings0};
false ->
Timings1 =
case Stage of
tiny_bloom ->
TBC = Timings#sst_timings.tiny_bloom_count,
Timings0#sst_timings{tiny_bloom_count = TBC + 1};
slot_index ->
SIC = Timings#sst_timings.slot_index_count,
Timings0#sst_timings{slot_index_count = SIC + 1};
slot_fetch ->
SFC = Timings#sst_timings.slot_fetch_count,
Timings0#sst_timings{slot_fetch_count = SFC + 1};
noncached_block ->
NCC = Timings#sst_timings.noncached_block_count,
Timings0#sst_timings{noncached_block_count = NCC + 1}
end,
SC = Timings1#sst_timings.sample_count,
{no_timing, Timings1#sst_timings{sample_count = SC + 1}}
end.
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
%%%============================================================================
%%% Test
%%%============================================================================
-ifdef(TEST).
generate_randomkeys(Seqn, Count, BucketRangeLow, BucketRangeHigh) ->
generate_randomkeys(Seqn,
Count,
[],
BucketRangeLow,
BucketRangeHigh).
generate_randomkeys(_Seqn, 0, Acc, _BucketLow, _BucketHigh) ->
Acc;
generate_randomkeys(Seqn, Count, Acc, BucketLow, BRange) ->
Add compatibility for old and new random / rand functions
2017-07-31 20:20:39 +02:00
BRand = leveled_rand:uniform(BRange),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
BNumber = string:right(integer_to_list(BucketLow + BRand), 4, $0),
Add compatibility for old and new random / rand functions
2017-07-31 20:20:39 +02:00
KNumber = string:right(integer_to_list(leveled_rand:uniform(1000)), 6, $0),
Extract Last Modified Date from Riak Object As part of process to supporting a recent changes index for near-real-time anti-entropy
2017-06-27 16:25:09 +01:00
LK = leveled_codec:to_ledgerkey("Bucket" ++ BNumber, "Key" ++ KNumber, o),
Add compatibility for old and new random / rand functions
2017-07-31 20:20:39 +02:00
Chunk = leveled_rand:rand_bytes(64),
Extract Last Modified Date from Riak Object As part of process to supporting a recent changes index for near-real-time anti-entropy
2017-06-27 16:25:09 +01:00
{_B, _K, MV, _H, _LMs} =
leveled_codec:generate_ledgerkv(LK, Seqn, Chunk, 64, infinity),
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
generate_randomkeys(Seqn + 1,
Count - 1,
Extract Last Modified Date from Riak Object As part of process to supporting a recent changes index for near-real-time anti-entropy
2017-06-27 16:25:09 +01:00
[{LK, MV}|Acc],
Initial functions and unit tests Try to replace SFT files with one that more natively supports features already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
BucketLow,
BRange).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
generate_indexkeys(Count) ->
generate_indexkeys(Count, []).
generate_indexkeys(0, IndexList) ->
IndexList;
generate_indexkeys(Count, IndexList) ->
Add compatibility for old and new random / rand functions
2017-07-31 20:20:39 +02:00
Changes = generate_indexkey(leveled_rand:uniform(8000), Count),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
generate_indexkeys(Count - 1, IndexList ++ Changes).
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
generate_indexkey(Term, Count) ->
IndexSpecs = [{add, "t1_int", Term}],
Add temporary aae index Pending ct tests. The aae index should expire after limit_minutes and be on an index which is rounded to unit_minutes.
2017-06-30 10:03:36 +01:00
leveled_codec:idx_indexspecs(IndexSpecs,
"Bucket",
"Key" ++ integer_to_list(Count),
Count,
infinity).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Unit tets of lookup over-size issue A mistake meant resetting to lookup on a skipped key would cause issues if the skipped key ocurred under a no_lookup slot after the ?SLOT_SIZE had been reached. This caused the slot to switch to lookup, but beyond the maximum size
2017-03-11 00:03:55 +00:00
form_slot_test() ->
% If a skip key happens, mustn't switch to loookup by accident as could be
% over the expected size
SkippingKV = {{o, "B1", "K9999", null}, {9999, tomb, 1234567, {}}},
Slot = [{{o, "B1", "K5", null}, {5, active, 99234567, {}}}],
R1 = form_slot([SkippingKV], [],
{true, 99999999},
no_lookup,
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?LOOK_SLOTSIZE + 1,
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
Slot,
{o, "B1", "K5", null}),
?assertMatch({[], [], {no_lookup, Slot}, {o, "B1", "K5", null}}, R1).
Unit tets of lookup over-size issue A mistake meant resetting to lookup on a skipped key would cause issues if the skipped key ocurred under a no_lookup slot after the ?SLOT_SIZE had been reached. This caused the slot to switch to lookup, but beyond the maximum size
2017-03-11 00:03:55 +00:00
Fix empty slot issue
2017-03-11 12:41:30 +00:00
merge_tombstonelist_test() ->
Allow for segment-acceleration of folds Initially with basic tests. If the SlotIndex has been cached, we can now use the slot index as it is based on the Segment hash algortihm. This looks like it should lead to an order of magnitude improvement in querying for keys/clocks by segment ID. This also required a slight tweak to the penciller keyfolder. It now caches the next answer from the SSTiter, rather than restart the iterator. When the IMMiter has many more entries than the SSTiter (as the sSTiter is being filtered but not the IMMiter) this could lead to lots of repeated folding.
2017-10-31 23:28:35 +00:00
% Merge lists with nothing but tombstones
Fix empty slot issue
2017-03-11 12:41:30 +00:00
SkippingKV1 = {{o, "B1", "K9995", null}, {9995, tomb, 1234567, {}}},
SkippingKV2 = {{o, "B1", "K9996", null}, {9996, tomb, 1234567, {}}},
SkippingKV3 = {{o, "B1", "K9997", null}, {9997, tomb, 1234567, {}}},
SkippingKV4 = {{o, "B1", "K9998", null}, {9998, tomb, 1234567, {}}},
SkippingKV5 = {{o, "B1", "K9999", null}, {9999, tomb, 1234567, {}}},
R = merge_lists([SkippingKV1, SkippingKV3, SkippingKV5],
[SkippingKV2, SkippingKV4],
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{true, 9999999},
native),
Fix empty file again No special definition of empty required, as now an empty list when empty
2017-03-14 00:17:09 +00:00
?assertMatch({[], [], [], null}, R).
Fix empty slot issue
2017-03-11 12:41:30 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_test() ->
io:format(user, "~nIndexed list timing test:~n", []),
Five alternatives explored
2016-12-29 22:22:13 +00:00
N = 150,
KVL0 = lists:ukeysort(1, generate_randomkeys(1, N, 1, 4)),
Experiment with alternate slot size Improves fpr. Does this change anything in volume tests?
2017-10-24 17:58:33 +01:00
KVL1 = lists:sublist(KVL0, ?LOOK_SLOTSIZE),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
SW0 = os:timestamp(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{_PosBinIndex1, FullBin, _HL, _LK} =
generate_binary_slot(lookup, KVL1, native),
Experiment Want to experiemnt with different datatypes for the slot - maybe use a raw list but with a mini hashtree index like the CDB file
2016-12-29 14:11:05 +00:00
io:format(user,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
"Indexed list created slot in ~w microseconds of size ~w~n",
[timer:now_diff(os:timestamp(), SW0), byte_size(FullBin)]),
Five alternatives explored
2016-12-29 22:22:13 +00:00
{TestK1, TestV1} = lists:nth(20, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH1 = leveled_codec:segment_hash(TestK1),
Five alternatives explored
2016-12-29 22:22:13 +00:00
{TestK2, TestV2} = lists:nth(40, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH2 = leveled_codec:segment_hash(TestK2),
Five alternatives explored
2016-12-29 22:22:13 +00:00
{TestK3, TestV3} = lists:nth(60, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH3 = leveled_codec:segment_hash(TestK3),
Five alternatives explored
2016-12-29 22:22:13 +00:00
{TestK4, TestV4} = lists:nth(80, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH4 = leveled_codec:segment_hash(TestK4),
Five alternatives explored
2016-12-29 22:22:13 +00:00
{TestK5, TestV5} = lists:nth(100, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH5 = leveled_codec:segment_hash(TestK5),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin, TestK1, MH1, {TestK1, TestV1}),
test_binary_slot(FullBin, TestK2, MH2, {TestK2, TestV2}),
test_binary_slot(FullBin, TestK3, MH3, {TestK3, TestV3}),
test_binary_slot(FullBin, TestK4, MH4, {TestK4, TestV4}),
test_binary_slot(FullBin, TestK5, MH5, {TestK5, TestV5}).
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_mixedkeys_test() ->
KVL0 = lists:ukeysort(1, generate_randomkeys(1, 50, 1, 4)),
KVL1 = lists:sublist(KVL0, 33),
Keys = lists:ukeysort(1, generate_indexkeys(60) ++ KVL1),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{_PosBinIndex1, FullBin, _HL, _LK} =
generate_binary_slot(lookup, Keys, native),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{TestK1, TestV1} = lists:nth(4, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH1 = leveled_codec:segment_hash(TestK1),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{TestK2, TestV2} = lists:nth(8, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH2 = leveled_codec:segment_hash(TestK2),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{TestK3, TestV3} = lists:nth(12, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH3 = leveled_codec:segment_hash(TestK3),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{TestK4, TestV4} = lists:nth(16, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH4 = leveled_codec:segment_hash(TestK4),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{TestK5, TestV5} = lists:nth(20, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH5 = leveled_codec:segment_hash(TestK5),
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin, TestK1, MH1, {TestK1, TestV1}),
test_binary_slot(FullBin, TestK2, MH2, {TestK2, TestV2}),
test_binary_slot(FullBin, TestK3, MH3, {TestK3, TestV3}),
test_binary_slot(FullBin, TestK4, MH4, {TestK4, TestV4}),
test_binary_slot(FullBin, TestK5, MH5, {TestK5, TestV5}).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
indexed_list_mixedkeys2_test() ->
KVL0 = lists:ukeysort(1, generate_randomkeys(1, 50, 1, 4)),
KVL1 = lists:sublist(KVL0, 33),
IdxKeys1 = lists:ukeysort(1, generate_indexkeys(30)),
IdxKeys2 = lists:ukeysort(1, generate_indexkeys(30)),
% this isn't actually ordered correctly
Keys = IdxKeys1 ++ KVL1 ++ IdxKeys2,
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{_PosBinIndex1, FullBin, _HL, _LK} =
generate_binary_slot(lookup, Keys, native),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
lists:foreach(fun({K, V}) ->
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH = leveled_codec:segment_hash(K),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
test_binary_slot(FullBin, K, MH, {K, V})
end,
KVL1).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_allindexkeys_test() ->
Experiment with alternate slot size Improves fpr. Does this change anything in volume tests?
2017-10-24 17:58:33 +01:00
Keys = lists:sublist(lists:ukeysort(1, generate_indexkeys(150)),
?LOOK_SLOTSIZE),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{PosBinIndex1, FullBin, _HL, _LK} =
generate_binary_slot(lookup, Keys, native),
Experiment with alternate slot size Improves fpr. Does this change anything in volume tests?
2017-10-24 17:58:33 +01:00
EmptySlotSize = ?LOOK_SLOTSIZE - 1,
?assertMatch(<<_BL:24/binary, EmptySlotSize:8/integer>>, PosBinIndex1),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
% SW = os:timestamp(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
BinToList = binaryslot_tolist(FullBin, native),
Five alternatives explored
2016-12-29 22:22:13 +00:00
% io:format(user,
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
% "Indexed list flattened in ~w microseconds ~n",
% [timer:now_diff(os:timestamp(), SW)]),
?assertMatch(Keys, BinToList),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
?assertMatch(Keys, binaryslot_trimmedlist(FullBin, all, all, native)).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
indexed_list_allindexkeys_nolookup_test() ->
Keys = lists:sublist(lists:ukeysort(1, generate_indexkeys(1000)),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?NOLOOK_SLOTSIZE),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{PosBinIndex1, FullBin, _HL, _LK} =
generate_binary_slot(no_lookup, Keys, native),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
?assertMatch(<<_BL:24/binary, 127:8/integer>>, PosBinIndex1),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
% SW = os:timestamp(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
BinToList = binaryslot_tolist(FullBin, native),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
% io:format(user,
% "Indexed list flattened in ~w microseconds ~n",
% [timer:now_diff(os:timestamp(), SW)]),
?assertMatch(Keys, BinToList),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
?assertMatch(Keys, binaryslot_trimmedlist(FullBin, all, all, native)).
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
indexed_list_allindexkeys_trimmed_test() ->
Experiment with alternate slot size Improves fpr. Does this change anything in volume tests?
2017-10-24 17:58:33 +01:00
Keys = lists:sublist(lists:ukeysort(1, generate_indexkeys(150)),
?LOOK_SLOTSIZE),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{PosBinIndex1, FullBin, _HL, _LK} =
generate_binary_slot(lookup, Keys, native),
Experiment with alternate slot size Improves fpr. Does this change anything in volume tests?
2017-10-24 17:58:33 +01:00
EmptySlotSize = ?LOOK_SLOTSIZE - 1,
?assertMatch(<<_BL:24/binary, EmptySlotSize:8/integer>>, PosBinIndex1),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(Keys, binaryslot_trimmedlist(FullBin,
{i,
"Bucket",
{"t1_int", 0},
null},
{i,
"Bucket",
{"t1_int", 99999},
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
null},
native)),
Ongoing timing tests
2016-12-29 14:14:09 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{SK1, _} = lists:nth(10, Keys),
{EK1, _} = lists:nth(100, Keys),
R1 = lists:sublist(Keys, 10, 91),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
O1 = binaryslot_trimmedlist(FullBin, SK1, EK1, native),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(91, length(O1)),
?assertMatch(R1, O1),
{SK2, _} = lists:nth(10, Keys),
{EK2, _} = lists:nth(20, Keys),
R2 = lists:sublist(Keys, 10, 11),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
O2 = binaryslot_trimmedlist(FullBin, SK2, EK2, native),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(11, length(O2)),
?assertMatch(R2, O2),
Experiment with alternate slot size Improves fpr. Does this change anything in volume tests?
2017-10-24 17:58:33 +01:00
{SK3, _} = lists:nth(?LOOK_SLOTSIZE - 1, Keys),
{EK3, _} = lists:nth(?LOOK_SLOTSIZE, Keys),
R3 = lists:sublist(Keys, ?LOOK_SLOTSIZE - 1, 2),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
O3 = binaryslot_trimmedlist(FullBin, SK3, EK3, native),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(2, length(O3)),
?assertMatch(R3, O3).
indexed_list_mixedkeys_bitflip_test() ->
KVL0 = lists:ukeysort(1, generate_randomkeys(1, 50, 1, 4)),
KVL1 = lists:sublist(KVL0, 33),
Keys = lists:ukeysort(1, generate_indexkeys(60) ++ KVL1),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{_PosBinIndex1, FullBin, _HL, LK} =
generate_binary_slot(lookup, Keys, native),
Don't double-loop on slots Previous version built a list of slots, then iterated over it to build a list of binaries. This converts the slot to a binary before building the list
2017-03-13 23:51:48 +00:00
?assertMatch(LK, element(1, lists:last(Keys))),
Test coverage ... and column width formatting
2017-11-01 15:11:14 +00:00
FullBin0 = flip_byte(FullBin),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{TestK1, _TestV1} = lists:nth(20, KVL1),
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
MH1 = leveled_codec:segment_hash(TestK1),
Ongoing timing tests
2016-12-29 14:14:09 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin0, TestK1, MH1, not_present),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
ToList = binaryslot_tolist(FullBin0, native),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch([], ToList),
Ongoing timing tests
2016-12-29 14:14:09 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
{SK1, _} = lists:nth(10, Keys),
{EK1, _} = lists:nth(50, Keys),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
O1 = binaryslot_trimmedlist(FullBin0, SK1, EK1, native),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(0, length(O1)),
?assertMatch([], O1).
Five alternatives explored
2016-12-29 22:22:13 +00:00
Test coverage ... and column width formatting
2017-11-01 15:11:14 +00:00
flip_byte(Binary) ->
L = byte_size(Binary),
Amend SST Timing Capture Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
Byte1 = leveled_rand:uniform(L) - 1,
Test coverage ... and column width formatting
2017-11-01 15:11:14 +00:00
<<PreB1:Byte1/binary, A:8/integer, PostByte1/binary>> = Binary,
case A of
0 ->
<<PreB1:Byte1/binary, 255:8/integer, PostByte1/binary>>;
_ ->
<<PreB1:Byte1/binary, 0:8/integer, PostByte1/binary>>
end.
Five alternatives explored
2016-12-29 22:22:13 +00:00
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
test_binary_slot(FullBin, Key, Hash, ExpectedValue) ->
% SW = os:timestamp(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{ReturnedValue, _BLs, _Idx} = binaryslot_get(FullBin, Key, Hash, native),
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
?assertMatch(ExpectedValue, ReturnedValue).
% io:format(user, "Fetch success in ~w microseconds ~n",
% [timer:now_diff(os:timestamp(), SW)]).
Five alternatives explored
2016-12-29 22:22:13 +00:00
Experiment Want to experiemnt with different datatypes for the slot - maybe use a raw list but with a mini hashtree index like the CDB file
2016-12-29 14:11:05 +00:00
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
merge_test() ->
N = 3000,
KVL1 = lists:ukeysort(1, generate_randomkeys(N + 1, N, 1, 20)),
KVL2 = lists:ukeysort(1, generate_randomkeys(1, N, 1, 20)),
KVL3 = lists:ukeymerge(1, KVL1, KVL2),
SW0 = os:timestamp(),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{ok, P1, {FK1, LK1}} =
sst_new("../test/", "level1_src", 1, KVL1, 6000, native),
{ok, P2, {FK2, LK2}} =
sst_new("../test/", "level2_src", 2, KVL2, 3000, native),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
ExpFK1 = element(1, lists:nth(1, KVL1)),
ExpLK1 = element(1, lists:last(KVL1)),
ExpFK2 = element(1, lists:nth(1, KVL2)),
ExpLK2 = element(1, lists:last(KVL2)),
?assertMatch(ExpFK1, FK1),
?assertMatch(ExpFK2, FK2),
?assertMatch(ExpLK1, LK1),
?assertMatch(ExpLK2, LK2),
Change to use manifest_entry not straight PID in unit test
2017-01-17 10:14:40 +00:00
ML1 = [{next, #manifest_entry{owner = P1}, FK1}],
ML2 = [{next, #manifest_entry{owner = P2}, FK2}],
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
NewR =
sst_new("../test/", "level2_merge", ML1, ML2, false, 2, N * 2, native),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
{ok, P3, {{Rem1, Rem2}, FK3, LK3}} = NewR,
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
?assertMatch([], Rem1),
?assertMatch([], Rem2),
?assertMatch(true, FK3 == min(FK1, FK2)),
Draft of branch to condense range_only keys
2017-03-10 20:43:37 +00:00
io:format("LK1 ~w LK2 ~w LK3 ~w~n", [LK1, LK2, LK3]),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
?assertMatch(true, LK3 == max(LK1, LK2)),
io:format(user,
"Created and merged two files of size ~w in ~w microseconds~n",
[N, timer:now_diff(os:timestamp(), SW0)]),
SW1 = os:timestamp(),
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(P3, K))
end,
KVL3),
io:format(user,
"Checked presence of all ~w objects in ~w microseconds~n",
[length(KVL3), timer:now_diff(os:timestamp(), SW1)]),
ok = sst_close(P1),
ok = sst_close(P2),
ok = sst_close(P3),
ok = file:delete("../test/level1_src.sst"),
ok = file:delete("../test/level2_src.sst"),
ok = file:delete("../test/level2_merge.sst").
Share final timing test
2016-12-23 23:30:15 +00:00
Build the table summary The table summary will be a skiplist, and this and the slot binary will be CRC checked
2016-12-24 01:23:40 +00:00
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
simple_persisted_range_test() ->
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{RP, Filename} = {"../test/", "simple_test"},
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList0 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 16, 1, 20),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
KVList1 = lists:ukeysort(1, KVList0),
[{FirstKey, _FV}|_Rest] = KVList1,
{LastKey, _LV} = lists:last(KVList1),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{ok, Pid, {FirstKey, LastKey}} =
sst_new(RP, Filename, 1, KVList1, length(KVList1), native),
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
{o, B, K, null} = LastKey,
SK1 = {o, B, K, 0},
EK1 = {o, B, K, 1},
FetchListA1 = sst_getkvrange(Pid, SK1, EK1, 1),
?assertMatch([], FetchListA1),
SK2 = element(1, lists:nth(127, KVList1)),
SK3 = element(1, lists:nth(128, KVList1)),
SK4 = element(1, lists:nth(129, KVList1)),
SK5 = element(1, lists:nth(130, KVList1)),
EK2 = element(1, lists:nth(255, KVList1)),
EK3 = element(1, lists:nth(256, KVList1)),
EK4 = element(1, lists:nth(257, KVList1)),
EK5 = element(1, lists:nth(258, KVList1)),
TestFun =
fun({SK, EK}) ->
FetchList = sst_getkvrange(Pid, SK, EK, 4),
?assertMatch(SK, element(1, lists:nth(1, FetchList))),
?assertMatch(EK, element(1, lists:last(FetchList)))
end,
TL2 = lists:map(fun(EK) -> {SK2, EK} end, [EK2, EK3, EK4, EK5]),
TL3 = lists:map(fun(EK) -> {SK3, EK} end, [EK2, EK3, EK4, EK5]),
TL4 = lists:map(fun(EK) -> {SK4, EK} end, [EK2, EK3, EK4, EK5]),
TL5 = lists:map(fun(EK) -> {SK5, EK} end, [EK2, EK3, EK4, EK5]),
lists:foreach(TestFun, TL2 ++ TL3 ++ TL4 ++ TL5).
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
additional_range_test() ->
% Test fetching ranges that fall into odd situations with regards to the
% summayr index
% - ranges which fall between entries in summary
% - ranges which go beyond the end of the range of the sst
% - ranges which match to an end key in the summary index
IK1 = lists:foldl(fun(X, Acc) ->
Acc ++ generate_indexkey(X, X)
end,
[],
lists:seq(1, ?NOLOOK_SLOTSIZE)),
Gap = 2,
IK2 = lists:foldl(fun(X, Acc) ->
Acc ++ generate_indexkey(X, X)
end,
[],
lists:seq(?NOLOOK_SLOTSIZE + Gap + 1,
2 * ?NOLOOK_SLOTSIZE + Gap)),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{ok, P1, {{Rem1, Rem2}, SK, EK}} =
sst_new("../test/", "range1_src", IK1, IK2, false, 1, 9999, native),
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
?assertMatch([], Rem1),
?assertMatch([], Rem2),
?assertMatch(SK, element(1, lists:nth(1, IK1))),
?assertMatch(EK, element(1, lists:last(IK2))),
% Basic test - checking scanwidth
R1 = sst_getkvrange(P1, SK, EK, 1),
?assertMatch(?NOLOOK_SLOTSIZE + 1, length(R1)),
QR1 = lists:sublist(R1, ?NOLOOK_SLOTSIZE),
?assertMatch(IK1, QR1),
R2 = sst_getkvrange(P1, SK, EK, 2),
?assertMatch(?NOLOOK_SLOTSIZE * 2, length(R2)),
QR2 = lists:sublist(R2, ?NOLOOK_SLOTSIZE),
QR3 = lists:sublist(R2, ?NOLOOK_SLOTSIZE + 1, 2 * ?NOLOOK_SLOTSIZE),
?assertMatch(IK1, QR2),
?assertMatch(IK2, QR3),
% Testing the gap
[GapSKV] = generate_indexkey(?NOLOOK_SLOTSIZE + 1, ?NOLOOK_SLOTSIZE + 1),
[GapEKV] = generate_indexkey(?NOLOOK_SLOTSIZE + 2, ?NOLOOK_SLOTSIZE + 2),
R3 = sst_getkvrange(P1, element(1, GapSKV), element(1, GapEKV), 1),
?assertMatch([], R3),
% Testing beyond the range
[PastEKV] = generate_indexkey(2 * ?NOLOOK_SLOTSIZE + Gap + 1,
2 * ?NOLOOK_SLOTSIZE + Gap + 1),
R4 = sst_getkvrange(P1, element(1, GapSKV), element(1, PastEKV), 2),
?assertMatch(IK2, R4),
Add unit test for more than one slot
2017-03-15 16:40:43 +00:00
R5 = sst_getkvrange(P1, SK, element(1, PastEKV), 2),
IKAll = IK1 ++ IK2,
?assertMatch(IKAll, R5),
RTrim RTrim only worked in special case of key matching, that would never occur in real world range query. RTrim should really check for key passing. Returning empty list should not be possible - unless the query is outside of the range entirely (and such a query should never go to this SST).
2017-03-16 08:37:36 +00:00
[MidREKV] = generate_indexkey(?NOLOOK_SLOTSIZE + Gap + 2,
?NOLOOK_SLOTSIZE + Gap + 2),
io:format(user, "Mid second range to past range test~n", []),
R6 = sst_getkvrange(P1, element(1, MidREKV), element(1, PastEKV), 2),
Exp6 = lists:sublist(IK2, 2, length(IK2)),
?assertMatch(Exp6, R6),
Additional unit tests for SST range fetches Resolve some of coverage issues
2017-03-15 11:27:46 +00:00
% Testing at a slot end
Slot1EK = element(1, lists:last(IK1)),
RTrim RTrim only worked in special case of key matching, that would never occur in real world range query. RTrim should really check for key passing. Returning empty list should not be possible - unless the query is outside of the range entirely (and such a query should never go to this SST).
2017-03-16 08:37:36 +00:00
R7 = sst_getkvrange(P1, SK, Slot1EK, 2),
?assertMatch(IK1, R7).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
Check query out of range It doesn't work - so protecting against it in fetch_range is pointless, will blow up in lookup_slots
2017-03-16 08:43:18 +00:00
% Testing beyond end (should never happen if manifest behaves)
% Test blows up anyway
% R8 = sst_getkvrange(P1, element(1, PastEKV), element(1, PastEKV), 2),
% ?assertMatch([], R8).
Additional unit test Need to test scenario where the key list the SST file created from is an exact multiple of the slot size
2017-03-19 23:42:24 +00:00
simple_persisted_slotsize_test() ->
{RP, Filename} = {"../test/", "simple_slotsize_test"},
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList0 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 2, 1, 20),
KVList1 = lists:sublist(lists:ukeysort(1, KVList0),
?LOOK_SLOTSIZE),
Additional unit test Need to test scenario where the key list the SST file created from is an exact multiple of the slot size
2017-03-19 23:42:24 +00:00
[{FirstKey, _FV}|_Rest] = KVList1,
{LastKey, _LV} = lists:last(KVList1),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{ok, Pid, {FirstKey, LastKey}} =
sst_new(RP, Filename, 1, KVList1, length(KVList1), native),
Additional unit test Need to test scenario where the key list the SST file created from is an exact multiple of the slot size
2017-03-19 23:42:24 +00:00
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(Pid, K))
end,
KVList1),
ok = sst_close(Pid),
ok = file:delete(filename:join(RP, Filename ++ ".sst")).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
Some end-to-end testing
2016-12-24 17:48:31 +00:00
simple_persisted_test() ->
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
{RP, Filename} = {"../test/", "simple_test"},
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList0 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 32, 1, 20),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
KVList1 = lists:ukeysort(1, KVList0),
[{FirstKey, _FV}|_Rest] = KVList1,
{LastKey, _LV} = lists:last(KVList1),
Make compression algorithm an option Compression can be switched between LZ4 and zlib (native). The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
{ok, Pid, {FirstKey, LastKey}} =
sst_new(RP, Filename, 1, KVList1, length(KVList1), native),
Fix broken SST cache
2017-01-03 13:03:59 +00:00
SW0 = os:timestamp(),
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(Pid, K))
end,
KVList1),
io:format(user,
"Checking for ~w keys (once) in file with cache hit took ~w "
++ "microseconds~n",
[length(KVList1), timer:now_diff(os:timestamp(), SW0)]),
Test for missing keys
2016-12-24 17:59:07 +00:00
SW1 = os:timestamp(),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
lists:foreach(fun({K, V}) ->
?assertMatch({K, V}, sst_get(Pid, K)),
?assertMatch({K, V}, sst_get(Pid, K))
end,
KVList1),
io:format(user,
"Checking for ~w keys (twice) in file with cache hit took ~w "
++ "microseconds~n",
Test for missing keys
2016-12-24 17:59:07 +00:00
[length(KVList1), timer:now_diff(os:timestamp(), SW1)]),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
ok = sst_printtimings(Pid),
Change to 5 blocks in SST Slot Change to 5 blocks is intended to make the blocks in lookup slots fractionally smaller, but more importantly to introduce a middle block that cna be opened in a binary-split style fashion to reduce the number of blocks that need to be opened for range queries. Worst case for full slots is 3 blocks now not 4.
2017-03-21 16:54:23 +00:00
KVList2 = generate_randomkeys(1, ?LOOK_SLOTSIZE * 32, 1, 20),
Test for missing keys
2016-12-24 17:59:07 +00:00
MapFun =
fun({K, V}, Acc) ->
In = lists:keymember(K, 1, KVList1),
case {K > FirstKey, LastKey > K, In} of
{true, true, false} ->
Change hash algorithm for penciller Switch from magic hash to md5 - to hopefully remove the need for some of the artificial jumps required to get expected fall positive ratios. Also split the hash into two 16-bit integers. We assume that SegmentID (from the perspective of AAE merkle/tictac trees) will always be at least 16 bits. the idea is that hashes should be used in blooms and indexes such that some advantage can be gained from just knowing the segmentID - in particular when folding over all the keys in a bucket. Performance testing has been difficult so far - I think due to “cloud” mysteries.
2017-10-20 23:04:29 +01:00
[{K, leveled_codec:segment_hash(K), V}|Acc];
Test for missing keys
2016-12-24 17:59:07 +00:00
_ ->
Acc
end
end,
KVList3 = lists:foldl(MapFun, [], KVList2),
SW2 = os:timestamp(),
Add hash to missing key test
2016-12-24 18:03:34 +00:00
lists:foreach(fun({K, H, _V}) ->
?assertMatch(not_present, sst_get(Pid, K, H))
Test for missing keys
2016-12-24 17:59:07 +00:00
end,
KVList3),
io:format(user,
"Checking for ~w missing keys took ~w microseconds~n",
[length(KVList3), timer:now_diff(os:timestamp(), SW2)]),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
ok = sst_printtimings(Pid),
FetchList1 = sst_getkvrange(Pid, all, all, 2),
FoldFun = fun(X, Acc) ->
case X of
{pointer, P, S, SK, EK} ->
Switched to indexed blocks as slot implementation Prior to this refactor, the slot and been made up of four blocks with an external binary index. Although the form of the index has changed again, micro-benchmarking once again showed that this was a relatively efficient mechanism.
2017-01-02 10:47:04 +00:00
Acc ++ sst_getslots(P, [{pointer, P, S, SK, EK}]);
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
_ ->
Acc ++ [X]
end end,
FetchedList1 = lists:foldl(FoldFun, [], FetchList1),
?assertMatch(KVList1, FetchedList1),
{TenthKey, _v10} = lists:nth(10, KVList1),
{Three000Key, _v300} = lists:nth(300, KVList1),
SubKVList1 = lists:sublist(KVList1, 10, 291),
SubKVList1L = length(SubKVList1),
FetchList2 = sst_getkvrange(Pid, TenthKey, Three000Key, 2),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
?assertMatch(pointer, element(1, lists:last(FetchList2))),
Refactoring of skiplist ranges and support for sst ranges the Skiplist range code was needlessly complicated. It may be faster than the new code, but the complexity delta cannot be support for such a small change. This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
FetchedList2 = lists:foldl(FoldFun, [], FetchList2),
?assertMatch(SubKVList1L, length(FetchedList2)),
?assertMatch(SubKVList1, FetchedList2),
Add basic merge support No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
{Eight000Key, _v800} = lists:nth(800, KVList1),
SubKVListA1 = lists:sublist(KVList1, 10, 791),
SubKVListA1L = length(SubKVListA1),
FetchListA2 = sst_getkvrange(Pid, TenthKey, Eight000Key, 2),
?assertMatch(pointer, element(1, lists:last(FetchListA2))),
FetchedListA2 = lists:foldl(FoldFun, [], FetchListA2),
?assertMatch(SubKVListA1L, length(FetchedListA2)),
?assertMatch(SubKVListA1, FetchedListA2),
FetchListB2 = sst_getkvrange(Pid, TenthKey, Eight000Key, 4),
?assertMatch(pointer, element(1, lists:last(FetchListB2))),
FetchedListB2 = lists:foldl(FoldFun, [], FetchListB2),
?assertMatch(SubKVListA1L, length(FetchedListB2)),
?assertMatch(SubKVListA1, FetchedListB2),
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
FetchListB3 = sst_getkvrange(Pid,
Eight000Key,
{o, null, null, null},
4),
FetchedListB3 = lists:foldl(FoldFun, [], FetchListB3),
SubKVListA3 = lists:nthtail(800 - 1, KVList1),
SubKVListA3L = length(SubKVListA3),
io:format("Length expected ~w~n", [SubKVListA3L]),
?assertMatch(SubKVListA3L, length(FetchedListB3)),
?assertMatch(SubKVListA3, FetchedListB3),
Fixed issues Two issues - when the key range falls in-between two marks in the summary, we didn't pick up any mark. then when trimming both right and left, the left trim was being discarded.
2016-12-29 04:37:49 +00:00
io:format("Eight hundredth key ~w~n", [Eight000Key]),
FetchListB4 = sst_getkvrange(Pid,
Eight000Key,
Eight000Key,
4),
FetchedListB4 = lists:foldl(FoldFun, [], FetchListB4),
?assertMatch([{Eight000Key, _v800}], FetchedListB4),
Some end-to-end testing
2016-12-24 17:48:31 +00:00
ok = sst_close(Pid),
Change SST reference to split filename The manifest and the logs are bloated by having the full file path for every filename in there - given the root path is constant. Could also cause issues if the mount point is ever changed.
2017-03-09 21:23:09 +00:00
ok = file:delete(filename:join(RP, Filename ++ ".sst")).
Some end-to-end testing
2016-12-24 17:48:31 +00:00
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
key_dominates_test() ->
KV1 = {{o, "Bucket", "Key1", null}, {5, {active, infinity}, 0, []}},
KV2 = {{o, "Bucket", "Key3", null}, {6, {active, infinity}, 0, []}},
KV3 = {{o, "Bucket", "Key2", null}, {3, {active, infinity}, 0, []}},
KV4 = {{o, "Bucket", "Key4", null}, {7, {active, infinity}, 0, []}},
KV5 = {{o, "Bucket", "Key1", null}, {4, {active, infinity}, 0, []}},
KV6 = {{o, "Bucket", "Key1", null}, {99, {tomb, 999}, 0, []}},
KV7 = {{o, "Bucket", "Key1", null}, {99, tomb, 0, []}},
KL1 = [KV1, KV2],
KL2 = [KV3, KV4],
?assertMatch({{next_key, KV1}, [KV2], KL2},
key_dominates(KL1, KL2, {undefined, 1})),
?assertMatch({{next_key, KV1}, KL2, [KV2]},
key_dominates(KL2, KL1, {undefined, 1})),
?assertMatch({skipped_key, KL2, KL1},
key_dominates([KV5|KL2], KL1, {undefined, 1})),
?assertMatch({{next_key, KV1}, [KV2], []},
key_dominates(KL1, [], {undefined, 1})),
?assertMatch({skipped_key, [KV6|KL2], [KV2]},
key_dominates([KV6|KL2], KL1, {undefined, 1})),
?assertMatch({{next_key, KV6}, KL2, [KV2]},
key_dominates([KV6|KL2], [KV2], {undefined, 1})),
?assertMatch({skipped_key, [KV6|KL2], [KV2]},
key_dominates([KV6|KL2], KL1, {true, 1})),
?assertMatch({skipped_key, [KV6|KL2], [KV2]},
key_dominates([KV6|KL2], KL1, {true, 1000})),
?assertMatch({{next_key, KV6}, KL2, [KV2]},
key_dominates([KV6|KL2], [KV2], {true, 1})),
?assertMatch({skipped_key, KL2, [KV2]},
key_dominates([KV6|KL2], [KV2], {true, 1000})),
?assertMatch({skipped_key, [], []},
key_dominates([KV6], [], {true, 1000})),
?assertMatch({skipped_key, [], []},
key_dominates([], [KV6], {true, 1000})),
?assertMatch({{next_key, KV6}, [], []},
key_dominates([KV6], [], {true, 1})),
?assertMatch({{next_key, KV6}, [], []},
key_dominates([], [KV6], {true, 1})),
?assertMatch({skipped_key, [], []},
key_dominates([KV7], [], {true, 1})),
?assertMatch({skipped_key, [], []},
key_dominates([], [KV7], {true, 1})),
?assertMatch({skipped_key, [KV7|KL2], [KV2]},
key_dominates([KV7|KL2], KL1, {undefined, 1})),
?assertMatch({{next_key, KV7}, KL2, [KV2]},
key_dominates([KV7|KL2], [KV2], {undefined, 1})),
?assertMatch({skipped_key, [KV7|KL2], [KV2]},
key_dominates([KV7|KL2], KL1, {true, 1})),
?assertMatch({skipped_key, KL2, [KV2]},
key_dominates([KV7|KL2], [KV2], {true, 1})).
Test coverage Add some furthe runit tests to improve test coverage
2017-01-04 21:36:59 +00:00
nonsense_coverage_test() ->
{ok, Pid} = gen_fsm:start(?MODULE, [], []),
ok = gen_fsm:send_all_state_event(Pid, nonsense),
?assertMatch({next_state, reader, #state{}}, handle_info(nonsense,
reader,
#state{})),
?assertMatch({ok, reader, #state{}}, code_change(nonsense,
reader,
#state{},
Coverage checks
2017-02-26 21:37:47 +00:00
nonsense)),
?assertMatch({reply, undefined, reader, #state{}},
Test coverage ... and column width formatting
2017-11-01 15:11:14 +00:00
handle_sync_event("hello", self(), reader, #state{})),
SampleBin = <<0:128/integer>>,
FlippedBin = flip_byte(SampleBin),
?assertMatch(false, FlippedBin == SampleBin).
Removed SFT Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
Add tests for different tree sizes Note that accelerating segment_list queries will not work for tree sizes smaller than small. How to flag this up? Should smaller tree sizes just be removed from leveled_tictac?
2017-11-01 11:51:51 +00:00
hashmatching_bytreesize_test() ->
B = <<"Bucket">>,
V = leveled_codec:riak_metadata_to_binary(term_to_binary([{"actor1", 1}]),
<<1:32/integer,
0:32/integer,
0:32/integer>>),
GenKeyFun =
fun(X) ->
LK =
{?RIAK_TAG,
B,
list_to_binary("Key" ++ integer_to_list(X)),
null},
LKV = leveled_codec:generate_ledgerkv(LK,
X,
V,
byte_size(V),
{active, infinity}),
{_Bucket, _Key, MetaValue, _Hashes, _LastMods} = LKV,
{LK, MetaValue}
end,
KVL = lists:map(GenKeyFun, lists:seq(1, 128)),
{PosBinIndex1, _FullBin, _HL, _LK} =
Merge branch 'master' into mas-aae-segementfoldplus
2017-11-07 11:22:56 +00:00
generate_binary_slot(lookup, KVL, native),
Add tests for different tree sizes Note that accelerating segment_list queries will not work for tree sizes smaller than small. How to flag this up? Should smaller tree sizes just be removed from leveled_tictac?
2017-11-01 11:51:51 +00:00
check_segment_match(PosBinIndex1, KVL, small),
check_segment_match(PosBinIndex1, KVL, medium).
check_segment_match(PosBinIndex1, KVL, TreeSize) ->
CheckFun =
fun({{_T, B, K, null}, _V}) ->
Seg =
leveled_tictac:get_segment(
leveled_tictac:keyto_segment32(<<B/binary, K/binary>>),
TreeSize),
SegList0 = tune_seglist([Seg]),
PosList = find_pos(PosBinIndex1, SegList0, [], 0),
?assertMatch(true, length(PosList) >= 1)
end,
lists:foreach(CheckFun, KVL).
Cleanup dialyzer errrors in leveled_sst
2017-07-31 19:30:29 +02:00
-endif.
Reference in a new issue Copy permalink