umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
leveled/src/leveled_inker.erl

946 lines
38 KiB
Erlang
Raw Normal View History

Re-naming Naming things is hard. This change renames things based on the Bookie/Inker/Penciller terminology
2016-08-02 13:44:48 +01:00
%% -------- Inker ---------
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%%
%% The Inker is responsible for managing access and updates to the Journal.
%%
%% The Inker maintains a manifest of what files make up the Journal, and which
%% file is the current append-only nursery log to accept new PUTs into the
%% Journal. The Inker also marshals GET requests to the appropriate database
%% file within the Journal (routed by sequence number). The Inker is also
%% responsible for scheduling compaction work to be carried out by the Inker's
%% clerk.
%%
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% -------- Journal Files ---------
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%%
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% The Journal is a series of files originally named as <SQN>_<GUID>
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%% where the sequence number is the first object sequence number (key) within
%% the given database file. The files will be named *.cdb at the point they
%% have been made immutable (through a rename operation). Prior to this, they
%% will originally start out as a *.pnd file.
%%
%% At some stage in the future compacted versions of old journal cdb files may
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% be produced. These files will be named <SQN>-<NewGUID>.cdb, and once
%% the manifest is updated the original <SQN>_<GUID>.cdb (or
%% <SQN>_<previousGUID>.cdb) files they replace will be erased.
Re-naming Naming things is hard. This change renames things based on the Bookie/Inker/Penciller terminology
2016-08-02 13:44:48 +01:00
%%
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% The current Journal is made up of a set of files referenced in the manifest.
%% No PUTs are made to files which are not in the manifest.
Re-naming Naming things is hard. This change renames things based on the Bookie/Inker/Penciller terminology
2016-08-02 13:44:48 +01:00
%%
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%% The Journal is ordered by sequence number from front to back both within
%% and across files.
%%
%% On startup the Inker should open the manifest with the highest sequence
%% number, and this will contain the list of filenames that make up the
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% non-recent part of the Journal. All the filenames should then be opened.
%% How they are opened depends on the file extension:
%%
%% - If the file extension is *.cdb the file is opened read only
%% - If the file extension is *.pnd and the file is not the most recent in the
%% manifest, then the file should be completed bfore being opened read-only
%% - If the file extension is *.pnd the file is opened for writing
%%
%% -------- Manifest Files ---------
%%
%% The manifest is just saved as a straight term_to_binary blob, with a
%% filename ordered by the Manifest SQN. The Manifest is first saved with a
%% *.pnd extension, and then renamed to one with a *.man extension.
%%
%% On startup the *.man manifest file with the highest manifest sequence
%% number should be used.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%%
%% -------- Objects ---------
%%
%% From the perspective of the Inker, objects to store are made up of:
Move towards Inker Key Types The current mechanism of re-loading data from the Journla to the Ledger from any potential SQN is not safe when combined with Journla compaction. This commit doesn't resolve thes eproblems, but starts the groundwork for resolving by introducing Inker Key Types. These types would differentiate between objects which are standard Key/Value pairs, objects which are tombstones for keys, and objects whihc represent Key Changes only. The idea is that there will be flexible reload strategies based on object tags - retain (retain a key change object when compacting a standard object) - recalc (allow key changes to be recalculated from objects and ledger state when loading the Ledger from the journal - recover (allow for the potential loss of data on loss within the perisste dpart of the ledger, potentially due to recovery through externla anti-entropy operations).
2016-10-25 01:57:12 +01:00
%% - An Inker Key formed from
%% - A sequence number (assigned by the Inker)
%% - An Inker key type (stnd, tomb or keyd)
%% - A Ledger Key (as an Erlang term)
%% - A value formed from
%% - An object (an Erlang term) which should be null for tomb types, and
%% maybe null for keyd types
%% - A set of Key Deltas associated with the change (which may be an
%% empty list )
%%
%% Note that only the Inker key type of stnd is directly fetchable, other
%% key types are to be found only in scans and so can be added without being
%% entered into the hashtree
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%%
%% -------- Compaction ---------
%%
%% Compaction is a process whereby an Inker's clerk will:
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% - Request a view of the current Inker manifest and a snaphot of the Ledger
Doc update and clerk<->penciller changes Reviewing code to update comments revealed a weakness in the sequence of events between penciller and clerk committing a manifest change wherby an ill-timed crash could lead to files being deleted without the manifest changing. A different, and safer pattern now used between theses two actors.
2016-10-09 22:33:45 +01:00
%% - Test all files within the Journal to find the approximate comapction
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% potential percentage (the volume of the Journal that has been replaced)
%% - Attempts to find the optimal "run" of files to compact
%% - Compacts those files in the run, by rolling over the files re-writing
%% to a new Journal if and only if the Key is still present in the Ledger (or
%% the sequence number of the Key is higher than the SQN of the snapshot)
%% - Requests the Inker update the manifest with the new changes
%% - Instructs the files to destroy themselves when they are next closed
Re-naming Naming things is hard. This change renames things based on the Bookie/Inker/Penciller terminology
2016-08-02 13:44:48 +01:00
%%
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
%% TODO: how to instruct the files to close is tbd
Re-naming Naming things is hard. This change renames things based on the Bookie/Inker/Penciller terminology
2016-08-02 13:44:48 +01:00
%%
Work on descriptions Add further descriptions of roles following name changes. Attempt to simplify manifest management in the Penciller by assuming there is only one Penciller's Clerk active - and so only one piece of work can be ongoing
2016-08-02 17:51:43 +01:00
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
Work on descriptions Add further descriptions of roles following name changes. Attempt to simplify manifest management in the Penciller by assuming there is only one Penciller's Clerk active - and so only one piece of work can be ongoing
2016-08-02 17:51:43 +01:00
-module(leveled_inker).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
-behaviour(gen_server).
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
-include("include/leveled.hrl").
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
-export([init/1,
handle_call/3,
handle_cast/2,
handle_info/2,
terminate/2,
code_change/3,
ink_start/1,
ink_put/4,
ink_get/3,
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
ink_fetch/3,
Refactor snapshot Better reuse snapshotting fucntions in the Bookie, and use it to support doing Inker clone checks
2016-10-31 17:26:28 +00:00
ink_keycheck/3,
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
ink_loadpcl/4,
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
ink_registersnapshot/2,
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
ink_confirmdelete/2,
Snapshot testing Work to test the checking of sequence numbers in snapshots as required by the inkers clerk to calculate the percentage of a file which is compactable
2016-09-26 10:55:08 +01:00
ink_compactjournal/3,
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ink_compactioncomplete/1,
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
ink_compactionpending/1,
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
ink_getmanifest/1,
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ink_updatemanifest/3,
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
ink_print_manifest/1,
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
ink_close/1,
Add destruction Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
ink_doom/1,
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
build_dummy_journal/0,
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
clean_testdir/1,
Inker Clerk - Further Testing Expanded the unit tetsing of the Inker Clerk actor. Still WIP
2016-09-28 11:41:56 +01:00
filepath/2,
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
filepath/3]).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
-include_lib("eunit/include/eunit.hrl").
-define(MANIFEST_FP, "journal_manifest").
-define(FILES_FP, "journal_files").
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
-define(COMPACT_FP, "post_compact").
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
-define(WASTE_FP, "waste").
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
-define(JOURNAL_FILEX, "cdb").
-define(PENDING_FILEX, "pnd").
Add slow offer support on write pressure When there is write pressure on the penciller and it returns to the bookie, the bookie will now punish the next PUT (and itself) with a pause. The longer the back-pressure state has been in place, the more frequent the pauses
2016-11-05 14:31:10 +00:00
-define(LOADING_PAUSE, 1000).
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
-define(LOADING_BATCH, 1000).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
-record(state, {manifest = [] :: list(),
manifest_sqn = 0 :: integer(),
journal_sqn = 0 :: integer(),
active_journaldb :: pid(),
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
pending_removals = [] :: list(),
Snapshot testing Work to test the checking of sequence numbers in snapshots as required by the inkers clerk to calculate the percentage of a file which is compactable
2016-09-26 10:55:08 +01:00
registered_snapshots = [] :: list(),
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
root_path :: string(),
Snapshot testing Work to test the checking of sequence numbers in snapshots as required by the inkers clerk to calculate the percentage of a file which is compactable
2016-09-26 10:55:08 +01:00
cdb_options :: #cdb_options{},
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
clerk :: pid(),
compaction_pending = false :: boolean(),
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
is_snapshot = false :: boolean(),
Trigger log points, deprecate lower PUT log points Only record PUT timings at the Bookie, and trigger them correctly on the right log point
2017-02-26 19:07:55 +00:00
source_inker :: pid()}).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%%%============================================================================
%%% API
%%%============================================================================
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
ink_start(InkerOpts) ->
gen_server:start(?MODULE, [InkerOpts], []).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
ink_put(Pid, PrimaryKey, Object, KeyChanges) ->
gen_server:call(Pid, {put, PrimaryKey, Object, KeyChanges}, infinity).
ink_get(Pid, PrimaryKey, SQN) ->
gen_server:call(Pid, {get, PrimaryKey, SQN}, infinity).
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
ink_fetch(Pid, PrimaryKey, SQN) ->
gen_server:call(Pid, {fetch, PrimaryKey, SQN}, infinity).
Scan over CDB file Make scanning over a CDB file generic rather than specific to read-in of active nursery log - open to be called as an external function to support other scanning behaviour.
2016-09-09 15:58:19 +01:00
Refactor snapshot Better reuse snapshotting fucntions in the Bookie, and use it to support doing Inker clone checks
2016-10-31 17:26:28 +00:00
ink_keycheck(Pid, PrimaryKey, SQN) ->
gen_server:call(Pid, {key_check, PrimaryKey, SQN}, infinity).
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
ink_registersnapshot(Pid, Requestor) ->
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
gen_server:call(Pid, {register_snapshot, Requestor}, infinity).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
ink_releasesnapshot(Pid, Snapshot) ->
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
gen_server:cast(Pid, {release_snapshot, Snapshot}).
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
ink_confirmdelete(Pid, ManSQN) ->
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
gen_server:call(Pid, {confirm_delete, ManSQN}).
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
Basic GET/PUT and rolling in Inker Add support to roll file on PUT in the inker
2016-09-06 17:17:31 +01:00
ink_close(Pid) ->
Quality review Minor test fix-up and quality changes
2016-10-30 22:06:44 +00:00
gen_server:call(Pid, close, infinity).
Basic GET/PUT and rolling in Inker Add support to roll file on PUT in the inker
2016-09-06 17:17:31 +01:00
Add destruction Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
ink_doom(Pid) ->
gen_server:call(Pid, doom, 60000).
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
ink_loadpcl(Pid, MinSQN, FilterFun, Penciller) ->
gen_server:call(Pid, {load_pcl, MinSQN, FilterFun, Penciller}, infinity).
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
ink_compactjournal(Pid, Bookie, Timeout) ->
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
CheckerInitiateFun = fun initiate_penciller_snapshot/1,
CheckerFilterFun = fun leveled_penciller:pcl_checksequencenumber/3,
gen_server:call(Pid,
{compact,
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
Bookie,
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
CheckerInitiateFun,
CheckerFilterFun,
Timeout},
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
infinity).
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
%% Allows the Checker to be overriden in test, use something other than a
%% penciller
ink_compactjournal(Pid, Checker, InitiateFun, FilterFun, Timeout) ->
gen_server:call(Pid,
{compact,
Checker,
InitiateFun,
FilterFun,
Timeout},
infinity).
Snapshot testing Work to test the checking of sequence numbers in snapshots as required by the inkers clerk to calculate the percentage of a file which is compactable
2016-09-26 10:55:08 +01:00
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ink_compactioncomplete(Pid) ->
gen_server:call(Pid, compaction_complete, infinity).
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
ink_compactionpending(Pid) ->
gen_server:call(Pid, compaction_pending, infinity).
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
ink_getmanifest(Pid) ->
gen_server:call(Pid, get_manifest, infinity).
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ink_updatemanifest(Pid, ManifestSnippet, DeletedFiles) ->
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
gen_server:call(Pid,
{update_manifest,
ManifestSnippet,
DeletedFiles},
infinity).
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
ink_print_manifest(Pid) ->
gen_server:call(Pid, print_manifest, infinity).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%%%============================================================================
%%% gen_server callbacks
%%%============================================================================
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
init([InkerOpts]) ->
Seed randomnes at Actor startup
2017-03-06 21:35:02 +00:00
SW = os:timestamp(),
random:seed(erlang:phash2(self()), element(2, SW), element(3, SW)),
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
case {InkerOpts#inker_options.root_path,
InkerOpts#inker_options.start_snapshot} of
{undefined, true} ->
SrcInker = InkerOpts#inker_options.source_inker,
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
{Manifest,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
ActiveJournalDB} = ink_registersnapshot(SrcInker, self()),
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
{ok, #state{manifest=Manifest,
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
active_journaldb=ActiveJournalDB,
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
source_inker=SrcInker,
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
is_snapshot=true}};
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
%% Need to do something about timeout
{_RootPath, false} ->
start_from_file(InkerOpts)
end.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
handle_call({put, Key, Object, KeyChanges}, _From, State) ->
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
case put_object(Key, Object, KeyChanges, State) of
Resolve test issues
2017-01-19 09:47:56 +00:00
{_, UpdState, ObjSize} ->
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{reply, {ok, UpdState#state.journal_sqn, ObjSize}, UpdState}
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
end;
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
handle_call({fetch, Key, SQN}, _From, State) ->
Remove binary_to_term when fetching don't need to binary_to_term key changes
2017-03-29 15:37:04 +01:00
case get_object(Key, SQN, State#state.manifest, true) of
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
{{SQN, Key}, {Value, _IndexSpecs}} ->
{reply, {ok, Value}, State};
Other ->
Log improvements Continuation of log review and conversion to using central log function. Fixup of convoluted shutdown process between Bookie, Inker and Inker's Clerk
2016-11-03 16:05:43 +00:00
leveled_log:log("I0001", [Key, SQN, Other]),
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
{reply, not_present, State}
end;
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
handle_call({get, Key, SQN}, _From, State) ->
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{reply, get_object(Key, SQN, State#state.manifest), State};
Refactor snapshot Better reuse snapshotting fucntions in the Bookie, and use it to support doing Inker clone checks
2016-10-31 17:26:28 +00:00
handle_call({key_check, Key, SQN}, _From, State) ->
{reply, key_check(Key, SQN, State#state.manifest), State};
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
handle_call({load_pcl, StartSQN, FilterFun, Penciller}, _From, State) ->
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
Manifest = lists:reverse(leveled_imanifest:to_list(State#state.manifest)),
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
Reply = load_from_sequence(StartSQN, FilterFun, Penciller, Manifest),
{reply, Reply, State};
Snapshot testing Work to test the checking of sequence numbers in snapshots as required by the inkers clerk to calculate the percentage of a file which is compactable
2016-09-26 10:55:08 +01:00
handle_call({register_snapshot, Requestor}, _From , State) ->
Rs = [{Requestor,
State#state.manifest_sqn}|State#state.registered_snapshots],
Log improvements Continuation of log review and conversion to using central log function. Fixup of convoluted shutdown process between Bookie, Inker and Inker's Clerk
2016-11-03 16:05:43 +00:00
leveled_log:log("I0002", [Requestor, State#state.manifest_sqn]),
Basic GET/PUT and rolling in Inker Add support to roll file on PUT in the inker
2016-09-06 17:17:31 +01:00
{reply, {State#state.manifest,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
State#state.active_journaldb},
Snapshot testing Work to test the checking of sequence numbers in snapshots as required by the inkers clerk to calculate the percentage of a file which is compactable
2016-09-26 10:55:08 +01:00
State#state{registered_snapshots=Rs}};
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
handle_call({confirm_delete, ManSQN}, _From, State) ->
Reply = lists:foldl(fun({_R, SnapSQN}, Bool) ->
Registering and releasing of Journal snapshots Added a test of journal compaction with a registered snapshot and it showed that the deleting of files did not correctly check the list of registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
case SnapSQN >= ManSQN of
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
true ->
Bool;
false ->
false
end end,
true,
State#state.registered_snapshots),
{reply, Reply, State};
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
handle_call(get_manifest, _From, State) ->
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
{reply, leveled_imanifest:to_list(State#state.manifest), State};
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
handle_call({update_manifest,
ManifestSnippet,
DeletedFiles}, _From, State) ->
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
DropFun =
fun(E, Acc) ->
leveled_imanifest:remove_entry(Acc, E)
end,
Man0 = lists:foldl(DropFun, State#state.manifest, DeletedFiles),
AddFun =
fun(E, Acc) ->
leveled_imanifest:add_entry(Acc, E, false)
end,
Man1 = lists:foldl(AddFun, Man0, ManifestSnippet),
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
NewManifestSQN = State#state.manifest_sqn + 1,
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:printer(Man1),
leveled_imanifest:writer(Man1, NewManifestSQN, State#state.root_path),
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
{reply,
{ok, NewManifestSQN},
State#state{manifest=Man1,
manifest_sqn=NewManifestSQN,
pending_removals=DeletedFiles}};
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
handle_call(print_manifest, _From, State) ->
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:printer(State#state.manifest),
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
{reply, ok, State};
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
handle_call({compact,
Checker,
InitiateFun,
FilterFun,
Timeout},
_From, State) ->
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
leveled_iclerk:clerk_compact(State#state.clerk,
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
Checker,
InitiateFun,
FilterFun,
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
self(),
Timeout),
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
{reply, ok, State#state{compaction_pending=true}};
handle_call(compaction_complete, _From, State) ->
{reply, ok, State#state{compaction_pending=false}};
CDB - switch to gen_fsm The CDB file management server has distinct states, and was growing case logic to prevent certain messages from being handled in ceratin states, and to handle different messages differently. So this has now been converted to a gen_fsm. As part of resolving this, the space_clear_ondelete test has been completed, and completing this revealed that the Penciller could not cope with a change which emptied the ledger. So a series of changes has been handled to allow it to smoothly progress to an empty manifest.
2016-10-26 20:39:16 +01:00
handle_call(compaction_pending, _From, State) ->
{reply, State#state.compaction_pending, State};
Quality review Minor test fix-up and quality changes
2016-10-30 22:06:44 +00:00
handle_call(close, _From, State) ->
Add destruction Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
{stop, normal, ok, State};
handle_call(doom, _From, State) ->
FPs = [filepath(State#state.root_path, journal_dir),
filepath(State#state.root_path, manifest_dir),
filepath(State#state.root_path, journal_compact_dir),
filepath(State#state.root_path, journal_waste_dir)],
leveled_log:log("I0018", []),
{stop, normal, {ok, FPs}, State}.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
Inker Close nastiness Try to stop some of the potential deadlocking around Inker close and prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
handle_cast({release_snapshot, Snapshot}, State) ->
Rs = lists:keydelete(Snapshot, 1, State#state.registered_snapshots),
leveled_log:log("I0003", [Snapshot]),
leveled_log:log("I0004", [length(Rs)]),
{noreply, State#state{registered_snapshots=Rs}}.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
handle_info(_Info, State) ->
{noreply, State}.
Basic GET/PUT and rolling in Inker Add support to roll file on PUT in the inker
2016-09-06 17:17:31 +01:00
terminate(Reason, State) ->
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
case State#state.is_snapshot of
true ->
Add more complex snapshot test This exposed another off-by-one error on startup. This commit also includes an unsafe change to reply early from a rolling CDB file (with lots of objects writing the hash table can take too long). This is bad, but will be resolved through a refactor of the manifest writing: essentially we deferred writing of the manifest update which was an unnecessary performance optimisation. If instead we wait on this, the process is made substantially simpler, and it is safer to perform the roll of the complete CDB journal asynchronously. If the manifest update takes too long, an append-only log may be used instead.
2016-10-07 10:04:48 +01:00
ok = ink_releasesnapshot(State#state.source_inker, self());
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
false ->
Log improvements Continuation of log review and conversion to using central log function. Fixup of convoluted shutdown process between Bookie, Inker and Inker's Clerk
2016-11-03 16:05:43 +00:00
leveled_log:log("I0005", [Reason]),
leveled_log:log("I0006", [State#state.journal_sqn,
State#state.manifest_sqn]),
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
leveled_iclerk:clerk_stop(State#state.clerk),
lists:foreach(fun({Snap, _SQN}) -> ok = ink_close(Snap) end,
State#state.registered_snapshots),
Log improvements Continuation of log review and conversion to using central log function. Fixup of convoluted shutdown process between Bookie, Inker and Inker's Clerk
2016-11-03 16:05:43 +00:00
leveled_log:log("I0007", []),
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:printer(State#state.manifest),
ManAsList = leveled_imanifest:to_list(State#state.manifest),
ok = close_allmanifest(ManAsList)
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
end.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
code_change(_OldVsn, State, _Extra) ->
{ok, State}.
%%%============================================================================
%%% Internal functions
%%%============================================================================
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
start_from_file(InkOpts) ->
RootPath = InkOpts#inker_options.root_path,
CDBopts = InkOpts#inker_options.cdb_options,
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
JournalFP = filepath(RootPath, journal_dir),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
filelib:ensure_dir(JournalFP),
CompactFP = filepath(RootPath, journal_compact_dir),
filelib:ensure_dir(CompactFP),
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
WasteFP = filepath(RootPath, journal_waste_dir),
filelib:ensure_dir(WasteFP),
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
ManifestFP = filepath(RootPath, manifest_dir),
Quality Review Looked through test coverage and dialyzer output and attempted to fill test gaps and strip out untestable code (to let it crash).
2016-10-29 00:52:49 +01:00
ok = filelib:ensure_dir(ManifestFP),
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
Quality Review Looked through test coverage and dialyzer output and attempted to fill test gaps and strip out untestable code (to let it crash).
2016-10-29 00:52:49 +01:00
{ok, ManifestFilenames} = file:list_dir(ManifestFP),
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
IClerkCDBOpts = CDBopts#cdb_options{file_path = CompactFP,
waste_path = WasteFP},
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
ReloadStrategy = InkOpts#inker_options.reload_strategy,
Quality Review Looked through test coverage and dialyzer output and attempted to fill test gaps and strip out untestable code (to let it crash).
2016-10-29 00:52:49 +01:00
MRL = InkOpts#inker_options.max_run_length,
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
WRP = InkOpts#inker_options.waste_retention_period,
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
IClerkOpts = #iclerk_options{inker = self(),
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
cdb_options=IClerkCDBOpts,
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
waste_retention_period = WRP,
Quality Review Looked through test coverage and dialyzer output and attempted to fill test gaps and strip out untestable code (to let it crash).
2016-10-29 00:52:49 +01:00
reload_strategy = ReloadStrategy,
max_run_length = MRL},
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
{ok, Clerk} = leveled_iclerk:clerk_new(IClerkOpts),
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
{Manifest,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
ManifestSQN,
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
JournalSQN,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
ActiveJournal} = build_manifest(ManifestFilenames,
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
RootPath,
CDBopts),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{ok, #state{manifest = Manifest,
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
manifest_sqn = ManifestSQN,
journal_sqn = JournalSQN,
active_journaldb = ActiveJournal,
root_path = RootPath,
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
cdb_options = CDBopts#cdb_options{waste_path=WasteFP},
Snapshot testing Work to test the checking of sequence numbers in snapshots as required by the inkers clerk to calculate the percentage of a file which is compactable
2016-09-26 10:55:08 +01:00
clerk = Clerk}}.
Snapshot Work - Interim Commit Some initial work to get snapshots going. Changes required, as need to snapshot through the Bookie to ensure that there is no race between extracting the Bookie's in-memory view and the Penciller's view if a push_to_mem has occurred inbetween. A lot still outstanding, especially around Inker snapshots, and handling timeouts
2016-09-23 18:50:29 +01:00
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
put_object(LedgerKey, Object, KeyChanges, State) ->
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
NewSQN = State#state.journal_sqn + 1,
Resolve test issues
2017-01-19 09:47:56 +00:00
ActiveJournal = State#state.active_journaldb,
Rolback hash|no_hash and batch journal compaction The no_hash option in CDB files became too hard to manage, in particular the need to scan the whole file to find the last_key rather than cheat and use the index. It has been removed for now. The writing to the journal during journal compaction has now been enhanced by a mput option on the CDB file write - so it can write each batch as one pwrite operation.
2016-10-26 11:39:27 +01:00
{JournalKey, JournalBin} = leveled_codec:to_inkerkv(LedgerKey,
NewSQN,
Object,
KeyChanges),
Resolve test issues
2017-01-19 09:47:56 +00:00
case leveled_cdb:cdb_put(ActiveJournal,
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
JournalKey,
Rolback hash|no_hash and batch journal compaction The no_hash option in CDB files became too hard to manage, in particular the need to scan the whole file to find the last_key rather than cheat and use the index. It has been removed for now. The writing to the journal during journal compaction has now been enhanced by a mput option on the CDB file write - so it can write each batch as one pwrite operation.
2016-10-26 11:39:27 +01:00
JournalBin) of
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
ok ->
Add put timing points
2016-12-20 23:11:50 +00:00
{ok,
Trigger log points, deprecate lower PUT log points Only record PUT timings at the Bookie, and trigger them correctly on the right log point
2017-02-26 19:07:55 +00:00
State#state{journal_sqn=NewSQN},
Add put timing points
2016-12-20 23:11:50 +00:00
byte_size(JournalBin)};
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
roll ->
Sort out different timestamps
2016-12-20 23:16:52 +00:00
SWroll = os:timestamp(),
Resolve test issues
2017-01-19 09:47:56 +00:00
LastKey = leveled_cdb:cdb_lastkey(ActiveJournal),
ok = leveled_cdb:cdb_roll(ActiveJournal),
Manifest0 = leveled_imanifest:append_lastkey(State#state.manifest,
ActiveJournal,
LastKey),
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
CDBopts = State#state.cdb_options,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
ManEntry = start_new_activejournal(NewSQN,
State#state.root_path,
CDBopts),
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
{_, _, NewJournalP, _} = ManEntry,
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
Manifest1 = leveled_imanifest:add_entry(Manifest0, ManEntry, true),
ok = leveled_imanifest:writer(Manifest1,
State#state.manifest_sqn + 1,
State#state.root_path),
Move towards Inker Key Types The current mechanism of re-loading data from the Journla to the Ledger from any potential SQN is not safe when combined with Journla compaction. This commit doesn't resolve thes eproblems, but starts the groundwork for resolving by introducing Inker Key Types. These types would differentiate between objects which are standard Key/Value pairs, objects which are tombstones for keys, and objects whihc represent Key Changes only. The idea is that there will be flexible reload strategies based on object tags - retain (retain a key change object when compacting a standard object) - recalc (allow key changes to be recalculated from objects and ledger state when loading the Ledger from the journal - recover (allow for the potential loss of data on loss within the perisste dpart of the ledger, potentially due to recovery through externla anti-entropy operations).
2016-10-25 01:57:12 +01:00
ok = leveled_cdb:cdb_put(NewJournalP,
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
JournalKey,
Rolback hash|no_hash and batch journal compaction The no_hash option in CDB files became too hard to manage, in particular the need to scan the whole file to find the last_key rather than cheat and use the index. It has been removed for now. The writing to the journal during journal compaction has now been enhanced by a mput option on the CDB file write - so it can write each batch as one pwrite operation.
2016-10-26 11:39:27 +01:00
JournalBin),
Sort out different timestamps
2016-12-20 23:16:52 +00:00
leveled_log:log_timer("I0008", [], SWroll),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{rolling,
State#state{journal_sqn=NewSQN,
Resolve test issues
2017-01-19 09:47:56 +00:00
manifest=Manifest1,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
manifest_sqn = State#state.manifest_sqn + 1,
active_journaldb=NewJournalP},
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
byte_size(JournalBin)}
Basement Tombstones Further progress towards the tidying up of basement tombstones in the Ledger, with support added for key-listing to help with testing (and as a potentially required feature). The test is incomplete, but committing at this stage as the last commit broke some tests (within the test code). There are some outstanding questions about the handling of tombstones in the Journal during compaction. There exists a condition whereby values could return if a recent journal is compacted and tombstones are removed (as they are no longer present), but older journals have not been compacted. Now on stop/start - if the Ledger is wiped the removal of the keys will be forgotten but the original PUTs would still remain. The safest thing maybe to have rule that tombstones are never deleted from the Inker's Journal - and accept the build-up of garbage. Or there could be an addition to the compaction process that checks back through all the inker files to check that the Key of a tombstone is not present in the past, before it is removed in the compaction.
2016-10-23 22:45:43 +01:00
end.
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
get_object(LedgerKey, SQN, Manifest) ->
Remove binary_to_term when fetching don't need to binary_to_term key changes
2017-03-29 15:37:04 +01:00
get_object(LedgerKey, SQN, Manifest, false).
get_object(LedgerKey, SQN, Manifest, ToIgnoreKeyChanges) ->
Broken WIP
2017-01-17 16:30:04 +00:00
JournalP = leveled_imanifest:find_entry(SQN, Manifest),
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
{InkerKey, _V, true} = leveled_codec:to_inkerkv(LedgerKey,
SQN,
to_fetch,
null),
Obj = leveled_cdb:cdb_get(JournalP, InkerKey),
Remove binary_to_term when fetching don't need to binary_to_term key changes
2017-03-29 15:37:04 +01:00
leveled_codec:from_inkerkv(Obj, ToIgnoreKeyChanges).
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
Refactor snapshot Better reuse snapshotting fucntions in the Bookie, and use it to support doing Inker clone checks
2016-10-31 17:26:28 +00:00
key_check(LedgerKey, SQN, Manifest) ->
Broken WIP
2017-01-17 16:30:04 +00:00
JournalP = leveled_imanifest:find_entry(SQN, Manifest),
Refactor snapshot Better reuse snapshotting fucntions in the Bookie, and use it to support doing Inker clone checks
2016-10-31 17:26:28 +00:00
{InkerKey, _V, true} = leveled_codec:to_inkerkv(LedgerKey,
SQN,
to_fetch,
null),
leveled_cdb:cdb_keycheck(JournalP, InkerKey).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
build_manifest(ManifestFilenames,
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
RootPath,
CDBopts) ->
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
% Find the manifest with a highest Manifest sequence number
% Open it and read it to get the current Confirmed Manifest
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
ManifestRegex = "(?<MSQN>[0-9]+)\\." ++ leveled_imanifest:complete_filex(),
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
ValidManSQNs = sequencenumbers_fromfilenames(ManifestFilenames,
ManifestRegex,
'MSQN'),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{Manifest,
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
ManifestSQN} = case length(ValidManSQNs) of
0 ->
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{[], 1};
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
_ ->
PersistedManSQN = lists:max(ValidManSQNs),
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
M1 = leveled_imanifest:reader(PersistedManSQN,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
RootPath),
{M1, PersistedManSQN}
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
end,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
% Open the manifest files, completing if necessary and ensure there is
% a valid active journal at the head of the manifest
OpenManifest = open_all_manifest(Manifest, RootPath, CDBopts),
Broken WIP
2017-01-17 16:30:04 +00:00
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
{ActiveLowSQN,
_FN,
ActiveJournal,
_LK} = leveled_imanifest:head_entry(OpenManifest),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
JournalSQN = case leveled_cdb:cdb_lastkey(ActiveJournal) of
empty ->
ActiveLowSQN;
Move towards Inker Key Types The current mechanism of re-loading data from the Journla to the Ledger from any potential SQN is not safe when combined with Journla compaction. This commit doesn't resolve thes eproblems, but starts the groundwork for resolving by introducing Inker Key Types. These types would differentiate between objects which are standard Key/Value pairs, objects which are tombstones for keys, and objects whihc represent Key Changes only. The idea is that there will be flexible reload strategies based on object tags - retain (retain a key change object when compacting a standard object) - recalc (allow key changes to be recalculated from objects and ledger state when loading the Ledger from the journal - recover (allow for the potential loss of data on loss within the perisste dpart of the ledger, potentially due to recovery through externla anti-entropy operations).
2016-10-25 01:57:12 +01:00
{JSQN, _Type, _LastKey} ->
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
JSQN
end,
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
% Update the manifest if it has been changed by the process of laoding
% the manifest (must also increment the manifest SQN).
Broken WIP
2017-01-17 16:30:04 +00:00
UpdManifestSQN =
if
length(OpenManifest) > length(Manifest) ->
leveled_log:log("I0009", []),
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:printer(OpenManifest),
Broken WIP
2017-01-17 16:30:04 +00:00
NextSQN = ManifestSQN + 1,
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:writer(OpenManifest, NextSQN, RootPath),
Broken WIP
2017-01-17 16:30:04 +00:00
NextSQN;
true ->
leveled_log:log("I0010", []),
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:printer(OpenManifest),
Broken WIP
2017-01-17 16:30:04 +00:00
ManifestSQN
end,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{OpenManifest, UpdManifestSQN, JournalSQN, ActiveJournal}.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
close_allmanifest([]) ->
ok;
close_allmanifest([H|ManifestT]) ->
Broken WIP
2017-01-17 16:30:04 +00:00
{_, _, Pid, _} = H,
Basic GET/PUT and rolling in Inker Add support to roll file on PUT in the inker
2016-09-06 17:17:31 +01:00
ok = leveled_cdb:cdb_close(Pid),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
close_allmanifest(ManifestT).
open_all_manifest([], RootPath, CDBOpts) ->
Log improvements Continuation of log review and conversion to using central log function. Fixup of convoluted shutdown process between Bookie, Inker and Inker's Clerk
2016-11-03 16:05:43 +00:00
leveled_log:log("I0011", []),
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:add_entry([],
start_new_activejournal(1, RootPath, CDBOpts),
true);
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
open_all_manifest(Man0, RootPath, CDBOpts) ->
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
Man1 = leveled_imanifest:to_list(Man0),
Broken WIP
2017-01-17 16:30:04 +00:00
[{HeadSQN, HeadFN, _IgnorePid, HeadLK}|ManifestTail] = Man1,
OpenJournalFun =
fun(ManEntry) ->
Syntax
2017-02-26 22:46:57 +00:00
{LowSQN, FN, _, LK_RO} = ManEntry,
CFN = FN ++ "." ++ ?JOURNAL_FILEX,
PFN = FN ++ "." ++ ?PENDING_FILEX,
case filelib:is_file(CFN) of
true ->
{ok, Pid} = leveled_cdb:cdb_reopen_reader(CFN,
LK_RO),
{LowSQN, FN, Pid, LK_RO};
false ->
W = leveled_cdb:cdb_open_writer(PFN, CDBOpts),
{ok, Pid} = W,
ok = leveled_cdb:cdb_roll(Pid),
LK_WR = leveled_cdb:cdb_lastkey(Pid),
{LowSQN, FN, Pid, LK_WR}
end
Broken WIP
2017-01-17 16:30:04 +00:00
end,
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
OpenedTailAsList = lists:map(OpenJournalFun, ManifestTail),
OpenedTail = leveled_imanifest:from_list(OpenedTailAsList),
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
CompleteHeadFN = HeadFN ++ "." ++ ?JOURNAL_FILEX,
PendingHeadFN = HeadFN ++ "." ++ ?PENDING_FILEX,
case filelib:is_file(CompleteHeadFN) of
true ->
leveled_log:log("I0012", [HeadFN]),
{ok, HeadR} = leveled_cdb:cdb_open_reader(CompleteHeadFN),
LastKey = leveled_cdb:cdb_lastkey(HeadR),
LastSQN = element(1, LastKey),
ManToHead = leveled_imanifest:add_entry(OpenedTail,
{HeadSQN,
HeadFN,
HeadR,
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
LastKey},
true),
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
NewManEntry = start_new_activejournal(LastSQN + 1,
RootPath,
CDBOpts),
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
leveled_imanifest:add_entry(ManToHead,
NewManEntry,
true);
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
false ->
{ok, HeadW} = leveled_cdb:cdb_open_writer(PendingHeadFN,
CDBOpts),
leveled_imanifest:add_entry(OpenedTail,
Inker Manifest to be a two-level list To eas seraching from front to back, change the inker manifest to be a two-level list
2017-01-19 12:23:28 +00:00
{HeadSQN, HeadFN, HeadW, HeadLK},
true)
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
end.
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
start_new_activejournal(SQN, RootPath, CDBOpts) ->
Filename = filepath(RootPath, SQN, new_journal),
{ok, PidW} = leveled_cdb:cdb_open_writer(Filename, CDBOpts),
Broken WIP
2017-01-17 16:30:04 +00:00
{SQN, Filename, PidW, empty}.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
%% Scan between sequence numbers applying FilterFun to each entry where
%% FilterFun{K, V, Acc} -> Penciller Key List
%% Load the output for the CDB file into the Penciller.
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
load_from_sequence(_MinSQN, _FilterFun, _PCL, []) ->
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
ok;
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
load_from_sequence(MinSQN, FilterFun, PCL, [{LowSQN, FN, Pid, _LK}|Rest])
Further bookie test - CDB optimisation and Inker manifest correction Additional bookie test revealed that the persisting/reading of inker manifests was inconsistent and buggy. Also, the CDB files were inffeciently writing the top index table - needed to be improved as this is blokicng on a roll
2016-09-19 15:31:26 +01:00
when LowSQN >= MinSQN ->
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
load_between_sequence(MinSQN,
MinSQN + ?LOADING_BATCH,
FilterFun,
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
PCL,
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
Pid,
undefined,
FN,
Rest);
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
load_from_sequence(MinSQN, FilterFun, PCL, [{_LowSQN, FN, Pid, _LK}|Rest]) ->
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
case Rest of
[] ->
load_between_sequence(MinSQN,
MinSQN + ?LOADING_BATCH,
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
FilterFun,
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
PCL,
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
Pid,
undefined,
FN,
Rest);
Resolve test issues
2017-01-19 09:47:56 +00:00
[{NextSQN, _NxtFN, _NxtPid, _NxtLK}|_Rest] when NextSQN > MinSQN ->
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
load_between_sequence(MinSQN,
MinSQN + ?LOADING_BATCH,
FilterFun,
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
PCL,
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
Pid,
undefined,
FN,
Rest);
_ ->
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
load_from_sequence(MinSQN, FilterFun, PCL, Rest)
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
end.
load_between_sequence(MinSQN, MaxSQN, FilterFun, Penciller,
CDBpid, StartPos, FN, Rest) ->
Log improvements Continuation of log review and conversion to using central log function. Fixup of convoluted shutdown process between Bookie, Inker and Inker's Clerk
2016-11-03 16:05:43 +00:00
leveled_log:log("I0014", [FN, MinSQN]),
Magic Hash - and no L0 Index Move to using the DJ Bernstein Magic Hash consistently, and trying to make sure we only hash once for each operation (as the hash is more expensive than phash2). The improved lookup time for missing keys should allow for the L0 index to be removed, and hence speed up the completion time for push_mem operations. It is expected there will be a second stage of creating a tinybloom as part of the SFT creation process, and then adding that tinybloom to the manifest. This will then reduce the message passing required for a GET not in the cache or higher levels
2016-12-11 01:02:56 +00:00
InitAcc = {MinSQN, MaxSQN, leveled_bookie:empty_ledgercache()},
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
Res = case leveled_cdb:cdb_scan(CDBpid, FilterFun, InitAcc, StartPos) of
Magic Hash - and no L0 Index Move to using the DJ Bernstein Magic Hash consistently, and trying to make sure we only hash once for each operation (as the hash is more expensive than phash2). The improved lookup time for missing keys should allow for the L0 index to be removed, and hence speed up the completion time for push_mem operations. It is expected there will be a second stage of creating a tinybloom as part of the SFT creation process, and then adding that tinybloom to the manifest. This will then reduce the message passing required for a GET not in the cache or higher levels
2016-12-11 01:02:56 +00:00
{eof, {AccMinSQN, _AccMaxSQN, AccLC}} ->
ok = push_to_penciller(Penciller, AccLC),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
{ok, AccMinSQN};
Magic Hash - and no L0 Index Move to using the DJ Bernstein Magic Hash consistently, and trying to make sure we only hash once for each operation (as the hash is more expensive than phash2). The improved lookup time for missing keys should allow for the L0 index to be removed, and hence speed up the completion time for push_mem operations. It is expected there will be a second stage of creating a tinybloom as part of the SFT creation process, and then adding that tinybloom to the manifest. This will then reduce the message passing required for a GET not in the cache or higher levels
2016-12-11 01:02:56 +00:00
{LastPosition, {_AccMinSQN, _AccMaxSQN, AccLC}} ->
ok = push_to_penciller(Penciller, AccLC),
Add initial end-to-end common tests These tests highlighted some logical issues when scanning over databases on startup, so fixes are wrapped in here.
2016-10-05 09:54:53 +01:00
NextSQN = MaxSQN + 1,
load_between_sequence(NextSQN,
NextSQN + ?LOADING_BATCH,
FilterFun,
Penciller,
CDBpid,
LastPosition,
FN,
Rest)
end,
case Res of
{ok, LMSQN} ->
load_from_sequence(LMSQN, FilterFun, Penciller, Rest);
ok ->
ok
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
end.
Scan over CDB file Make scanning over a CDB file generic rather than specific to read-in of active nursery log - open to be called as an external function to support other scanning behaviour.
2016-09-09 15:58:19 +01:00
Magic Hash - and no L0 Index Move to using the DJ Bernstein Magic Hash consistently, and trying to make sure we only hash once for each operation (as the hash is more expensive than phash2). The improved lookup time for missing keys should allow for the L0 index to be removed, and hence speed up the completion time for push_mem operations. It is expected there will be a second stage of creating a tinybloom as part of the SFT creation process, and then adding that tinybloom to the manifest. This will then reduce the message passing required for a GET not in the cache or higher levels
2016-12-11 01:02:56 +00:00
push_to_penciller(Penciller, LedgerCache) ->
Smoother handling of back-pressure The Penciller had two problems in previous commits: - If it had a push_mem soon after a L0 file had been created, the push_mem would stall waiting for the L0 file to complete - and this count take 100-200ms - The penciller's clerk favoured L0 work, but was lazy about asking for other work in-between, so often the L1 layer was bursting over capacity and the clerk was doing nothing but merging more L0 files in (with those merges getting more and more expensive as they had to cover more and more files) There are some partial resolutions to this. There is now an aggressive timeout when checking whther the L0 file is ready on a push_mem, and if the timeout is breached the error is caught and a 'returned' message goes back to the Bookie. the Bookie doesn't now empty its cache, it carrie son filling it, but on some probability it will keep trying to push_mem on future pushes. This increases Jitter around the expensive operation and split out the L0 delay into defined chunks. The penciller's clerk is now more aggressive in asking for work. There is also some simplification of the relationship between clerk timeouts and penciller back-pressure. Also resolved is an issue of inconcistency between the loader and the on startup (replaying the transaction log) and the standard push_mem process. The loader was not correctly de-duplicating by adding first (in order) to a tree before outputting the list from the tree. Some thought will be given later as to whether non-L0 work can be safely prioritised if the merge process still keeps getting behind.
2016-10-20 02:23:45 +01:00
% The push to penciller must start as a tree to correctly de-duplicate
% the list by order before becoming a de-duplicated list for loading
Switch the LM1 cache to be a tree Use a tree of lists not a skiplist
2017-01-20 16:36:20 +00:00
LC0 = leveled_bookie:loadqueue_ledgercache(LedgerCache),
push_to_penciller_loop(Penciller, LC0).
push_to_penciller_loop(Penciller, LedgerCache) ->
Magic Hash - and no L0 Index Move to using the DJ Bernstein Magic Hash consistently, and trying to make sure we only hash once for each operation (as the hash is more expensive than phash2). The improved lookup time for missing keys should allow for the L0 index to be removed, and hence speed up the completion time for push_mem operations. It is expected there will be a second stage of creating a tinybloom as part of the SFT creation process, and then adding that tinybloom to the manifest. This will then reduce the message passing required for a GET not in the cache or higher levels
2016-12-11 01:02:56 +00:00
case leveled_bookie:push_ledgercache(Penciller, LedgerCache) of
Penciller Memory Refactor Plugged the ne wpencille rmemory into the Penciller, and took advantage of the increased speed to simplify the callbacks involved. The outcome is much simpler code
2016-10-30 18:25:30 +00:00
returned ->
Smoother handling of back-pressure The Penciller had two problems in previous commits: - If it had a push_mem soon after a L0 file had been created, the push_mem would stall waiting for the L0 file to complete - and this count take 100-200ms - The penciller's clerk favoured L0 work, but was lazy about asking for other work in-between, so often the L1 layer was bursting over capacity and the clerk was doing nothing but merging more L0 files in (with those merges getting more and more expensive as they had to cover more and more files) There are some partial resolutions to this. There is now an aggressive timeout when checking whther the L0 file is ready on a push_mem, and if the timeout is breached the error is caught and a 'returned' message goes back to the Bookie. the Bookie doesn't now empty its cache, it carrie son filling it, but on some probability it will keep trying to push_mem on future pushes. This increases Jitter around the expensive operation and split out the L0 delay into defined chunks. The penciller's clerk is now more aggressive in asking for work. There is also some simplification of the relationship between clerk timeouts and penciller back-pressure. Also resolved is an issue of inconcistency between the loader and the on startup (replaying the transaction log) and the standard push_mem process. The loader was not correctly de-duplicating by adding first (in order) to a tree before outputting the list from the tree. Some thought will be given later as to whether non-L0 work can be safely prioritised if the merge process still keeps getting behind.
2016-10-20 02:23:45 +01:00
timer:sleep(?LOADING_PAUSE),
Switch the LM1 cache to be a tree Use a tree of lists not a skiplist
2017-01-20 16:36:20 +00:00
push_to_penciller_loop(Penciller, LedgerCache);
Penciller Refactor Removed o(100) lines of code by refactoring the Penciller to no longer use ETS tables. The code is less confusing, and probably not an awful lot slower.
2016-10-27 20:56:18 +01:00
ok ->
WIP - First draft of Bookie code First draft of untested bookie code
2016-09-15 10:53:24 +01:00
ok
end.
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
sequencenumbers_fromfilenames(Filenames, Regex, IntName) ->
lists:foldl(fun(FN, Acc) ->
case re:run(FN,
Regex,
[{capture, [IntName], list}]) of
nomatch ->
Acc;
{match, [Int]} when is_list(Int) ->
Inker test for pending manifest Should ignore the corrupted pending manifest file
2016-11-03 20:00:55 +00:00
Acc ++ [list_to_integer(Int)]
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
end end,
[],
Filenames).
filepath(RootPath, journal_dir) ->
RootPath ++ "/" ++ ?FILES_FP ++ "/";
filepath(RootPath, manifest_dir) ->
Initial work on Journal Compaction Largely untested work at this stage to allow for the Inker to request the Inker's clerk to perform a single round of compact based on the best run of files it can find.
2016-09-27 14:58:26 +01:00
RootPath ++ "/" ++ ?MANIFEST_FP ++ "/";
filepath(RootPath, journal_compact_dir) ->
Deferred Deletion of Journals This allows for deleted journals to be retained for a period (the waste_retnetion_period). The idea being that a backup strategy can ensure that all journals are backed up, even ones created and removed from within a backup period - so that any restore pont is possible. This is also a pre-cursor to removing some of the PromptDelete complexity from the Inker Clerk - all compactions can prompt deletion as deletion is now deferred.
2016-11-14 11:17:14 +00:00
filepath(RootPath, journal_dir) ++ "/" ++ ?COMPACT_FP ++ "/";
filepath(RootPath, journal_waste_dir) ->
filepath(RootPath, journal_dir) ++ "/" ++ ?WASTE_FP ++ "/".
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
filepath(RootPath, NewSQN, new_journal) ->
filename:join(filepath(RootPath, journal_dir),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
integer_to_list(NewSQN) ++ "_"
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
++ leveled_codec:generate_uuid()
Inker Clerk - Further Testing Expanded the unit tetsing of the Inker Clerk actor. Still WIP
2016-09-28 11:41:56 +01:00
++ "." ++ ?PENDING_FILEX);
filepath(CompactFilePath, NewSQN, compact_journal) ->
filename:join(CompactFilePath,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
integer_to_list(NewSQN) ++ "_"
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
++ leveled_codec:generate_uuid()
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
++ "." ++ ?PENDING_FILEX).
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
initiate_penciller_snapshot(Bookie) ->
2i order of events When running a load of mainly 2i queries, there is a huge cost in the previous snapshot code. The time taken to create a clone of the Penciller (duplicating all the LoopState) varied between 1 and 200ms depedning on the size of the LoopState. For 2i queries, most of that LoopState was then being thrown away after running the query against the levelzero_cache. This was taking < 1ms on average. It would be better to avoid the o(100)ms of CPU burning and block for o(1)ms - so th eorder of events have been changed ot filter first so only the small part of the LoopState actually required is copied to the clone.
2017-03-06 18:42:32 +00:00
{ok, LedgerSnap, _} = leveled_bookie:book_snapshotledger(Bookie, self(), undefined),
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
MaxSQN = leveled_penciller:pcl_getstartupsequencenumber(LedgerSnap),
{LedgerSnap, MaxSQN}.
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
Add put timing points
2016-12-20 23:11:50 +00:00
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
%%%============================================================================
%%% Test
%%%============================================================================
-ifdef(TEST).
Trying to standardise binary manipulation of value Looking into theory that use of term_to_binary is imperfect. Also may be better to compress values only when they are compacted?
2017-03-20 15:43:54 +00:00
create_value_for_journal(Obj, Comp) ->
leveled_codec:create_value_for_journal(Obj, Comp).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
build_dummy_journal() ->
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
F = fun(X) -> X end,
build_dummy_journal(F).
build_dummy_journal(KeyConvertF) ->
Improved testing Improve testing of Penciller to show startup and shutdown with push, merging and fetch
2016-09-08 14:21:30 +01:00
RootPath = "../test/journal",
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
clean_testdir(RootPath),
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
JournalFP = filepath(RootPath, journal_dir),
ManifestFP = filepath(RootPath, manifest_dir),
ok = filelib:ensure_dir(RootPath),
ok = filelib:ensure_dir(JournalFP),
ok = filelib:ensure_dir(ManifestFP),
F1 = filename:join(JournalFP, "nursery_1.pnd"),
{ok, J1} = leveled_cdb:cdb_open_writer(F1),
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
{K1, V1} = {KeyConvertF("Key1"), "TestValue1"},
{K2, V2} = {KeyConvertF("Key2"), "TestValue2"},
Trying to standardise binary manipulation of value Looking into theory that use of term_to_binary is imperfect. Also may be better to compress values only when they are compacted?
2017-03-20 15:43:54 +00:00
ok = leveled_cdb:cdb_put(J1,
{1, stnd, K1},
create_value_for_journal({V1, []}, false)),
ok = leveled_cdb:cdb_put(J1,
{2, stnd, K2},
create_value_for_journal({V2, []}, false)),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
ok = leveled_cdb:cdb_roll(J1),
Broken WIP
2017-01-17 16:30:04 +00:00
LK1 = leveled_cdb:cdb_lastkey(J1),
Tidy up closure of CDB Files
2016-11-08 22:43:22 +00:00
lists:foldl(fun(X, Closed) ->
case Closed of
true -> true;
false ->
case leveled_cdb:cdb_checkhashtable(J1) of
true -> leveled_cdb:cdb_close(J1), true;
false -> timer:sleep(X), false
end
end
end,
false,
lists:seq(1, 5)),
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
F2 = filename:join(JournalFP, "nursery_3.pnd"),
{ok, J2} = leveled_cdb:cdb_open_writer(F2),
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
{K1, V3} = {KeyConvertF("Key1"), "TestValue3"},
{K4, V4} = {KeyConvertF("Key4"), "TestValue4"},
Trying to standardise binary manipulation of value Looking into theory that use of term_to_binary is imperfect. Also may be better to compress values only when they are compacted?
2017-03-20 15:43:54 +00:00
ok = leveled_cdb:cdb_put(J2,
{3, stnd, K1},
create_value_for_journal({V3, []}, false)),
ok = leveled_cdb:cdb_put(J2,
{4, stnd, K4},
create_value_for_journal({V4, []}, false)),
Broken WIP
2017-01-17 16:30:04 +00:00
LK2 = leveled_cdb:cdb_lastkey(J2),
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
ok = leveled_cdb:cdb_close(J2),
Broken WIP
2017-01-17 16:30:04 +00:00
Manifest = [{1, "../test/journal/journal_files/nursery_1", "pid1", LK1},
{3, "../test/journal/journal_files/nursery_3", "pid2", LK2}],
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
ManifestBin = term_to_binary(Manifest),
{ok, MF1} = file:open(filename:join(ManifestFP, "1.man"),
[binary, raw, read, write]),
ok = file:write(MF1, ManifestBin),
ok = file:close(MF1).
clean_testdir(RootPath) ->
clean_subdir(filepath(RootPath, journal_dir)),
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
clean_subdir(filepath(RootPath, journal_compact_dir)),
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
clean_subdir(filepath(RootPath, manifest_dir)).
clean_subdir(DirPath) ->
Add initial system test Add some initial system tests. This highlighted issues: - That files deleted by compaction would be left orphaned and not close, and would not in fact delete (now deleted by closure only) - There was an issue on stratup that the first few keys in each journal would not be re-loaded into the ledger
2016-10-03 23:34:28 +01:00
ok = filelib:ensure_dir(DirPath),
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
{ok, Files} = file:list_dir(DirPath),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
lists:foreach(fun(FN) ->
File = filename:join(DirPath, FN),
case file:delete(File) of
ok -> io:format("Success deleting ~s~n", [File]);
_ -> io:format("Error deleting ~s~n", [File])
end
end,
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
Files).
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
simple_inker_test() ->
Improved testing Improve testing of Penciller to show startup and shutdown with push, merging and fetch
2016-09-08 14:21:30 +01:00
RootPath = "../test/journal",
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
build_dummy_journal(),
Trying to standardise binary manipulation of value Looking into theory that use of term_to_binary is imperfect. Also may be better to compress values only when they are compacted?
2017-03-20 15:43:54 +00:00
CDBopts = #cdb_options{max_size=300000, binary_mode=true},
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
{ok, Ink1} = ink_start(#inker_options{root_path=RootPath,
cdb_options=CDBopts}),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
Obj1 = ink_get(Ink1, "Key1", 1),
?assertMatch({{1, "Key1"}, {"TestValue1", []}}, Obj1),
Trying to standardise binary manipulation of value Looking into theory that use of term_to_binary is imperfect. Also may be better to compress values only when they are compacted?
2017-03-20 15:43:54 +00:00
Obj3 = ink_get(Ink1, "Key1", 3),
?assertMatch({{3, "Key1"}, {"TestValue3", []}}, Obj3),
Obj4 = ink_get(Ink1, "Key4", 4),
?assertMatch({{4, "Key4"}, {"TestValue4", []}}, Obj4),
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
ink_close(Ink1),
clean_testdir(RootPath).
Code reduction refactor An attempt to refactor out more complex code. The Penciller clerk and Penciller have been re-shaped so that there relationship is much simpler, and also to make sure that they shut down much more neatly when the clerk is busy to avoid crashdumps in ct tests. The CDB now has a binary_mode - so that we don't do binary_to_term twice ... although this may have made things slower ??!!? Perhaps the is_binary check now required on read is an overhead. Perhaps it is some other mystery. There is now a more effiicient fetching of the size on pcl_load now as well.
2016-10-08 22:15:48 +01:00
simple_inker_completeactivejournal_test() ->
RootPath = "../test/journal",
build_dummy_journal(),
Trying to standardise binary manipulation of value Looking into theory that use of term_to_binary is imperfect. Also may be better to compress values only when they are compacted?
2017-03-20 15:43:54 +00:00
CDBopts = #cdb_options{max_size=300000, binary_mode=true},
Improve inker unit test Change in filename labelling had stopped a unit test from covering stratup correctly. Now offering better coverage
2016-10-16 16:58:55 +01:00
JournalFP = filepath(RootPath, journal_dir),
F2 = filename:join(JournalFP, "nursery_3.pnd"),
{ok, PidW} = leveled_cdb:cdb_open_writer(F2),
{ok, _F2} = leveled_cdb:cdb_complete(PidW),
Support for 2i query part1 Added basic support for 2i query. This involved some refactoring of the test code to share functions between suites. There is sill a need for a Part 2 as no tests currently cover removal of index entries.
2016-10-18 01:59:03 +01:00
F1 = filename:join(JournalFP, "nursery_1.cdb"),
F1r = filename:join(JournalFP, "nursery_1.pnd"),
ok = file:rename(F1, F1r),
Code reduction refactor An attempt to refactor out more complex code. The Penciller clerk and Penciller have been re-shaped so that there relationship is much simpler, and also to make sure that they shut down much more neatly when the clerk is busy to avoid crashdumps in ct tests. The CDB now has a binary_mode - so that we don't do binary_to_term twice ... although this may have made things slower ??!!? Perhaps the is_binary check now required on read is an overhead. Perhaps it is some other mystery. There is now a more effiicient fetching of the size on pcl_load now as well.
2016-10-08 22:15:48 +01:00
{ok, Ink1} = ink_start(#inker_options{root_path=RootPath,
cdb_options=CDBopts}),
Obj1 = ink_get(Ink1, "Key1", 1),
?assertMatch({{1, "Key1"}, {"TestValue1", []}}, Obj1),
Obj2 = ink_get(Ink1, "Key4", 4),
?assertMatch({{4, "Key4"}, {"TestValue4", []}}, Obj2),
ink_close(Ink1),
clean_testdir(RootPath).
Testing of Inker rolling Journal Add test to show inker rolling journal. to achieve needs to make CDB size an option, and also alter the manifest sorting so that find_in_manifest actually works!
2016-09-07 17:58:12 +01:00
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
test_ledgerkey(Key) ->
{o, "Bucket", Key, null}.
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
compact_journal_test() ->
RootPath = "../test/journal",
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
build_dummy_journal(fun test_ledgerkey/1),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
CDBopts = #cdb_options{max_size=300000},
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
RStrategy = [{?STD_TAG, recovr}],
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
{ok, Ink1} = ink_start(#inker_options{root_path=RootPath,
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
cdb_options=CDBopts,
reload_strategy=RStrategy}),
{ok, NewSQN1, _ObjSize} = ink_put(Ink1,
test_ledgerkey("KeyAA"),
"TestValueAA", []),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
?assertMatch(NewSQN1, 5),
ok = ink_print_manifest(Ink1),
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
R0 = ink_get(Ink1, test_ledgerkey("KeyAA"), 5),
?assertMatch(R0, {{5, test_ledgerkey("KeyAA")}, {"TestValueAA", []}}),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
FunnyLoop = lists:seq(1, 48),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
Checker = lists:map(fun(X) ->
PK = "KeyZ" ++ integer_to_list(X),
{ok, SQN, _} = ink_put(Ink1,
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
test_ledgerkey(PK),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
crypto:rand_bytes(10000),
[]),
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
{SQN, test_ledgerkey(PK)}
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
end,
FunnyLoop),
Work on reload strategies Further work on variable reload srategies wiht some unit test coverage. Also work on potentially supporting no_hash on PUT to journal files for objects which will never be directly fetched.
2016-10-25 23:13:14 +01:00
{ok, NewSQN2, _ObjSize} = ink_put(Ink1,
test_ledgerkey("KeyBB"),
"TestValueBB", []),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
?assertMatch(NewSQN2, 54),
ActualManifest = ink_getmanifest(Ink1),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
ok = ink_print_manifest(Ink1),
?assertMatch(3, length(ActualManifest)),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
ok = ink_compactjournal(Ink1,
Checker,
Further work on system tests Another issue exposed with laziness in the using an incomplete ledger when checking for presence during compaction.
2016-10-05 18:28:31 +01:00
fun(X) -> {X, 55} end,
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
fun(L, K, SQN) -> lists:member({SQN, K}, L) end,
5000),
timer:sleep(1000),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
CompactedManifest1 = ink_getmanifest(Ink1),
?assertMatch(2, length(CompactedManifest1)),
Checker2 = lists:sublist(Checker, 16),
ok = ink_compactjournal(Ink1,
Checker2,
fun(X) -> {X, 55} end,
fun(L, K, SQN) -> lists:member({SQN, K}, L) end,
5000),
timer:sleep(1000),
CompactedManifest2 = ink_getmanifest(Ink1),
Split out inker manifest - load last key from manifest The process of re-opening file included an expensive search for the LastKey - now the LastKey can be provided out of the manifest.
2017-01-18 15:23:06 +00:00
R = lists:foldl(fun({_SQN, FN, _P, _LK}, Acc) ->
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
case string:str(FN, "post_compact") of
N when N > 0 ->
true;
0 ->
Acc
end end,
false,
CompactedManifest2),
Reduce compaction targets Cmpaction is overly aggressive. It is a lot of work to compact a run of files for just 20% reduction in disk space, when disk space for the Journal (i.e. low IOPS disk space should be relatively inexpensive). Require at least a 40% reduction for a compaction job.
2017-03-30 12:15:36 +01:00
?assertMatch(false, R),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
?assertMatch(2, length(CompactedManifest2)),
Further Journal compaction tests Improved unit testing
2016-09-28 18:26:52 +01:00
ink_close(Ink1),
clean_testdir(RootPath).
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
empty_manifest_test() ->
RootPath = "../test/journal",
clean_testdir(RootPath),
CDBopts = #cdb_options{max_size=300000},
{ok, Ink1} = ink_start(#inker_options{root_path=RootPath,
cdb_options=CDBopts}),
?assertMatch(not_present, ink_fetch(Ink1, "Key1", 1)),
Pick-up test misses There were some coverage misses in tests, so check in unit test coverage or remove branches not currently needed.
2016-11-18 21:35:45 +00:00
CheckFun = fun(L, K, SQN) -> lists:member({SQN, K}, L) end,
?assertMatch(false, CheckFun([], "key", 1)),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
ok = ink_compactjournal(Ink1,
[],
fun(X) -> {X, 55} end,
Pick-up test misses There were some coverage misses in tests, so check in unit test coverage or remove branches not currently needed.
2016-11-18 21:35:45 +00:00
CheckFun,
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
5000),
timer:sleep(1000),
?assertMatch(1, length(ink_getmanifest(Ink1))),
ok = ink_close(Ink1),
Inker test for pending manifest Should ignore the corrupted pending manifest file
2016-11-03 20:00:55 +00:00
% Add pending manifest to be ignored
FN = filepath(RootPath, manifest_dir) ++ "999.pnd",
ok = file:write_file(FN, term_to_binary("Hello")),
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
{ok, Ink2} = ink_start(#inker_options{root_path=RootPath,
cdb_options=CDBopts}),
?assertMatch(not_present, ink_fetch(Ink2, "Key1", 1)),
{ok, SQN, Size} = ink_put(Ink2, "Key1", "Value1", []),
?assertMatch(2, SQN),
?assertMatch(true, Size > 0),
{ok, V} = ink_fetch(Ink2, "Key1", 2),
?assertMatch("Value1", V),
ink_close(Ink2),
clean_testdir(RootPath).
Coverage cheats You juke the stats, and majors become colonels. I've been here before
2016-11-14 20:43:38 +00:00
coverage_cheat_test() ->
{noreply, _State0} = handle_info(timeout, #state{}),
{ok, _State1} = code_change(null, #state{}, null).
Inker Refactor Inker refactored to block on manifest write. If this is inefficient the manifets write can be converted ot an append only operation. Waiting on the manifest write makes the logic at startup much easier to manage.
2016-10-07 18:07:03 +01:00
Inker - Initial Code An attempt to get a first inker that can build a ledger from a manifest as well as support simple get and put operations. Basic tests surround the building of manifests only at this stage - more work required for get and put.
2016-09-05 15:01:23 +01:00
-endif.
Reference in a new issue Copy permalink