* Support sub-key queries
Also requires a refactoring of types.
In head-only mode - the metadata in the ledger is just the value, and the value can be anything. So metadata() definition needs to reflect that.
There are then issues with appdefined functions for extracting metadata. In theory an appdefined function could extract some unsopprted type. So made explicit that the appdefined function must extract std_metadata() as metadata - otherwise functionality will not work.
This means that if it is an object key, that is not a ?HEAD key, then the Metadata must be a tuple (of either Riak or Standard type).
* Fix coverage issues
* Add eqwalizer and clear for codec & sst
The eqwalizer errors highlighted the need in several places for type clarification.
Within tests there are some issue where a type is assumed, and so ignore has been used to handle this rather than write more complex code to be explicit about the assumption.
The handling of arrays isn't great by eqwalizer - to be specific about the content of array causes issues when initialising an array. Perhaps a type (map maybe) where one can be more explicit about types might be a better option (even if there is a minimal performance impact).
The use of a ?TOMB_COUNT defined option complicated the code much more with eqwalizer. So for now, there is no developer option to disable ?TOMB_COUNT.
Test fixes required where strings have been used for buckets/keys not binaries.
The leveled_sst statem needs a different state record for starting when compared to other modes. The state record has been divided up to reflect this, to make type management easier. The impact on performance needs to be tested.
* Update ct tests to support binary keys/buckets only
* Eqwalizer for leveled_cdb and leveled_tictac
As array is used in leveled_tictac - there is the same issue as with leveled_sst
* Remove redundant indirection of leveled_rand
A legacy of pre-20 OTP
* Morde modules eqwalized
ebloom/log/util/monitor
* Eqwalize further modules
elp eqwalize leveled_codec; elp eqwalize leveled_sst; elp eqwalize leveled_cdb; elp eqwalize leveled_tictac; elp eqwalize leveled_log; elp eqwalize leveled_monitor; elp eqwalize leveled_head; elp eqwalize leveled_ebloom; elp eqwalize leveled_iclerk
All concurrently OK
* Refactor unit tests to use binary() no string() in key
Previously string() was allowed just to avoid having to change all these tests. Go through the pain now, as part of eqwalizing.
* Add fixes for penciller, inker
Add a new ?IS_DEF macro to replace =/= undefined.
Now more explicit about primary, object and query keys
* Further fixes
Need to clarify functions used by runner - where keys , query keys and object keys are used
* Further eqwalisation
* Eqwalize leveled_pmanifest
Also make implementation independent of choice of dict - i.e. one can save a manifest using dict for blooms/pending_deletions and then open a manifest with code that uses a different type. Allow for slow dict to be replaced with map.
Would not be backwards compatible though, without further thought - i.e. if you upgrade then downgrade.
Redundant code created by leveled_sst refactoring removed.
* Fix backwards compatibility issues
* Manifest Entry to belong to leveled_pmanifest
There are two manifests - leveled_pmanifest and leveled_imanifest. Both have manifest_entry() type objects, but these types are different. To avoid confusion don't include the pmanifest manifest_entry() within the global include file - be specific that it belongs to the leveled_pmanifest module
* Ignore elp file - large binary
* Update src/leveled_pmem.erl
Remove unnecessary empty list from type definition
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
The old leveled_tictac had a pure binary L1. this was slower than the new map version.
However, in a Riak cluster, when running a merge_tree_range during a rolling update, the fold the query coordinator will initiate a tree. If this tree is not a map-based tree (as that node has not yet been upgraded), then a node that has been upgraded would previously fail the query as it cannot handle a level 1 in a binary form. This now enables updated nodes to handle both forms of trees.
Obviously, if the coordinating node has been updated non-updated nodes will crash queries as they cannot handle the tree with the map at Level 1. The aim is to make it configurable to force non-map trees in a cluster, until all nodes have been upgraded. So as long as each node understands how to update both non-map trees and map-based trees - evrything should be OK.
* Switch to logger
Use logger rather than io:format when logging. The ct tests have besn switched to log to file, testutil/init_per_suite/1 may offer useful guidance on configuring logger with leveled.
As all logs are produced by the leveled_log module, the MFA metadata is uninteresting for log outputs, but can be used for explicit filter controls for leveled logs.
* iolist_to_binary not unicode_binary()
logger filters will be error and be removed if the format line is a binary(). Must be either a charlist() or a unicode_binary() - so iolist_to_binary() can't be used
* Add metadata for filter
* Update test/end_to_end/tictac_SUITE.erl
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Extend perf_SUITE
This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries.
There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself:
e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE
When testing as `perf_prof` summarised versions of the eprof results are now printed to screen.
The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features.
* Load chunk in spawned processes
Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24.
* Correctly account for pause
alos try and improve test stability by increasing pause
* Add microstate accounting to profile
* Add memory tracking during test phases
Identify and log out memory usage by test phase
* Use macros instead (#437)
* Don't print memory to screen in standard ct test
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Allow snapshots to be reused in queries
Allow for a full bookie snapshot to be re-used for multiple queries, not just KV fetches.
* Reduce log noise
The internal dummy tag is expected so should not prompt a log on reload
* Snapshot should have same status of active db
wrt head_only and head_lookup
* Allow logging to specified on snapshots
* Shutdown snapshot bookie is primary goes down
Inker and Penciller already will shut down based on `erlang:monitor/2`
* Review feedback
Formatting and code readability fixes
This allows for leveled_iclerk:clerk_stop to be a sync call, so that files will only be closed once the iclerk has stopped. This is designed ot prevent iclerk crashes during shutdowns when files it is depnding on are closed mid shutdown.
This helps with kv_index_tictcatree with the leveled_so backend. Now this cna do folds over ranges of keys with modified filters (as folds over ranges of keys must go over lal keys if the backend is segment_ordered)
As the fold functions have been added to get_runner in an ad hoc way,
naturally, given the ongoing development of levelEd to support Riak,
it was difficult for a new user (in this case Quviq) to see what folds
are supported, and with what arguments, and expectations.
This PR is for discussion. It is one of many ways to group, spec, and
document the fold functions.
A test is also added for coverage of range queries.
head_only mode cna be run with_lookup - but there is no L0 index created in this case.
So the L0 index wasn't returning a potition list and the L0 cache wasn't being checked.
Code now checks every position in the L0 cache, when a lookup is attempted in head_only mode.
Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook.
Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting).
So reverted to re-adding support for HEAD requests with these keys.
Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger).
The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
Idea being that sometimes you may wish to compare a tictac tree between leveled and something that doesn't understand erlang:phash or term_to_binary. So allow the magic_hash to be used instead - and perhaps an extract function that does base64 encoding or something similar.
The "small" tree will serialise to 1.5MB - which seems large. Much smaller trees seem to be more suitable for things like recently modified aae indexes.
Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads.
The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU.
However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends.
Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak.
In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend.
The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
Need to know {Bucket, Key} not just Key if all buckets are being covered
by nrt aae. So shoehorning this in - will also allow for proper use of
FilterFun when filtering by partition.
With basic ct test.
Doesn't currently prove expiry of index. Doesn't prove ability to find
segments.
Assumes that either "all" buckets or a special list of buckets require
indexing this way. Will lead to unexpected results if the same bucket
name is used across different Tags.
The format of the index has been chosen so that hopeully standard index
features can be used (e.g. return_terms).
Just some initial WIP code for this. Will revisit this again after
exploring some ideas as to how to reduce the cost of the
get_keys_by_segment.
The overlal idea is that there are trees of recent modifications, with
recent being some rolling time window made up of hourly blocks, and
recency being dtermined by the last-modified date on the object metadata
- which should be conistent across a cluster.
So if we were at 15:30 we would get the tree for 14:00 - 15:00 and the
tree for 15:00-16:00 from two different queries which cover the same
partitions and then compare.
Comparison may find differences, and we know what segment the difference
is in - but how to then find all keys in that segment which have been
modified in the period? Three ways:
Do it inefficeintly and infrequently using a fold_keys and a filter
(perhaps with SST files having a highest LMD in the metadata so that
they can be skipped).
Add a special index, where verye entry has a TTL, and the Key is
{$segment, Segment, Bucket, Key} so that a normal 2i query cna be used.
Align hashing for segments with hashing for penciller lookup so that a
query over the actual keys cna be optimised skipping chunks of the
in-memory part, and chunks of the SST file
