* Switch to logger
Use logger rather than io:format when logging. The ct tests have besn switched to log to file, testutil/init_per_suite/1 may offer useful guidance on configuring logger with leveled.
As all logs are produced by the leveled_log module, the MFA metadata is uninteresting for log outputs, but can be used for explicit filter controls for leveled logs.
* iolist_to_binary not unicode_binary()
logger filters will be error and be removed if the format line is a binary(). Must be either a charlist() or a unicode_binary() - so iolist_to_binary() can't be used
* Add metadata for filter
* Update test/end_to_end/tictac_SUITE.erl
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Extend perf_SUITE
This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries.
There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself:
e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE
When testing as `perf_prof` summarised versions of the eprof results are now printed to screen.
The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features.
* Load chunk in spawned processes
Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24.
* Correctly account for pause
alos try and improve test stability by increasing pause
* Add microstate accounting to profile
* Add memory tracking during test phases
Identify and log out memory usage by test phase
* Use macros instead (#437)
* Don't print memory to screen in standard ct test
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Allow snapshots to be reused in queries
Allow for a full bookie snapshot to be re-used for multiple queries, not just KV fetches.
* Reduce log noise
The internal dummy tag is expected so should not prompt a log on reload
* Snapshot should have same status of active db
wrt head_only and head_lookup
* Allow logging to specified on snapshots
* Shutdown snapshot bookie is primary goes down
Inker and Penciller already will shut down based on `erlang:monitor/2`
* Review feedback
Formatting and code readability fixes
This allows for leveled_iclerk:clerk_stop to be a sync call, so that files will only be closed once the iclerk has stopped. This is designed ot prevent iclerk crashes during shutdowns when files it is depnding on are closed mid shutdown.
This helps with kv_index_tictcatree with the leveled_so backend. Now this cna do folds over ranges of keys with modified filters (as folds over ranges of keys must go over lal keys if the backend is segment_ordered)
As the fold functions have been added to get_runner in an ad hoc way,
naturally, given the ongoing development of levelEd to support Riak,
it was difficult for a new user (in this case Quviq) to see what folds
are supported, and with what arguments, and expectations.
This PR is for discussion. It is one of many ways to group, spec, and
document the fold functions.
A test is also added for coverage of range queries.
head_only mode cna be run with_lookup - but there is no L0 index created in this case.
So the L0 index wasn't returning a potition list and the L0 cache wasn't being checked.
Code now checks every position in the L0 cache, when a lookup is attempted in head_only mode.
Originally had disabled the ability to lookup individual values when running in head_only mode. This is a saving of about 11% at PUT time (about 3 microseconds per PUT) on a macbook.
Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting).
So reverted to re-adding support for HEAD requests with these keys.
Initial commit to add head_only mode to leveled. This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger).
The aim is to reduce significantly the cost of compaction. Also, the objects ar enot directly accessible (they can only be accessed through folds). Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
Idea being that sometimes you may wish to compare a tictac tree between leveled and something that doesn't understand erlang:phash or term_to_binary. So allow the magic_hash to be used instead - and perhaps an extract function that does base64 encoding or something similar.
The "small" tree will serialise to 1.5MB - which seems large. Much smaller trees seem to be more suitable for things like recently modified aae indexes.
Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads.
The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU.
However, this does give the flexibility to change the hash algorithm. This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends.
Also can vary whether the fold_heads checks for presence of the object in the Inker. So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
Build the AAE tree equally using fold_heads. This is a pre-cursor to running this within Riak.
In part this leans on some of the work done to improve standard Riak AAE with leveled. When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend.
The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
Need to know {Bucket, Key} not just Key if all buckets are being covered
by nrt aae. So shoehorning this in - will also allow for proper use of
FilterFun when filtering by partition.
With basic ct test.
Doesn't currently prove expiry of index. Doesn't prove ability to find
segments.
Assumes that either "all" buckets or a special list of buckets require
indexing this way. Will lead to unexpected results if the same bucket
name is used across different Tags.
The format of the index has been chosen so that hopeully standard index
features can be used (e.g. return_terms).
Just some initial WIP code for this. Will revisit this again after
exploring some ideas as to how to reduce the cost of the
get_keys_by_segment.
The overlal idea is that there are trees of recent modifications, with
recent being some rolling time window made up of hourly blocks, and
recency being dtermined by the last-modified date on the object metadata
- which should be conistent across a cluster.
So if we were at 15:30 we would get the tree for 14:00 - 15:00 and the
tree for 15:00-16:00 from two different queries which cover the same
partitions and then compare.
Comparison may find differences, and we know what segment the difference
is in - but how to then find all keys in that segment which have been
modified in the period? Three ways:
Do it inefficeintly and infrequently using a fold_keys and a filter
(perhaps with SST files having a highest LMD in the metadata so that
they can be skipped).
Add a special index, where verye entry has a TTL, and the Key is
{$segment, Segment, Bucket, Key} so that a normal 2i query cna be used.
Align hashing for segments with hashing for penciller lookup so that a
query over the actual keys cna be optimised skipping chunks of the
in-memory part, and chunks of the SST file
Allow tictac tree sizes to be flexible.
Tested lots of different sizes. Having both level 1 and level 2 the
same size seemed to be consistently quicker than trying to make either
of the levels relatively wider.
There's an 8% performance improvement if the SegmentCount is reduced by
a quarter.
