* Extend perf_SUITE
This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries.
There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself:
e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE
When testing as `perf_prof` summarised versions of the eprof results are now printed to screen.
The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features.
* Load chunk in spawned processes
Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24.
* Correctly account for pause
alos try and improve test stability by increasing pause
* Add microstate accounting to profile
* Add memory tracking during test phases
Identify and log out memory usage by test phase
* Use macros instead (#437)
* Don't print memory to screen in standard ct test
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Add performance/profiling test
Add test to perf_SUITE to do performance tests and also profile different activities in leveled.
This can then be used to highlight functions with unexpectedly high execution times, and prove the impact of changes.
Switch between riak_ctperf and riak_fullperf to change from standard test (with profile option) to full-scale performance test
* Change shape of default perfTest
* Refactor SST
Compare and contrast profile for guess, before and after refactor:
pre
```
lists:map_1/2 313370 2.33 32379 [ 0.10]
lists:foldl_1/3 956590 4.81 66992 [ 0.07]
leveled_sst:'-expand_list_by_pointer/5-fun-0-'/4 925020 6.13 85318 [ 0.09]
erlang:binary_to_term/1 3881 8.55 119012 [ 30.67]
erlang:'++'/2 974322 11.55 160724 [ 0.16]
lists:member/2 4000180 15.00 208697 [ 0.05]
leveled_sst:find_pos/4 4029220 21.01 292347 [ 0.07]
leveled_sst:member_check/2 4000000 21.17 294601 [ 0.07]
-------------------------------------------------- -------- ------- ------- [----------]
Total: 16894665 100.00% 1391759 [ 0.08]
```
post
```
lists:map_1/2 63800 0.79 6795 [ 0.11]
erlang:term_to_binary/1 15726 0.81 6950 [ 0.44]
lists:keyfind/3 180967 0.92 7884 [ 0.04]
erlang:spawn_link/3 15717 1.08 9327 [ 0.59]
leveled_sst:'-read_slots/5-fun-1-'/8 31270 1.15 9895 [ 0.32]
gen:do_call/4 7881 1.31 11243 [ 1.43]
leveled_penciller:find_nextkey/8 180936 2.01 17293 [ 0.10]
prim_file:pread_nif/3 15717 3.89 33437 [ 2.13]
leveled_sst:find_pos/4 4028940 17.85 153554 [ 0.04]
erlang:binary_to_term/1 15717 51.97 447048 [ 28.44]
-------------------------------------------------- ------- ------- ------ [----------]
Total: 6704100 100.00% 860233 [ 0.13]
```
* Update leveled_penciller.erl
* Mas d31 nhskv16sstpcl (#426)
Performance updates to leveled:
- Refactoring of pointer expansion when fetching from leveled_sst files to avoid expensive list concatenation.
- Refactoring of leveled_ebloom to make more flexible, reduce code, and improve check time.
- Refactoring of querying within leveled_sst to reduce the number of blocks that need to be de-serialised per query.
- Refactoring of the leveled_penciller's query key comparator, to make use of maps and simplify the filtering.
- General speed-up of frequently called functions.
* Mas i410 looptoclose (#420)
* Stop waiting full SHUTDOWN_PAUSE
If there is a snapshot outstanding at shutdown time, there was a wait of SHUTDOWN_PAUSE to give the snapshot time to close down.
This causes an issue in kv_index_tictactree when rebuilds complete, when an exchange was in flight at the point the rebuild completed - the aae_controller will become blocked for the full shutdown pause, whilst it waits for the replaced key store to be closed.
This change is to loop within the shutdown pause, so that if the snapshot supporting the exchange is closed, the paused bookie can close more quickly (unblocking the controller).
Without this fix, there are intermittent issues in kv_index_tictactree's mockvnode_SUITE tests.
* Address test reliability
Be a bit clearer with waiting round seconds, Was intermittently failing on QR4 previously (but QR5 1s later was always OK).
* Update iterator_SUITE.erl
* Refine test assertion
At Stage C there might be 0 files left, in which case equality with Stage D result is ok.
* Add compression controls (#417)
* Add compression controls
Add configuration options to allow for a compression algorithm of `none` to disable compression altogether. Also an option to change the point in the LSM tree when compression is applied.
* Handle configurable defaults consistently
Move them into leveled.hrl. This forces double-definitions to be resolved.
There are some other constants in leveled_bookie that are relevant outside of leveled_bookie. These are all now in the non-configurable startup defaults section.
* Clarify referred-to default is OTP not leveled
* Update leveled_bookie.erl
Handle xref issue with eunit include
To show how this works, and prove that it does work thta way.
Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
The function should see the buckets in order, so it accumulates in such a way to reverse the order - it makes sense that the outcome should be in reverse.
This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected.
This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception). This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
If random integer was low, total could be below threshold - so calculate total correctly.
Should make value re-generate random uniform, but test is still valid without this
As the fold functions have been added to get_runner in an ad hoc way,
naturally, given the ongoing development of levelEd to support Riak,
it was difficult for a new user (in this case Quviq) to see what folds
are supported, and with what arguments, and expectations.
This PR is for discussion. It is one of many ways to group, spec, and
document the fold functions.
A test is also added for coverage of range queries.
Introduce a dedicated module for all the different fold types. Also simplify the list of folders by deprecating those folds that should eb achieveable by fold_heads/fold_objects type folds but with smarter functions.
Makes sure that the fold functiosn also have better spec coverage, and are dialyzer checked.
the new code requires bucket listing to be on binary keys not just
binary buckets. As this is only intended for use within Riak (where
all keys are buckets are binaries), this constraint seems OK.
A test needed changing to ensure it had a binary key in the bucket.
Leveled will now signal the need for a pause due to back-pressure, but
not actually pause itself. The hope is that in a riak implementation
this pause can be managed by the put_fsm, and so not lock the store.
Clena the API of Riak specific methods, and also resolve timing issue in
simple_server unit test. Previously this would end up with missing data
(and a lower sequence number after start) because of the penciller_clerk
timeout being relatively large in the context of this test. Now the
timeout has bene reduced the L0 slot is cleared by the time of the
close. To make sure an extra sleep has been added as a precaution to
avoid any intermittent issues.
Changes the stratup otpions to a prolist to make it easier to get
environment variables as default.
Tried application:start - and completely baffled as to how to get this
to work.
Test added for the "retain" recovery strategy. This strategy makes sure
a full history of index changes is made so that if the Ledger is wiped
out, the Ledger cna be fully rebuilt from the Journal.
This exposed two journal compaction problems
- The BestRun selected did not have the source files correctly sorted in
order before compaction
- The compaction process incorrectly dealt with the KeyDelta object
left after a compaction - i.e. compacting twice the same key caused that
key history to be lost.
These issues have now been corrected.
The unit tests for the Penciller couldn't cope with the returned status
- and so would intermittently fail (after tightening the timeout on sft
check_ready.
There was a test that failed to close down a bookie and that caused some
issues. The issues are double-reoslved, the close down was tidied as
well as the forgotten close being added back in.
There is some generla tidy around in anticipation of TTL support.
Recent fixes have been made to problems associated with rapidly changing
objexts especially on re-opening of the bookie. Test of rotating
objects from both an index query and a fetch perspective added to better
detect such issues in the future.
The penciller had the concept of a manifest_lock - but it wasn't clear
what the purpose of it was.
The updating of the manifest has now been updated to reduce the code and
make the process cleaner and more obvious. Now the committed manifest
only covers non-L0 levels. A clerk can work concurrently on a manifest
change whilst the Penciller is accepting a new L0 file.
On startup the manifets is opened as well as any L0 file. There is a
possible race condition with killing process where there may be a L0
file which is merged but undeleted - and this is believed to be inert.
There is some outstanding work still. Currently the whole store is
paused if a push_mem is received by the Penciller, and the writing of a
L0 sft file has not been completed. The creation of a L0 file appears
to take about 300ms, so if the ledger_cache fills in this period a pause
will occurr (perhaps due to objects with lots of index entries). It
would be preferable to pause more elegantly in this situation. Perhaps
there should be a harsh timeout on the call to check the SFT complete,
and catching it should cause a refused response. The next PUT will then
wait, but a any queued GETs can progress.
The 2i work now has tests for removals as well as regex etc.
Some initial refactoring work has also been tried - to try and take some
tasks of the critical path of push_mem. The primary change has been to
avoid putting index keys into the gb_tree, and building the KeyChanges
list in parallel to the gb_tree (now known as ObjectTree) within the
Ledger Cache.
Some initial experiments done as to changing the ETS table in the
Penciller now that it will now be used for iterating - but that has been
reverted for now.
Added basic support for 2i query. This involved some refactoring of the
test code to share functions between suites.
There is sill a need for a Part 2 as no tests currently cover removal of
index entries.
