* Switch to logger
Use logger rather than io:format when logging. The ct tests have besn switched to log to file, testutil/init_per_suite/1 may offer useful guidance on configuring logger with leveled.
As all logs are produced by the leveled_log module, the MFA metadata is uninteresting for log outputs, but can be used for explicit filter controls for leveled logs.
* iolist_to_binary not unicode_binary()
logger filters will be error and be removed if the format line is a binary(). Must be either a charlist() or a unicode_binary() - so iolist_to_binary() can't be used
* Add metadata for filter
* Update test/end_to_end/tictac_SUITE.erl
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Improve perf_SUITE test
The update teat is refactored so as not to generate. a large KV list which dominates the memory utilisation.
The update and the get tests changes to do a head before each operation - which emulates how this will work in RIAK.
* Revert default setting change
* Don't pre-calculate key list
For fetches - reduces memory required for test process not database (and consequent distortion to measured results)
* Tidy ++ in tests
Removes some rogue results from profile
* Update testutil.erl
* Test fixes
* Tidy generate_chunk for profiling
* Revert "Tidy generate_chunk for profiling"
This reverts commit 1f6cff446ca6b9855f1e3aa732b32e0e5c14c9a5.
* Resize profile test
* Extend perf_SUITE
This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries.
There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself:
e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE
When testing as `perf_prof` summarised versions of the eprof results are now printed to screen.
The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features.
* Load chunk in spawned processes
Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24.
* Correctly account for pause
alos try and improve test stability by increasing pause
* Add microstate accounting to profile
* Add memory tracking during test phases
Identify and log out memory usage by test phase
* Use macros instead (#437)
* Don't print memory to screen in standard ct test
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Add support for zstd and split compression
Add support for using zstd as an alternative to native, lz4.
Upgrade lz4 to v1.9.4 (with ARM enhancements).
Allow for split compression algorithms - i.e. use native on journal, but lz4 on ledger.
* Switch to AdRoll zstd
Development appears to be active and ongoing. No issues running on different linux flavours.
* Use realistic bucket name
* Update README.md
* Switch branch
* Add comment following review
* Add performance/profiling test
Add test to perf_SUITE to do performance tests and also profile different activities in leveled.
This can then be used to highlight functions with unexpectedly high execution times, and prove the impact of changes.
Switch between riak_ctperf and riak_fullperf to change from standard test (with profile option) to full-scale performance test
* Change shape of default perfTest
* Change fullPerf
Change the fullPerf test to run more tests, but with fewer keys.
Given that RS of 512 is being pushed in Riak, 2M objects is till a 300M+ object cluster. 10M >> 1B. so these are still reasonable sizes to test.
A profilePerf test also added to generate all the profiles base don 2M objects.
* Extend test
Queries where previously all returning a large number of index entries - changes made to make number of entries per query more realistic. Also an update process added to show difference between loading and rotating keys.
* Relabel as AAE fold
* Test v5
Test mini-queries - where generally a small number of entries are returned
* Default to ctperf
Resolve issue with OTP 22 performance https://github.com/martinsumner/leveled/issues/326 - by changing refernces to loop state.
The test perf_SUITE proves the issue.
OTP 22, without fixes:
Fold pre-close 41209 ms post-close 688 ms
OTP 22, with fixes:
Fold pre-close 401 ms post-close 317 ms
