* Add eqwalizer and clear for codec & sst
The eqwalizer errors highlighted the need in several places for type clarification.
Within tests there are some issue where a type is assumed, and so ignore has been used to handle this rather than write more complex code to be explicit about the assumption.
The handling of arrays isn't great by eqwalizer - to be specific about the content of array causes issues when initialising an array. Perhaps a type (map maybe) where one can be more explicit about types might be a better option (even if there is a minimal performance impact).
The use of a ?TOMB_COUNT defined option complicated the code much more with eqwalizer. So for now, there is no developer option to disable ?TOMB_COUNT.
Test fixes required where strings have been used for buckets/keys not binaries.
The leveled_sst statem needs a different state record for starting when compared to other modes. The state record has been divided up to reflect this, to make type management easier. The impact on performance needs to be tested.
* Update ct tests to support binary keys/buckets only
* Eqwalizer for leveled_cdb and leveled_tictac
As array is used in leveled_tictac - there is the same issue as with leveled_sst
* Remove redundant indirection of leveled_rand
A legacy of pre-20 OTP
* Morde modules eqwalized
ebloom/log/util/monitor
* Eqwalize further modules
elp eqwalize leveled_codec; elp eqwalize leveled_sst; elp eqwalize leveled_cdb; elp eqwalize leveled_tictac; elp eqwalize leveled_log; elp eqwalize leveled_monitor; elp eqwalize leveled_head; elp eqwalize leveled_ebloom; elp eqwalize leveled_iclerk
All concurrently OK
* Refactor unit tests to use binary() no string() in key
Previously string() was allowed just to avoid having to change all these tests. Go through the pain now, as part of eqwalizing.
* Add fixes for penciller, inker
Add a new ?IS_DEF macro to replace =/= undefined.
Now more explicit about primary, object and query keys
* Further fixes
Need to clarify functions used by runner - where keys , query keys and object keys are used
* Further eqwalisation
* Eqwalize leveled_pmanifest
Also make implementation independent of choice of dict - i.e. one can save a manifest using dict for blooms/pending_deletions and then open a manifest with code that uses a different type. Allow for slow dict to be replaced with map.
Would not be backwards compatible though, without further thought - i.e. if you upgrade then downgrade.
Redundant code created by leveled_sst refactoring removed.
* Fix backwards compatibility issues
* Manifest Entry to belong to leveled_pmanifest
There are two manifests - leveled_pmanifest and leveled_imanifest. Both have manifest_entry() type objects, but these types are different. To avoid confusion don't include the pmanifest manifest_entry() within the global include file - be specific that it belongs to the leveled_pmanifest module
* Ignore elp file - large binary
* Update src/leveled_pmem.erl
Remove unnecessary empty list from type definition
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Switch to logger
Use logger rather than io:format when logging. The ct tests have besn switched to log to file, testutil/init_per_suite/1 may offer useful guidance on configuring logger with leveled.
As all logs are produced by the leveled_log module, the MFA metadata is uninteresting for log outputs, but can be used for explicit filter controls for leveled logs.
* iolist_to_binary not unicode_binary()
logger filters will be error and be removed if the format line is a binary(). Must be either a charlist() or a unicode_binary() - so iolist_to_binary() can't be used
* Add metadata for filter
* Update test/end_to_end/tictac_SUITE.erl
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Improve perf_SUITE test
The update teat is refactored so as not to generate. a large KV list which dominates the memory utilisation.
The update and the get tests changes to do a head before each operation - which emulates how this will work in RIAK.
* Revert default setting change
* Don't pre-calculate key list
For fetches - reduces memory required for test process not database (and consequent distortion to measured results)
* Tidy ++ in tests
Removes some rogue results from profile
* Update testutil.erl
* Test fixes
* Tidy generate_chunk for profiling
* Revert "Tidy generate_chunk for profiling"
This reverts commit 1f6cff446ca6b9855f1e3aa732b32e0e5c14c9a5.
* Resize profile test
* Extend perf_SUITE
This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries.
There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself:
e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE
When testing as `perf_prof` summarised versions of the eprof results are now printed to screen.
The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features.
* Load chunk in spawned processes
Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24.
* Correctly account for pause
alos try and improve test stability by increasing pause
* Add microstate accounting to profile
* Add memory tracking during test phases
Identify and log out memory usage by test phase
* Use macros instead (#437)
* Don't print memory to screen in standard ct test
---------
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
* Add support for zstd and split compression
Add support for using zstd as an alternative to native, lz4.
Upgrade lz4 to v1.9.4 (with ARM enhancements).
Allow for split compression algorithms - i.e. use native on journal, but lz4 on ledger.
* Switch to AdRoll zstd
Development appears to be active and ongoing. No issues running on different linux flavours.
* Use realistic bucket name
* Update README.md
* Switch branch
* Add comment following review
* Add performance/profiling test
Add test to perf_SUITE to do performance tests and also profile different activities in leveled.
This can then be used to highlight functions with unexpectedly high execution times, and prove the impact of changes.
Switch between riak_ctperf and riak_fullperf to change from standard test (with profile option) to full-scale performance test
* Change shape of default perfTest
* Change fullPerf
Change the fullPerf test to run more tests, but with fewer keys.
Given that RS of 512 is being pushed in Riak, 2M objects is till a 300M+ object cluster. 10M >> 1B. so these are still reasonable sizes to test.
A profilePerf test also added to generate all the profiles base don 2M objects.
* Extend test
Queries where previously all returning a large number of index entries - changes made to make number of entries per query more realistic. Also an update process added to show difference between loading and rotating keys.
* Relabel as AAE fold
* Test v5
Test mini-queries - where generally a small number of entries are returned
* Default to ctperf
Resolve issue with OTP 22 performance https://github.com/martinsumner/leveled/issues/326 - by changing refernces to loop state.
The test perf_SUITE proves the issue.
OTP 22, without fixes:
Fold pre-close 41209 ms post-close 688 ms
OTP 22, with fixes:
Fold pre-close 401 ms post-close 317 ms
