umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions

Mas i335 otp24 (#336)

Browse source 
* Address OTP24 warnings, ct and eunit paths

* Reorg to add OTP 24 support

* Update VOLUME.md

* Correct broken refs

* Update README.md

* CI on all main branches

Co-authored-by: Ulf Wiger <ulf@wiger.net>
This commit is contained in:
Martin Sumner 2021-05-25 13:41:20 +01:00 • committed by GitHub
parent 2c53c0a85a
commit ed0301e2cf
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
43 changed files with 99 additions and 553 deletions
Download patch file Download diff file Expand all files Collapse all files

52
README.md
View file

@ -44,48 +44,13 @@ For more details on the store:
- There is also a ["Why"](docs/WHY.md) section looking at lower level design choices and the rationale that supports them.
## Is this interesting?
Making a positive contribution to this space is hard - given the superior brainpower and experience of those that have contributed to the KV store problem space in general, and the Riak backend space in particular.
The target at inception was to do something interesting, to re-think certain key assumptions and trade-offs, and prove through working software the potential for improvements to be realised.
[Initial volume tests](docs/VOLUME.md) indicate that it is at least interesting. With improvements in throughput for multiple configurations, with this improvement becoming more marked as the test progresses (and the base data volume becomes more realistic).
The delta in the table below is the comparison in Riak throughput between the identical test run with a leveled backend in comparison to leveldb. The realism of the tests increase as the test progresses - so focus is given to the throughput delta in the last hour of the test.
Test Description | Hardware | Duration |Avg TPS | TPS Delta (Overall) | TPS Delta (Last Hour)
:---------------------------------|:-------------|:--------:|----------:|-----------------:|-------------------:
8KB value, 60 workers, sync | 5 x i2.2x | 4 hr | 12,679.91 | <b>+ 70.81%</b> | <b>+ 63.99%</b>
8KB value, 100 workers, no_sync | 5 x i2.2x | 6 hr | 14,100.19 | <b>+ 16.15%</b> | <b>+ 35.92%</b>
8KB value, 50 workers, no_sync | 5 x d2.2x | 4 hr | 10,400.29 | <b>+ 8.37%</b> | <b>+ 23.51%</b>
4KB value, 100 workers, no_sync | 5 x i2.2x | 6 hr | 14,993.95 | - 10.44% | - 4.48%
16KB value, 60 workers, no_sync | 5 x i2.2x | 6 hr | 11,167.44 | <b>+ 80.48%</b> | <b>+ 113.55%</b>
8KB value, 80 workers, no_sync, 2i queries | 5 x i2.2x | 6 hr | 9,855.96 | <b>+ 4.48%</b> | <b>+ 22.36%</b>
Tests generally show a 5:1 improvement in tail latency for leveled.
All tests have in common:
- Target Key volume - 200M with pareto distribution of load
- 5 GETs per 1 update
- RAID 10 (software) drives
- allow_mult=false, lww=false
- modified riak optimised for leveled used in leveled tests
The throughput in leveled is generally CPU-bound, whereas in comparative tests for leveledb the throughput was disk bound. This potentially makes capacity planning simpler, and opens up the possibility of scaling out to equivalent throughput at much lower cost (as CPU is relatively low cost when compared to disk space at high I/O) - [offering better alignment between resource constraints and the cost of resource](docs/INTRO.md).
More information can be found in the [volume testing section](docs/VOLUME.md).
As a general rule though, the most interesting thing is the potential to enable [new features](docs/FUTURE.md). The tagging of different object types, with an ability to set different rules for both compaction and metadata creation by tag, is a potential enabler for further change. Further, having a separate key/metadata store which can be scanned without breaking the page cache or working against mitigation for write amplifications, is also potentially an enabler to offer features to both the developer and the operator.
## Feedback
Please create an issue if you have any suggestions. You can ping me <b>@masleeds</b> if you wish
## Running Leveled
Unit and current tests in leveled should run with rebar3. Leveled has been tested in OTP18, but it can be started with OTP16 to support Riak (although tests will not work as expected).
Unit and current tests in leveled should run with rebar3.
A new database can be started by running
@ -99,13 +64,18 @@ The book_start method should respond once startup is complete. The [leveled_boo
Running in Riak requires Riak 2.9 or beyond, which is available from January 2019.
There are three main branches:
[`develop-3.1` - default](https://github.com/martinsumner/leveled/tree/develop-3.1): Target for the Riak 3.1 release with support for OTP 22 and OTP 24;
[`develop-3.0`](https://github.com/martinsumner/leveled/tree/develop-3.0): Used in the Riak 3.0 release with support for OTP 20 and OTP 22;
[`develop-2.9`](https://github.com/martinsumner/leveled/tree/develop-2.9): Used in the Riak 2.9 release with support for OTP R16 through to OTP 20.
### Contributing
In order to contribute to leveled, fork the repository, make a branch
for your changes, and open a pull request. The acceptance criteria for
updating leveled is that it passes rebar3 dialyzer, xref, eunit, and
ct with 100% coverage.
In order to contribute to leveled, fork the repository, make a branch for your changes, and open a pull request. The acceptance criteria for updating leveled is that it passes rebar3 dialyzer, xref, eunit, and ct with 100% coverage.
To have rebar3 execute the full set of tests, run:
`rebar3 as test do cover --reset, eunit --cover, ct --cover, cover --verbose`
`rebar3 as test do xref, dialyzer, cover --reset, eunit --cover, ct --cover, cover --verbose`