umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions

Mas d34 leveled.i459 partialmerge (#460)

Browse source 
* Add test to replicate issue 459

Nothing actually crashes due to the issue - but looking at the logs there is the polarised stats associated with the issue.  When  merging into L3, you would normally expect to merge into 4 files - but actually we see FileCounter occasionally spiking.

* Add partial merge support

There is a `max_mergebelow` size which can be a positive integer, or infinity.  It defaults to 32.

If a merge from Level N covers less than `max_mergebelow` files in level N + 1 - the merge will proceesd as before.  If it has >= `max_mergebelow`, the merge will be curtailed when `max_mergebelow div 2` files have been created at that level.  The remainder for Level N will then be written, as well as for Level N + 1 up to the next whole file that has no yet been touched by the merge.

The backlog that prompted the merge will still exist - as the files in Level N have not been changed.  However, it is likely the next file picked will not be the same one, and will in probability have a lower number of files to merge (as the average is =< 8).

This will stop progress from being halted by long merge jobs, as they will exit out in a safe way after partial completion.  In the case where the majority of files covered  do not require a merge, then those files will be skipped the next time the remainder file is picked up for merge at Level N
This commit is contained in:
Martin Sumner 2024-11-30 13:16:13 +00:00 • committed by GitHub
parent c642575caa
commit 69e8b29d1f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 360 additions and 66 deletions
Download patch file Download diff file Expand all files Collapse all files

8
rebar.config
View file

@ -13,11 +13,7 @@
{eunit_opts, [verbose]}.
{project_plugins, [
{eqwalizer_rebar3,
{git_subdir,
"https://github.com/whatsapp/eqwalizer.git",
{branch, "main"},
"eqwalizer_rebar3"}}
{eqwalizer_rebar3, {git_subdir, "https://github.com/OpenRiak/eqwalizer.git", {branch, "openriak-3.4"}, "eqwalizer_rebar3"}}
]}.
{profiles,
@ -37,7 +33,7 @@
{deps, [
{lz4, ".*", {git, "https://github.com/nhs-riak/erlang-lz4", {branch, "nhse-develop-3.4"}}},
{zstd, ".*", {git, "https://github.com/nhs-riak/zstd-erlang", {branch, "nhse-develop"}}},
{eqwalizer_support, {git_subdir, "https://github.com/whatsapp/eqwalizer.git", {branch, "main"}, "eqwalizer_support"}}
{eqwalizer_support, {git_subdir, "https://github.com/OpenRiak/eqwalizer.git", {branch, "openriak-3.4"}, "eqwalizer_support"}}
]}.
{ct_opts, [{dir, ["test/end_to_end"]}]}.