Mas d34 leveled.i459 partialmerge (#460)
* Add test to replicate issue 459 Nothing actually crashes due to the issue - but looking at the logs there is the polarised stats associated with the issue. When merging into L3, you would normally expect to merge into 4 files - but actually we see FileCounter occasionally spiking. * Add partial merge support There is a `max_mergebelow` size which can be a positive integer, or infinity. It defaults to 32. If a merge from Level N covers less than `max_mergebelow` files in level N + 1 - the merge will proceesd as before. If it has >= `max_mergebelow`, the merge will be curtailed when `max_mergebelow div 2` files have been created at that level. The remainder for Level N will then be written, as well as for Level N + 1 up to the next whole file that has no yet been touched by the merge. The backlog that prompted the merge will still exist - as the files in Level N have not been changed. However, it is likely the next file picked will not be the same one, and will in probability have a lower number of files to merge (as the average is =< 8). This will stop progress from being halted by long merge jobs, as they will exit out in a safe way after partial completion. In the case where the majority of files covered do not require a merge, then those files will be skipped the next time the remainder file is picked up for merge at Level N
This commit is contained in:
parent
c642575caa
commit
69e8b29d1f
8 changed files with 360 additions and 66 deletions
|
@ -165,7 +165,7 @@
|
|||
pc010 =>
|
||||
{info, <<"Merge to be commenced for FileToMerge=~s with MSN=~w">>},
|
||||
pc011 =>
|
||||
{info, <<"Merge completed with MSN=~w to Level=~w and FileCounter=~w">>},
|
||||
{info, <<"Merge completed with MSN=~w to Level=~w and FileCounter=~w merge_type=~w">>},
|
||||
pc012 =>
|
||||
{debug, <<"File to be created as part of MSN=~w Filename=~s IsBasement=~w">>},
|
||||
pc013 =>
|
||||
|
@ -190,6 +190,8 @@
|
|||
{info, <<"Grooming compaction picked file with tomb_count=~w">>},
|
||||
pc025 =>
|
||||
{info, <<"At level=~w file_count=~w average words for heap_block_size=~w heap_size=~w recent_size=~w bin_vheap_size=~w">>},
|
||||
pc026 =>
|
||||
{info, <<"Performing potential partial to level=~w merge as FileCounter=~w restricting to MaxAdditions=~w">>},
|
||||
pm002 =>
|
||||
{info, <<"Completed dump of L0 cache to list of l0cache_size=~w">>},
|
||||
sst03 =>
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue