umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions

Make tictac more efficient by making level1 a map (#441)

Browse source 
* Make tictac more efficient by making level1 a map

Pre-change (1M keys, tree size large):

Generating Keys took 2513 milliseconds
Memory footprint [{total,356732576},{processes,334051328},{processes_used,334044488},{system,22681248},{atom,540873},{atom_used,524383},{binary,1015120},{code,9692859},{ets,721496}]
Generating new tree took 1 milliseconds
Loading tree took 27967 milliseconds
Memory footprint [{total,36733040},{processes,8875472},{processes_used,8875048},{system,27857568},{atom,540873},{atom_used,524449},{binary,6236480},{code,9692859},{ets,721496}]
Exporting tree took 434 milliseconds
Importing tree took 100 milliseconds
Memory footprint [{total,155941512},{processes,123734808},{processes_used,123734384},{system,32206704},{atom,540873},{atom_used,524449},{binary,10401144},{code,9692859},{ets,721496}]
Garbage collect
Memory footprint [{total,39660504},{processes,8257520},{processes_used,8256968},{system,31402984},{atom,540873},{atom_used,524449},{binary,9781760},{code,9692859},{ets,721496}]

Post change:

Generating Keys took 2416 milliseconds
Memory footprint [{total,284678120},{processes,258349528},{processes_used,257758568},{system,26328592},{atom,893161},{atom_used,878150},{binary,1013880},{code,11770188},{ets,774224}]
Generating new tree took 0 milliseconds
Loading tree took 2072 milliseconds
Memory footprint [{total,49957448},{processes,17244856},{processes_used,16653896},{system,32712592},{atom,893161},{atom_used,878216},{binary,7397496},{code,11770188},{ets,774224}]
Exporting tree took 448 milliseconds
Importing tree took 108 milliseconds
Memory footprint [{total,46504880},{processes,11197344},{processes_used,10606384},{system,35307536},{atom,893161},{atom_used,878216},{binary,9992112},{code,11770188},{ets,774224}]
Garbage collect
Memory footprint [{total,47394048},{processes,12223608},{processes_used,11632520},{system,35170440},{atom,893161},{atom_used,878216},{binary,9855008},{code,11770188},{ets,774224}]

* Tidy-up

* Add type

* Remove ++ requiring copy of Acc

Rely on mechanism producing a sorted result, not sorting

* Update src/leveled_tictac.erl

Co-authored-by: Thomas Arts <thomas.arts@quviq.com>

* Update following review

---------

Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
This commit is contained in:
Martin Sumner 2024-09-03 16:34:41 +01:00
parent acf30599e9
commit af0f2bb2cf
2 changed files with 227 additions and 66 deletions
Download patch file Download diff file Expand all files Collapse all files

8
src/leveled_penciller.erl
View file

@ -159,7 +159,7 @@
-behaviour(gen_server).
-include("include/leveled.hrl").
-include("leveled.hrl").
-export([
init/1,
@ -207,23 +207,17 @@
-export([clean_testdir/1]).
-endif.
-define(MAX_WORK_WAIT, 300).
-define(MANIFEST_FP, "ledger_manifest").
-define(FILES_FP, "ledger_files").
-define(CURRENT_FILEX, "crr").
-define(PENDING_FILEX, "pnd").
-define(SST_FILEX, ".sst").
-define(ARCHIVE_FILEX, ".bak").
-define(SUPER_MAX_TABLE_SIZE, 40000).
-define(PROMPT_WAIT_ONL0, 5).
-define(WORKQUEUE_BACKLOG_TOLERANCE, 4).
-define(COIN_SIDECOUNT, 4).
-define(SLOW_FETCH, 500000). % Log a very slow fetch - longer than 500ms
-define(FOLD_SCANWIDTH, 32).
-define(ITERATOR_SCANWIDTH, 4).
-define(ITERATOR_MINSCANWIDTH, 1).
-define(TIMING_SAMPLECOUNTDOWN, 10000).
-define(TIMING_SAMPLESIZE, 100).
-define(SHUTDOWN_LOOPS, 10).
-define(SHUTDOWN_PAUSE, 10000).
% How long to wait for snapshots to be released on shutdown