Smoother handling of back-pressure

The Penciller had two problems in previous commits:
- If it had a push_mem soon after a L0 file had been created, the
push_mem would stall waiting for the L0 file to complete - and this
count take 100-200ms
- The penciller's clerk favoured L0 work, but was lazy about asking for
other work in-between, so often the L1 layer was bursting over capacity
and the clerk was doing nothing but merging more L0 files in (with those
merges getting more and more expensive as they had to cover more and
more files)

There are some partial resolutions to this.  There is now an aggressive
timeout when checking whther the L0 file is ready on a push_mem, and if
the timeout is breached the error is caught and a 'returned' message
goes back to the Bookie.  the Bookie doesn't now empty its cache, it
carrie son filling it, but on some probability it will keep trying to
push_mem on future pushes.  This increases Jitter around the expensive
operation and split out the L0 delay into defined chunks.

The penciller's clerk is now more aggressive in asking for work.  There
is also some simplification of the relationship between clerk timeouts
and penciller back-pressure.

Also resolved is an issue of inconcistency between the loader and the on
startup (replaying the transaction log) and the standard push_mem
process.  The loader was not correctly de-duplicating by adding first
(in order) to a tree before outputting the list from the tree.

Some thought will be given later as to whether non-L0 work can be safely
prioritised if the merge process still keeps getting behind.

src/leveled_inker.erl

@@ -609,7 +609,7 @@ load_from_sequence(MinSQN, FilterFun, Penciller, [{_LowSQN, FN, Pid}|Rest]) ->
 load_between_sequence(MinSQN, MaxSQN, FilterFun, Penciller,
                                 CDBpid, StartPos, FN, Rest) ->
     io:format("Loading from filename ~s from SQN ~w~n", [FN, MinSQN]),
-    InitAcc = {MinSQN, MaxSQN, []},
+    InitAcc = {MinSQN, MaxSQN, gb_trees:empty()},
     Res = case leveled_cdb:cdb_scan(CDBpid, FilterFun, InitAcc, StartPos) of
                 {eof, {AccMinSQN, _AccMaxSQN, AccKL}} ->
                     ok = push_to_penciller(Penciller, AccKL),
@@ -633,12 +633,18 @@ load_between_sequence(MinSQN, MaxSQN, FilterFun, Penciller,
             ok
     end.
 
-push_to_penciller(Penciller, KeyList) ->
+push_to_penciller(Penciller, KeyTree) ->
+    % The push to penciller must start as a tree to correctly de-duplicate
+    % the list by order before becoming a de-duplicated list for loading
+    KeyList = gb_trees:to_list(KeyTree),
     R = leveled_penciller:pcl_pushmem(Penciller, KeyList),
     if
         R == pause ->
             timer:sleep(?LOADING_PAUSE);
-        true ->
+        R == returned ->
+            timer:sleep(?LOADING_PAUSE),
+            push_to_penciller(Penciller, KeyTree);
+        R == ok ->
             ok
     end.