Close in stages - waiting for releases (#411)

* Close in stages - waiting for releases

Have a consistent approach to closing the inker and the penciller - so that the close can be interrupted by releasing of snapshots.  Then any unreleased snapshots are closed before shutdown - with a 10s pause to give queries a short opportunity to finish.

This should address some issues, primarily seen (but very rarely) in test whereby post-rebuild destruction of parallel AAE keystores cause the crashing of aae_folds.

The primary benefit is to stop an attempt to release a snapshot that has in fact already finished does not cause a crash of the database on normal stop.  this was primarily an issue when shutdown is delayed by an ongoing journal compaction job.

* Boost default test budget for EQC

* Update test to use correct type

* Update following review

Avoid filtering out exited PIDs when closing snapshots by catching the exit exception when the Pid is down

test/property/leveled_simpleeqc.erl

@@ -41,7 +41,8 @@
                 start_opts = []
                }).
 
--define(NUMTESTS, 1000).
+-define(NUMTESTS, 10000).
+-define(TIME_BUDGET, 300).
 -define(QC_OUT(P),
         eqc:on_output(fun(Str, Args) ->
                               io:format(user, Str, Args) end, P)).
@@ -49,7 +50,12 @@
 -type state() :: #state{}.
 
 eqc_test_() ->
-    {timeout, 60, ?_assertEqual(true, eqc:quickcheck(eqc:testing_time(50, ?QC_OUT(prop_db()))))}.
+    {timeout,
+        ?TIME_BUDGET + 10,
+        ?_assertEqual(
+            true,
+            eqc:quickcheck(
+                eqc:testing_time(?TIME_BUDGET, ?QC_OUT(prop_db()))))}.
 
 run() ->
     run(?NUMTESTS).