This has refactored code with the implementation of the manifest
isolated in to a seperate module, and the pure async relationship
between penciller and their clerk. However, the manifest is just a
simple list at each level.
This gives a new manifest implementation that is an array of lists.
Just basic unit testing of lookup insertion and removal in this module.
The API was changed subtly, and so nothing broader will work at this
stage
correct the previous change to make sure that deletes are not confirmed
when work is outstanding, but also make sure that the clerk only ever
casts the Penciller so no deadlocks can ensue.
Going to abandond this branch for now. The change is beoming
excessively time consuming, and it is not clear that a smaller change
might not achieve more of the objectives.
All this is broken - but perhaps could get picke dup another day.
The penciller requires a starting manifest SQN as well as a list of
filenames to open on startup. It is assumed that the penciller will
keep the mappings between fileames and PIDs outside of the manifest.
The manifest had previously been a list for eveyr leevl of the manifest,
and keys were found by folding over the list. By Level 4 the list will
be 4096 items long, and so the fold would be expensive, and would be
required many times.
To make this less expensive an ETS table is to use. However, the ETS
table needs to be shared between snapshots and so in order to use the
ETS the entries to the table need to support multi-versioning - whereby
each clone can see a version of the table at the Manifest SQN the clone
is supporting.
Remove the ets index in pmem and use a binary index instead. This may
be slower, but avoids the bulk upload to ets, and means that matches
know of position (so only skiplists with a match need be tried).
Also stops the discrepancy between snapshots and non-snapshots - as
previously the snapshots were always slowed by not having access to the
ETS table.
