Which exposed it wasn't working. If there is no segment list passed - just a modification filter, you don't need to check the position list (as checking the position list returns an empty position so sipping all the matching results!)
Acc in response is now of form {Reason, Acc} not just Acc so that the application can understand the reason for the results ending - and take appropriate action (e.g. restart again from the LastKey to return more results).
To support max_keys and the last modified date range.
This applies the last modified date check on all ledger folds. This is hard to avoid, but ultimately a very low cost.
The limit on the number of heads to fold, is the limit based on passing to the accumulator - not on the limit being added to the accumulator. So if the FoldFun perfoms a filter (e.g. for the preflist), then those filtered results will still count towards the maximum.
There needs to be someway at the end of signalling from the fold if the outcome was or was not 'constrained' by max_keys - as the fold cannot simply tel by lenght checking the outcome.
Note this is used rather than length checking the buffer and throwing a 'stop_fold' message when the limit is reached. The choice is made for simplicity, and ease of testing. The throw mechanism is necessary if there is a need to stop parallel folds across the the cluster - but in this case the node_worker_pool will be used.
Externally to leveled_sst all folds are actually managed through exapnd_list_by_pointer.
Make the API a bit clearer in this regards, and add specs to help dialyzer.
This also adds LowLastMod to the API for expanding pointers (although the leveled_penciller just defaults this to 0 for everything.
Although we are still pre-release in Leveled, for completeness it is a useful test of this code change to show that it can be done in a backwards compatible way.
So a boolean is added to indicate whether a file should index the modified date within the slot, and this can then be read when the file is opened.
Nothing happens with the boolean, yet.
Let us say a store has precalculated segments based on large tree size, but mismatched segmentIDs are found on a small tree size (for query efficiency).
the list of segment IDs need to be expanded out for matching. This may be more efficient that running match_segment for each check (expand out once and uses lists:member/2). Also this is necessary for leveled_so AAE stores (segment-ordered stores)
If a partial write is made at the beginning, the file was not being truncated. Consequently writes would be accepted after it opens, and the those writes would be after the corruption so would never be read in the future.
