umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1594 commits 1 branch 0 tags 25 MiB
Commit graph

1086 commits

Author SHA1 Message Date
Martin Sumner
bdd1762130 Missing use of extract_header 
Spotted by ct test crossbucket_aae
 2018-10-30 14:06:17 +00:00
Martin Sumner
75d2e2d546 Fix yield 
Wrong format of repsonse if was delete_pending
 2018-10-30 13:00:23 +00:00
Martin Sumner
7295a41321 Read (and ignore) last modified date 
Add presence of LMD into index - and check everything happily lets it pass by
 2018-10-30 11:47:03 +00:00
Martin Sumner
467c2fb89c Allow a boolean to be passed in to set IndexModDate 
Although we are still pre-release in Leveled, for completeness it is a useful test of this code change to show that it can be done in a backwards compatible way.

So a boolean is added to indicate whether a file should index the modified date within the slot, and this can then be read when the file is opened.

Nothing happens with the boolean, yet.
 2018-10-30 10:25:54 +00:00
Martin Sumner
8ba28700eb Start adding in last_moified dates 
With updated specs
 2018-10-29 21:50:32 +00:00
Martin Sumner
14fd67e535 Add specs and comments and split function 
Need to change this, so refactor and make neater in preparation
 2018-10-29 21:16:38 +00:00
Martin Sumner
baa4466923 Remove knowledge of tuple length from ledger value 
Nothing should now care about the current tuple length - and hence the tuple length may be increased (for example to add a max_mod_date)
 2018-10-29 20:24:54 +00:00
Martin Sumner
671b6e7f99 Strip ALL_BUCKET - only used in AAE 2018-10-29 16:56:58 +00:00
Martin Sumner
2e2c35fe1b Extract deprecated recent_aae 
Ready to add other forms of last modified filtering
 2018-10-29 15:49:50 +00:00
Martin Sumner
3803a190af No need to test at high volume every run 2018-10-24 16:01:41 +01:00
Martin Sumner
f4c65597ea log 2 compat 
Need to support math:log2 in early versions of OTP (for Riak)
 2018-10-24 15:42:21 +01:00
Martin Sumner
6d62803567 Test at higher volume 
Increase probability of hit
 2018-10-24 12:06:42 +01:00
Martin Sumner
d171fda705 Format comment 2018-10-24 11:58:25 +01:00
Martin Sumner
d5d54cc13a Add timing test to guide selection of approach 2018-10-24 11:50:36 +01:00
Martin Sumner
4981cdfc07 Expand segment lists when matches are produced form smaller trees 
Let us say a store has precalculated segments based on large tree size, but mismatched segmentIDs are found on a small tree size (for query efficiency).

the list of segment IDs need to be expanded out for matching.  This may be more efficient that running match_segment for each check (expand out once and uses lists:member/2).  Also this is necessary for leveled_so AAE stores (segment-ordered stores)
 2018-10-24 10:19:54 +01:00
Martin Sumner
dd9a6f0015 Add segment matching function 
For when matching between different tree sizes
 2018-10-23 18:46:10 +01:00
Martin Sumner
0bbaf1f25e Handle missing journal files without data loss 
Make sure data added after the journal has been lost will not be lost (even though lost data may stay lost)
 2018-10-03 18:38:56 +01:00
Martin Sumner
b7c8304d99 Never check inker if head_only 2018-09-28 16:58:28 +01:00
Martin Sumner
b1ae7837a1 Add unit test of frequency 
but a bit hard to test due to randomness
 2018-09-28 16:16:19 +01:00
Martin Sumner
575397229e Infrequent check of Journal for key presence 
with decreasing frequency if there are repeated successes when checking
 2018-09-28 15:46:43 +01:00
Martin Sumner
f95f078929 Merge branch 'master' into mas-i189-tuplebuckets 2018-09-27 16:53:18 +01:00
Martin Sumner
1fc2c7ff2b Switch log to warning 
As the database will continue to function
 2018-09-27 15:54:16 +01:00
Martin Sumner
88e4cf2361 Add unit test with bad write 
If a partial write is made at the beginning,  the file was not being truncated.  Consequently writes would be accepted after it opens, and the those writes would be after the corruption so would never be read in the future.
 2018-09-27 15:26:45 +01:00
Martin Sumner
b15fef8cf6 Add unit test of corrupted keys 
To be detected in direct fetch
 2018-09-27 14:20:13 +01:00
Martin Sumner
39e366a31b Add ct test 
Which passes
 2018-09-27 11:32:42 +01:00
Martin Sumner
0fb35e658f Add support for buckets that are tuples 
Only {binary(), binary()} tuples
 2018-09-27 09:34:40 +01:00
Martin Sumner
36537b4ae2 May need to wait longer for completion 
As esting more keys
 2018-09-26 15:38:43 +01:00
Martin Sumner
b2fce95b14 Add explainer to NaiveBestRun 2018-09-26 14:11:02 +01:00
Martin Sumner
7885f93ba6 Boost docs 
Try and make this easier to follow with docs and specs
 2018-09-26 14:07:44 +01:00
Martin Sumner
979c65f0af Merge branch 'mas-i185-docupdate' into mas-i186-iclerkscore 2018-09-26 13:33:07 +01:00
Martin Sumner
d9229c9782 Change score assessment 
Wans't actually calculating scores correctly previously.
 2018-09-26 12:56:28 +01:00
Martin Sumner
3df481941f Record time to calculate compation run 
As this is a factor in assessing suitability of Journal size
 2018-09-26 10:19:24 +01:00
Martin Sumner
051fcd672c Add test of switching from a big journal to a little journal 
And sending objects over the journal size
 2018-09-25 16:56:24 +01:00
Martin Sumner
f4b365438c Further comments in API docs 2018-09-24 20:43:21 +01:00
Martin Sumner
bed155761b Added comments 
This is still a clumsy feature, in terms of implementation.

Is the fact that some folds handle a throw, and some don't an issue?
 2018-09-24 20:05:48 +01:00
Martin Sumner
a9b097e392 Add a wrapper to fold_keys queries 
Queries that in Riak will be based on fold_keys need to be able to catch throws, and re-throw them to be detected by the worker (whilst still clearing up the snapshot)
 2018-09-24 19:54:28 +01:00
Martin Sumner
ba93ea149f Account for slower tests in older Erlang versions 
Tests run slower in OTP16 - so do less work in this version.
 2018-09-21 13:27:10 +01:00
Martin Sumner
1a3d3daa89 Add regex support to $key index 
Regex to be applied to key only
 2018-09-21 12:04:32 +01:00
Martin Sumner
d999bc1e4a Merge branch 'master' into mas-i178-bookhead 2018-09-20 12:08:46 +01:00
Martin Sumner
c439e4144a Add new book_headonly/4 API 
To address special situation of performing a head requets in head_only mode - where a sub-key is a required input.
 2018-09-20 12:08:33 +01:00
Russell Brown
f12d277875 Update docs for start key 
Quviq/leveled/issues/13 shows the confusion around the purpose of
start key. Hopefully this commit clarifies that start key is a further
refinement to the range start. It's essentially and AND.
 2018-09-20 11:21:55 +01:00
Martin Sumner
f4a7a729ae
Merge pull request #175 from martinsumner/rdb/quviq-i15-regex-doc 
Update docs to clarify that regex cannot be run on integers
 2018-09-19 18:31:39 +01:00
Russell Brown
bc35a7aa1e Update docs to clarify that regex cannot be run on integers 2018-09-19 16:28:28 +01:00
Martin Sumner
e02c6df3ed Refactor safe_read 
Make safe_read a bit more flexible,and allow it to catch a wider number of errors.
 2018-09-19 13:03:00 +01:00
Russell Brown
5a95e82af0 Callers of bucket list expect the traversal to be in order 
Due to the internal fold over buckets returning an un-reversed
accumulator, the API bucketlist code caller's fold fun traversed the
bucket list in reverse order. This lead to some inconsistencies when
comparing a buckelist of all buckets, vs, first bucket only. i.e. the
'first' bucket passed to the foldfun was in fact the last bucket read
from the ledger.
 2018-09-18 15:40:44 +01:00
Martin Sumner
413b8af900 Confirm undefined root_path fails 2018-09-17 10:09:57 +01:00
Martin Sumner
8ada5e78fa Max penciller cache change 
Missed a bit
 2018-09-14 17:22:25 +01:00
Martin Sumner
e18d681043 Need to support undefined max penciller cache size 
Without first converting it to the minimum.  So catch the undefined, at set_default
 2018-09-14 10:36:10 +01:00
Martin Sumner
534bf87a51 Set minimum cache sizes 
This avoids a divide by zero (if the Bookie's memory cache size is smaller than 4).
 2018-09-14 10:19:25 +01:00
Martin Sumner
91f751ddc6 Garbage collect backups 
If ther are backups made to the same folder, need to remove any files from that folder that are not included in this backup.

Some initial testing, needs more.
 2018-09-07 14:21:01 +01:00