umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1471 commits 1 branch 0 tags 25 MiB
Commit graph

1471 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
bdd1762130 Missing use of extract_header 
Spotted by ct test crossbucket_aae
 2018-10-30 14:06:17 +00:00
Martin Sumner
75d2e2d546 Fix yield 
Wrong format of repsonse if was delete_pending
 2018-10-30 13:00:23 +00:00
Martin Sumner
7295a41321 Read (and ignore) last modified date 
Add presence of LMD into index - and check everything happily lets it pass by
 2018-10-30 11:47:03 +00:00
Martin Sumner
467c2fb89c Allow a boolean to be passed in to set IndexModDate 
Although we are still pre-release in Leveled, for completeness it is a useful test of this code change to show that it can be done in a backwards compatible way.

So a boolean is added to indicate whether a file should index the modified date within the slot, and this can then be read when the file is opened.

Nothing happens with the boolean, yet.
 2018-10-30 10:25:54 +00:00
Martin Sumner
8ba28700eb Start adding in last_moified dates 
With updated specs
 2018-10-29 21:50:32 +00:00
Martin Sumner
14fd67e535 Add specs and comments and split function 
Need to change this, so refactor and make neater in preparation
 2018-10-29 21:16:38 +00:00
Martin Sumner
baa4466923 Remove knowledge of tuple length from ledger value 
Nothing should now care about the current tuple length - and hence the tuple length may be increased (for example to add a max_mod_date)
 2018-10-29 20:24:54 +00:00
Martin Sumner
671b6e7f99 Strip ALL_BUCKET - only used in AAE 2018-10-29 16:56:58 +00:00
Martin Sumner
2e2c35fe1b Extract deprecated recent_aae 
Ready to add other forms of last modified filtering
 2018-10-29 15:49:50 +00:00
Martin Sumner
3803a190af No need to test at high volume every run 2018-10-24 16:01:41 +01:00
Martin Sumner
f4c65597ea log 2 compat 
Need to support math:log2 in early versions of OTP (for Riak)
 2018-10-24 15:42:21 +01:00
Martin Sumner
6d62803567 Test at higher volume 
Increase probability of hit
 2018-10-24 12:06:42 +01:00
Martin Sumner
d171fda705 Format comment 2018-10-24 11:58:25 +01:00
Martin Sumner
d5d54cc13a Add timing test to guide selection of approach 2018-10-24 11:50:36 +01:00
Martin Sumner
4981cdfc07 Expand segment lists when matches are produced form smaller trees 
Let us say a store has precalculated segments based on large tree size, but mismatched segmentIDs are found on a small tree size (for query efficiency).

the list of segment IDs need to be expanded out for matching.  This may be more efficient that running match_segment for each check (expand out once and uses lists:member/2).  Also this is necessary for leveled_so AAE stores (segment-ordered stores)
 2018-10-24 10:19:54 +01:00
Martin Sumner
5cbda7c2f7
Merge pull request #197 from martinsumner/mas-i196-segfilter 
Add segment matching function
 2018-10-24 07:34:14 +01:00
Martin Sumner
dd9a6f0015 Add segment matching function 
For when matching between different tree sizes
 2018-10-23 18:46:10 +01:00
Martin Sumner
5f5255142d
Merge pull request #195 from martinsumner/mas-i194-startupsqn 
Mas i194 startupsqn
 2018-10-03 21:05:57 +01:00
Martin Sumner
0bbaf1f25e Handle missing journal files without data loss 
Make sure data added after the journal has been lost will not be lost (even though lost data may stay lost)
 2018-10-03 18:38:56 +01:00
Martin Sumner
9a0082db4e Test to expose startup issue 
See #194
 2018-10-03 18:29:20 +01:00
Martin Sumner
cc676190c4
Merge pull request #193 from martinsumner/mas-192-journalcheck4head 
Mas 192 journalcheck4head
 2018-10-01 11:59:41 +01:00
Martin Sumner
b7c8304d99 Never check inker if head_only 2018-09-28 16:58:28 +01:00
Martin Sumner
b1ae7837a1 Add unit test of frequency 
but a bit hard to test due to randomness
 2018-09-28 16:16:19 +01:00
Martin Sumner
575397229e Infrequent check of Journal for key presence 
with decreasing frequency if there are repeated successes when checking
 2018-09-28 15:46:43 +01:00
Martin Sumner
8be873f776
Merge pull request #191 from martinsumner/mas-i189-tuplebuckets 
Mas i189 tuplebuckets
 2018-09-27 18:14:30 +01:00
Martin Sumner
f95f078929 Merge branch 'master' into mas-i189-tuplebuckets 2018-09-27 16:53:18 +01:00
Martin Sumner
ee172468ce
Merge pull request #190 from martinsumner/mas-qi18-saferead 
Mas qi18 saferead
 2018-09-27 16:52:43 +01:00
Martin Sumner
1fc2c7ff2b Switch log to warning 
As the database will continue to function
 2018-09-27 15:54:16 +01:00
Martin Sumner
88e4cf2361 Add unit test with bad write 
If a partial write is made at the beginning,  the file was not being truncated.  Consequently writes would be accepted after it opens, and the those writes would be after the corruption so would never be read in the future.
 2018-09-27 15:26:45 +01:00
Martin Sumner
b15fef8cf6 Add unit test of corrupted keys 
To be detected in direct fetch
 2018-09-27 14:20:13 +01:00
Martin Sumner
39e366a31b Add ct test 
Which passes
 2018-09-27 11:32:42 +01:00
Martin Sumner
0fb35e658f Add support for buckets that are tuples 
Only {binary(), binary()} tuples
 2018-09-27 09:34:40 +01:00
Martin Sumner
cbd7713c71
Merge pull request #188 from martinsumner/mas-i186-iclerkscore 
Mas i186 iclerkscore
 2018-09-26 17:36:43 +01:00
Martin Sumner
36537b4ae2 May need to wait longer for completion 
As esting more keys
 2018-09-26 15:38:43 +01:00
Martin Sumner
b2fce95b14 Add explainer to NaiveBestRun 2018-09-26 14:11:02 +01:00
Martin Sumner
7885f93ba6 Boost docs 
Try and make this easier to follow with docs and specs
 2018-09-26 14:07:44 +01:00
Martin Sumner
979c65f0af Merge branch 'mas-i185-docupdate' into mas-i186-iclerkscore 2018-09-26 13:33:07 +01:00
Martin Sumner
d9229c9782 Change score assessment 
Wans't actually calculating scores correctly previously.
 2018-09-26 12:56:28 +01:00
Martin Sumner
3df481941f Record time to calculate compation run 
As this is a factor in assessing suitability of Journal size
 2018-09-26 10:19:24 +01:00
Martin Sumner
8d8a0a3f5b Improve explanation of max_journal_size 
Tighten guidance with regards to very small values, which aren't expected to make sense
 2018-09-26 09:37:50 +01:00
Martin Sumner
908f762a48
Merge pull request #186 from martinsumner/mas-i185-emptyjournal 
Mas i185 emptyjournal
 2018-09-25 19:54:57 +01:00
Martin Sumner
0772317247 Test mistake 
If random integer was low, total could be below threshold - so calculate total correctly.

Should make value re-generate random uniform, but test is still valid without this
 2018-09-25 18:32:48 +01:00
Martin Sumner
051fcd672c Add test of switching from a big journal to a little journal 
And sending objects over the journal size
 2018-09-25 16:56:24 +01:00
Martin Sumner
ab74d5c0dd
Merge pull request #184 from martinsumner/mas-i183-thowinasyncfold 
Mas i183 thowinasyncfold
 2018-09-24 22:41:12 +01:00
Martin Sumner
f4b365438c Further comments in API docs 2018-09-24 20:43:21 +01:00
Martin Sumner
bed155761b Added comments 
This is still a clumsy feature, in terms of implementation.

Is the fact that some folds handle a throw, and some don't an issue?
 2018-09-24 20:05:48 +01:00
Martin Sumner
a9b097e392 Add a wrapper to fold_keys queries 
Queries that in Riak will be based on fold_keys need to be able to catch throws, and re-throw them to be detected by the worker (whilst still clearing up the snapshot)
 2018-09-24 19:54:28 +01:00
Martin Sumner
bcda272fb9
Merge pull request #182 from martinsumner/mas-i180-dollarbucketregex 
Mas i180 dollarbucketregex
 2018-09-24 16:37:25 +01:00
Martin Sumner
8f320a054e Add object deletion to test of dolerkey index 
Related to tracking a failure in riak_test
 2018-09-21 17:08:59 +01:00
Martin Sumner
37f784f041 Add regex tests for dollar bucket queries 
Proce this works as well as for dollar key.  This isn't tested in riak_test currently, but seems illogical not to support it her eif it is supported on $key queries.
 2018-09-21 15:17:29 +01:00