umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1632 commits 1 branch 0 tags 25 MiB
Commit graph

1115 commits

Author SHA1 Message Date
Martin Sumner
1fc2c7ff2b Switch log to warning 
As the database will continue to function
 2018-09-27 15:54:16 +01:00
Martin Sumner
88e4cf2361 Add unit test with bad write 
If a partial write is made at the beginning,  the file was not being truncated.  Consequently writes would be accepted after it opens, and the those writes would be after the corruption so would never be read in the future.
 2018-09-27 15:26:45 +01:00
Martin Sumner
b15fef8cf6 Add unit test of corrupted keys 
To be detected in direct fetch
 2018-09-27 14:20:13 +01:00
Martin Sumner
39e366a31b Add ct test 
Which passes
 2018-09-27 11:32:42 +01:00
Martin Sumner
0fb35e658f Add support for buckets that are tuples 
Only {binary(), binary()} tuples
 2018-09-27 09:34:40 +01:00
Martin Sumner
36537b4ae2 May need to wait longer for completion 
As esting more keys
 2018-09-26 15:38:43 +01:00
Martin Sumner
b2fce95b14 Add explainer to NaiveBestRun 2018-09-26 14:11:02 +01:00
Martin Sumner
7885f93ba6 Boost docs 
Try and make this easier to follow with docs and specs
 2018-09-26 14:07:44 +01:00
Martin Sumner
979c65f0af Merge branch 'mas-i185-docupdate' into mas-i186-iclerkscore 2018-09-26 13:33:07 +01:00
Martin Sumner
d9229c9782 Change score assessment 
Wans't actually calculating scores correctly previously.
 2018-09-26 12:56:28 +01:00
Martin Sumner
3df481941f Record time to calculate compation run 
As this is a factor in assessing suitability of Journal size
 2018-09-26 10:19:24 +01:00
Martin Sumner
051fcd672c Add test of switching from a big journal to a little journal 
And sending objects over the journal size
 2018-09-25 16:56:24 +01:00
Martin Sumner
f4b365438c Further comments in API docs 2018-09-24 20:43:21 +01:00
Martin Sumner
bed155761b Added comments 
This is still a clumsy feature, in terms of implementation.

Is the fact that some folds handle a throw, and some don't an issue?
 2018-09-24 20:05:48 +01:00
Martin Sumner
a9b097e392 Add a wrapper to fold_keys queries 
Queries that in Riak will be based on fold_keys need to be able to catch throws, and re-throw them to be detected by the worker (whilst still clearing up the snapshot)
 2018-09-24 19:54:28 +01:00
Martin Sumner
ba93ea149f Account for slower tests in older Erlang versions 
Tests run slower in OTP16 - so do less work in this version.
 2018-09-21 13:27:10 +01:00
Martin Sumner
1a3d3daa89 Add regex support to $key index 
Regex to be applied to key only
 2018-09-21 12:04:32 +01:00
Martin Sumner
d999bc1e4a Merge branch 'master' into mas-i178-bookhead 2018-09-20 12:08:46 +01:00
Martin Sumner
c439e4144a Add new book_headonly/4 API 
To address special situation of performing a head requets in head_only mode - where a sub-key is a required input.
 2018-09-20 12:08:33 +01:00
Russell Brown
f12d277875 Update docs for start key 
Quviq/leveled/issues/13 shows the confusion around the purpose of
start key. Hopefully this commit clarifies that start key is a further
refinement to the range start. It's essentially and AND.
 2018-09-20 11:21:55 +01:00
Martin Sumner
f4a7a729ae
Merge pull request #175 from martinsumner/rdb/quviq-i15-regex-doc 
Update docs to clarify that regex cannot be run on integers
 2018-09-19 18:31:39 +01:00
Russell Brown
bc35a7aa1e Update docs to clarify that regex cannot be run on integers 2018-09-19 16:28:28 +01:00
Martin Sumner
e02c6df3ed Refactor safe_read 
Make safe_read a bit more flexible,and allow it to catch a wider number of errors.
 2018-09-19 13:03:00 +01:00
Russell Brown
5a95e82af0 Callers of bucket list expect the traversal to be in order 
Due to the internal fold over buckets returning an un-reversed
accumulator, the API bucketlist code caller's fold fun traversed the
bucket list in reverse order. This lead to some inconsistencies when
comparing a buckelist of all buckets, vs, first bucket only. i.e. the
'first' bucket passed to the foldfun was in fact the last bucket read
from the ledger.
 2018-09-18 15:40:44 +01:00
Martin Sumner
413b8af900 Confirm undefined root_path fails 2018-09-17 10:09:57 +01:00
Martin Sumner
8ada5e78fa Max penciller cache change 
Missed a bit
 2018-09-14 17:22:25 +01:00
Martin Sumner
e18d681043 Need to support undefined max penciller cache size 
Without first converting it to the minimum.  So catch the undefined, at set_default
 2018-09-14 10:36:10 +01:00
Martin Sumner
534bf87a51 Set minimum cache sizes 
This avoids a divide by zero (if the Bookie's memory cache size is smaller than 4).
 2018-09-14 10:19:25 +01:00
Martin Sumner
91f751ddc6 Garbage collect backups 
If ther are backups made to the same folder, need to remove any files from that folder that are not included in this backup.

Some initial testing, needs more.
 2018-09-07 14:21:01 +01:00
Martin Sumner
b99cde9599 Merge branch 'master' into mas-i164-hotbackup 2018-09-07 12:27:30 +01:00
Martin Sumner
faec45ad8e
Merge pull request #167 from russelldb/rdb/spec-folds-cp 
Provide a top level API for folds
 2018-09-07 11:18:23 +01:00
Martin Sumner
9a8ce88ed2 Add double-backup test 
Added a test with back-to-back backups.  This caused issues with the empty CDB file it created (on opening, couldn't cope with last key of empty).

So now backup won't roll the active journal if it is empty.
 2018-09-07 10:24:51 +01:00
Martin Sumner
0838ff34e5 Initial hot_backup 
The idea being that a consistent inker manifest and set of journal files is guaranteed - with hard links to the actual manifest files.
 2018-09-06 17:50:30 +01:00
Russell Brown
b7bd65d11f Provide a top level API for folds 
As the fold functions have been added to get_runner in an ad hoc way,
naturally, given the ongoing development of levelEd to support Riak,
it was difficult for a new user (in this case Quviq) to see what folds
are supported, and with what arguments, and expectations.

This PR is for discussion. It is one of many ways to group, spec, and
document the fold functions.

A test is also added for coverage of range queries.
 2018-09-06 15:01:54 +01:00
Russell Brown
4334e2d734 Add unit test for pclr snapshot closing 
This was tested by the eqc, but we merged the fix without the
test. This eunit test fixes that. Coverage!
 2018-09-06 14:08:09 +01:00
Russell Brown
ef9ac672e5 Stop snapshots when the bookie stops 
During EQC testing it was found that snapshots are still usable even
if the bookie process crashes. This change has snapshots monitor the
bookie and close when the bookie process dies.
 2018-09-06 11:47:52 +01:00
Martin Sumner
c64dc1df0d Change key() definition to not allow integer keys 2018-09-03 12:28:31 +01:00
Martin Sumner
402dd2a663 Remove integer() support for keys 
As number < atom() and so integer() < null.

This causes potential issues where we assume null < then any key value
 2018-09-03 10:50:28 +01:00
Martin Sumner
f400e8b46d Add test for string and binary buckets 
Tried an integer bucket - and this didn't work!

Integers are < null - and so is_empty won't work as expected.
 2018-09-03 10:38:19 +01:00
Martin Sumner
ca8aa051d7 Add head_only fold_keys test 2018-09-03 10:29:20 +01:00
Martin Sumner
bde188e691 Constrain keys 
Rather than supporting any() - constrain at least to binary()/integer() or string().
 2018-09-01 12:10:56 +01:00
Martin Sumner
50967438d3 Switch from binary_bucketlist 
Allow for bucket listing of non-binary buckets (integer buckets, buckets with ascii strings)
 2018-09-01 10:39:23 +01:00
Martin Sumner
41fb83abd1 Add tests for is_empty 
Where keys are strings or integers, and where subkeys are involved
 2018-08-31 15:29:38 +01:00
Russell Brown
0c018c5721 Don't leave commented out code lying around 2018-08-07 12:49:31 +01:00
Russell Brown
e8f462fab0 Fix timezone specific failing test 
Converts date time to a local now
 2018-08-07 12:35:07 +01:00
Martin Sumner
076d2646c6
Merge pull request #155 from martinsumner/mas-i152-iclerkconfig 
Mas i152 iclerkconfig
 2018-07-24 12:13:38 +01:00
Martin Sumner
fa513c1c42 Change start time to correct time 
Must have miscaulcated the start time originally
 2018-07-24 12:09:59 +01:00
Martin Sumner
66a75923e8 Expose configuration of Compactio Percentage targets 
I think these should be set differently, so make them configurable.
 2018-07-23 12:46:42 +01:00
Martin Sumner
c4e376ece5 Don't link snapshots 
If a snapshot breaks a penciller clone, this shouldn't crash the main process.
 2018-07-10 10:25:20 +01:00
Martin Sumner
082eabb65b Switch to start_link 
Start all processes linked - to collapse the whole tree if one process fails
 2018-06-28 12:16:43 +01:00