umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1634 commits 1 branch 0 tags 25 MiB
Commit graph

1634 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
9e56bfa947 Merge branch 'master' into mas-i311-mergeselector 2020-03-30 20:07:05 +01:00
Martin Sumner
febdac27f6
Merge pull request #310 from martinsumner/mas-i306-implementrecalc 
Mas i306 implementrecalc
 2020-03-30 20:06:48 +01:00
Martin Sumner
9838e255d2 Address review comments 
More efficient traversal of list to score.
 2020-03-29 20:02:21 +01:00
Martin Sumner
28c88ef8b8 Typo 2020-03-27 20:09:03 +00:00
Martin Sumner
42eb5f56bc Merge branch 'master' into mas-i311-mergeselector 2020-03-27 17:11:18 +00:00
Martin Sumner
da97d65a23 Add grooming compactions 
Make half of LSM-tree compactions grooming compactions i.e. compactions biased towards merging files with large numbers of tombstones.
 2020-03-27 15:09:48 +00:00
Martin Sumner
aca945a171 Add counting of tombstones to new SST files 
.. and that old-style SST files cna still be created, and opened, with a return of 'not_counted'
 2020-03-27 10:20:10 +00:00
Martin Sumner
4ec5a19db3
Merge pull request #308 from martinsumner/mas-i306-reviseretain 
Mas i306 reviseretain
 2020-03-27 09:25:23 +00:00
Martin Sumner
e175948378 Remove references ot 'skip' strategy 
Now called `recovr`
 2020-03-26 14:25:09 +00:00
Martin Sumner
4ef0f4006d Extend mergefile_selector for strategy 
Strategy only applied below L1, and only random strategy supported
 2020-03-26 14:18:57 +00:00
Martin Sumner
20a7a22571 Add documentation for recalc option 2020-03-24 20:21:44 +00:00
Martin Sumner
8a9db9e75e Add log of startegy when clerk starts compaction 2020-03-23 16:45:28 +00:00
Martin Sumner
50cb98ecdd Resolve intermittent test failure 
the previous regex filter still allowed files with cdb in the body of the name (which can be true as filenames are guid based)
 2020-03-17 17:29:59 +00:00
Martin Sumner
5b4edfebb6 Coverage cheat 
Very rarely, this line in the tests this line is not covered - so cheating here to consistently pass coverage
 2020-03-17 14:20:57 +00:00
Martin Sumner
808a858d09 Don't score a rolling file 
In giving an empty file a score of 0, a race condition was exposed.  A file might not be active, but might still be rolling - and then cna get scored as 0, and immediately compacted.  It will then be removed from the journal manifest.

Check each file is not rolling before making it a candidate for rolling.
 2020-03-16 21:41:47 +00:00
Martin Sumner
5f7d261a87 Improve test 
Genuine overhang
 2020-03-16 18:53:40 +00:00
Martin Sumner
b49a5ff53d Additional unit tests of MetaBin handling 2020-03-16 17:35:38 +00:00
Martin Sumner
dbceda876c Issue with tag order 
https://github.com/martinsumner/leveled/issues/309

Resolve issue, and remove test log entries used when discovering issue.
 2020-03-16 16:35:06 +00:00
Martin Sumner
6350302ea8 Uncomment test 2020-03-16 13:32:52 +00:00
Martin Sumner
9d92ca0773 Add tests for appDefined functions 2020-03-16 12:51:14 +00:00
Martin Sumner
706ba8a674 Resolve issues with passing specs around 2020-03-15 23:15:09 +00:00
Martin Sumner
694d2c39f8 Support for recalc 
Initial test included for running with recallc, and also transition from retain to recalc.

Moves all logic for startup fold into leveled_bookie - avoid the Inker requiring any direct knowledge about implementation of the Penciller.
 2020-03-15 22:14:42 +00:00
Martin Sumner
1242dd4991 Merge branch 'master' into mas-i306-reviseretain 2020-03-13 19:56:35 +00:00
Martin Sumner
aaf58dd343
Merge pull request #307 from martinsumner/mas-i306-lessrandomreads 
Mas i306 lessrandomreads
 2020-03-13 19:55:16 +00:00
Martin Sumner
444011ac64 Merge branch 'master' into mas-i306-reviseretain 2020-03-09 21:40:19 +00:00
Martin Sumner
207aeb8b99 Remove additional log 2020-03-09 20:42:48 +00:00
Martin Sumner
6b3328f4a3 Rationalise logging in commit 
Also:

Sort the output from an 'all' fetch one loop at a time

Make sure the test of scoring na empty file  is scoring an empty file

If it is an emtpy file we want to compact the fragment away - in which case it should score 0.0 not 100.0
 2020-03-09 17:45:06 +00:00
Martin Sumner
156e7b064d Compaction, retain and recovery 
Change the penciller check so that it returns current/replaced/missing not just true/false.

Reduce unnecessary penciller checks for non-standard keys that will always be retained - and remove redunandt code.

Expand tests of retain and recover to make sure that compaction on delete is well covered.

Also move the SQN number laong during initial loads - to stop aggressive loop to find starting SQN every file.
 2020-03-09 15:12:48 +00:00
Martin Sumner
60e29f2ff0 (slightly) less random reads on journal compaction 2020-03-06 11:29:25 +00:00
Martin Sumner
5b11462054
Merge pull request #305 from martinsumner/mas-i304-otp22 
Mas i304 otp22
 2020-02-24 12:49:25 +00:00
Martin Sumner
12e84ad628 Don't add later OTP until we can test it 2020-02-24 12:42:40 +00:00
Martin Sumner
009abdd599 Build and test on OTP 22 2020-02-24 09:55:05 +00:00
Martin Sumner
02155558df
Tag for Riak 2.9.1 2020-02-12 09:26:53 +00:00
Martin Sumner
681f243e87 Merge branch 'master' into develop-3.0 2019-11-20 10:34:03 +00:00
Martin Sumner
4d550ef2a1
Bump version for new release 2019-11-20 10:33:51 +00:00
Martin Sumner
b3de5d46c0 Merge branch 'master' into develop-2.9 2019-11-20 09:51:24 +00:00
Martin Sumner
b8976531b1
Merge pull request #302 from martinsumner/mas-i301-saferename 
Mas i301 saferename
 2019-11-19 21:47:24 +00:00
Martin Sumner
bcf10c9709 Fixup comments 2019-11-19 16:36:57 +00:00
Martin Sumner
693defb6d3 Use the same file write/sync/rename path where needed 
When we want to be sure a file has been written before proceeding - we need a safer (that `file:write_file/2`) mechanism to be sure that it is written before proceeding.

This will:
open, write, sync, rename and then optionally read-back.

Changed so that manifest writing uses the safest form (including read back), and that sst writing uses a slightly looser form (with no read back to avoid performance issues).
 2019-11-19 15:50:59 +00:00
Martin Sumner
65bdb40277 Merge branch 'master' into develop-2.9 2019-11-18 13:37:30 +00:00
Martin Sumner
b946271195 Merge branch 'master' into develop-3.0 2019-11-05 20:17:39 +00:00
Martin Sumner
e16b650823
Merge pull request #300 from martinsumner/travisci-test 
Travisci test
 2019-11-05 20:01:59 +00:00
Martin Sumner
8ce9f29c88 Add travis status 2019-11-05 19:54:11 +00:00
Martin Sumner
293b5abed1 Add latest rebar3 3.12.0 2019-11-05 19:42:39 +00:00
Martin Sumner
7bf4faed9a Create .travis.yml 2019-11-05 19:36:20 +00:00
Martin Sumner
e4a5ac8776 Merge branch 'master' into develop-3.0 2019-11-04 22:27:10 +00:00
Martin Sumner
4b967bee4a
Merge pull request #299 from martinsumner/mas-i298-corruptedblocks 
Mas i298 corruptedblocks
 2019-11-04 18:19:34 +00:00
Martin Sumner
96779e667e Add specs and further tests 
Prove corrupted blocks are handled as expected when detected from within check_blocks function.
 2019-11-04 15:32:18 +00:00
Martin Sumner
843b28313d Handle when corrupted blocks creates empty list 
So can't get nth or last.
 2019-11-04 12:49:56 +00:00
Martin Sumner
aa7bb78cab Update rebar.config 
Switch to develop-3.0 lz4
 2019-11-01 13:41:10 +00:00