umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1512 commits 1 branch 0 tags 25 MiB
Commit graph

1512 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
5f7d261a87 Improve test 
Genuine overhang
 2020-03-16 18:53:40 +00:00
Martin Sumner
b49a5ff53d Additional unit tests of MetaBin handling 2020-03-16 17:35:38 +00:00
Martin Sumner
dbceda876c Issue with tag order 
https://github.com/martinsumner/leveled/issues/309

Resolve issue, and remove test log entries used when discovering issue.
 2020-03-16 16:35:06 +00:00
Martin Sumner
6350302ea8 Uncomment test 2020-03-16 13:32:52 +00:00
Martin Sumner
9d92ca0773 Add tests for appDefined functions 2020-03-16 12:51:14 +00:00
Martin Sumner
706ba8a674 Resolve issues with passing specs around 2020-03-15 23:15:09 +00:00
Martin Sumner
694d2c39f8 Support for recalc 
Initial test included for running with recallc, and also transition from retain to recalc.

Moves all logic for startup fold into leveled_bookie - avoid the Inker requiring any direct knowledge about implementation of the Penciller.
 2020-03-15 22:14:42 +00:00
Martin Sumner
1242dd4991 Merge branch 'master' into mas-i306-reviseretain 2020-03-13 19:56:35 +00:00
Martin Sumner
aaf58dd343
Merge pull request #307 from martinsumner/mas-i306-lessrandomreads 
Mas i306 lessrandomreads
 2020-03-13 19:55:16 +00:00
Martin Sumner
444011ac64 Merge branch 'master' into mas-i306-reviseretain 2020-03-09 21:40:19 +00:00
Martin Sumner
207aeb8b99 Remove additional log 2020-03-09 20:42:48 +00:00
Martin Sumner
6b3328f4a3 Rationalise logging in commit 
Also:

Sort the output from an 'all' fetch one loop at a time

Make sure the test of scoring na empty file  is scoring an empty file

If it is an emtpy file we want to compact the fragment away - in which case it should score 0.0 not 100.0
 2020-03-09 17:45:06 +00:00
Martin Sumner
156e7b064d Compaction, retain and recovery 
Change the penciller check so that it returns current/replaced/missing not just true/false.

Reduce unnecessary penciller checks for non-standard keys that will always be retained - and remove redunandt code.

Expand tests of retain and recover to make sure that compaction on delete is well covered.

Also move the SQN number laong during initial loads - to stop aggressive loop to find starting SQN every file.
 2020-03-09 15:12:48 +00:00
Martin Sumner
60e29f2ff0 (slightly) less random reads on journal compaction 2020-03-06 11:29:25 +00:00
Martin Sumner
02155558df
Tag for Riak 2.9.1 2020-02-12 09:26:53 +00:00
Martin Sumner
4d550ef2a1
Bump version for new release 2019-11-20 10:33:51 +00:00
Martin Sumner
b3de5d46c0 Merge branch 'master' into develop-2.9 2019-11-20 09:51:24 +00:00
Martin Sumner
b8976531b1
Merge pull request #302 from martinsumner/mas-i301-saferename 
Mas i301 saferename
 2019-11-19 21:47:24 +00:00
Martin Sumner
bcf10c9709 Fixup comments 2019-11-19 16:36:57 +00:00
Martin Sumner
693defb6d3 Use the same file write/sync/rename path where needed 
When we want to be sure a file has been written before proceeding - we need a safer (that `file:write_file/2`) mechanism to be sure that it is written before proceeding.

This will:
open, write, sync, rename and then optionally read-back.

Changed so that manifest writing uses the safest form (including read back), and that sst writing uses a slightly looser form (with no read back to avoid performance issues).
 2019-11-19 15:50:59 +00:00
Martin Sumner
65bdb40277 Merge branch 'master' into develop-2.9 2019-11-18 13:37:30 +00:00
Martin Sumner
e16b650823
Merge pull request #300 from martinsumner/travisci-test 
Travisci test
 2019-11-05 20:01:59 +00:00
Martin Sumner
8ce9f29c88 Add travis status 2019-11-05 19:54:11 +00:00
Martin Sumner
293b5abed1 Add latest rebar3 3.12.0 2019-11-05 19:42:39 +00:00
Martin Sumner
7bf4faed9a Create .travis.yml 2019-11-05 19:36:20 +00:00
Martin Sumner
4b967bee4a
Merge pull request #299 from martinsumner/mas-i298-corruptedblocks 
Mas i298 corruptedblocks
 2019-11-04 18:19:34 +00:00
Martin Sumner
96779e667e Add specs and further tests 
Prove corrupted blocks are handled as expected when detected from within check_blocks function.
 2019-11-04 15:32:18 +00:00
Martin Sumner
843b28313d Handle when corrupted blocks creates empty list 
So can't get nth or last.
 2019-11-04 12:49:56 +00:00
Martin Sumner
8388a147f2
Merge pull request #297 from martinsumner/master 
Update from master
 2019-08-29 21:45:20 +01:00
Martin Sumner
432fe71bf0
Merge pull request #296 from martinsumner/mas-i293-cachemonitor 
Mas i293 cachemonitor
 2019-08-29 19:41:03 +01:00
Martin Sumner
78ef767a96 Merge branch 'master' into mas-i293-cachemonitor 2019-08-29 17:49:04 +01:00
Martin Sumner
d5c1d1e51e Log about cache ratio and object hit ratio 
If not a snapshot.
 2019-08-29 17:48:46 +01:00
Martin Sumner
b7eb2b08dc
Merge pull request #295 from martinsumner/mas-i294-syncintest 
Test improvements
 2019-08-29 15:44:18 +01:00
Martin Sumner
ecda13872a Add logging of cache ratio 
Two reasons for logging this:

- to assist in sizing the ledger cache;
- to resolve the mystery when there appear to be no fetches from the penciller (as the penciller does not report fetches from the ledger cache)
 2019-08-29 11:26:29 +01:00
Martin Sumner
0966ce9929 Test improvements 
Improve the speed of leveled_cdb tests by disabling sync on write.

Improve the strength of check of the correct behaviour when compacting with a reduced journal size.
 2019-08-29 10:32:07 +01:00
Martin Sumner
e3913a6c07
Merge pull request #292 from martinsumner/mas-i287-config 
Make page cache level configurable
 2019-07-26 21:43:58 +01:00
Martin Sumner
8587686783 Add testing to ensure keydeltas are compacted in test 2019-07-26 21:43:00 +01:00
Martin Sumner
57d73fc548 Make page cache level configurable 2019-07-25 12:23:10 +01:00
Martin Sumner
6cd898b731
Update leveled.app.src 2019-07-25 11:12:35 +01:00
Martin Sumner
02c1c57edf
Merge pull request #291 from martinsumner/mas-i289-config 
Mas i289 config
 2019-07-25 11:11:07 +01:00
Martin Sumner
e7c8dd7a78 Typo round-up 
Also reduce log noise when persisting new Journal files
 2019-07-25 10:24:40 +01:00
Martin Sumner
dab9652f6c Add ability to control journal size by object count 
This helps when there are files wiht large numbers of key deltas (and hence small values), where otherwise the object count may get out of control.
 2019-07-25 09:45:23 +01:00
Martin Sumner
0ba7b3347e
Merge pull request #290 from martinsumner/mas-i289-compactingkeydeltas 
Compaction of already compacted journals
 2019-07-24 19:41:38 +01:00
Martin Sumner
22e732841c Compaction of already compacted journals 
Ensure that journals with a large volume of key deltas do not erroneously get repeatedly compacted.
 2019-07-24 18:03:22 +01:00
Martin Sumner
21e0ce70e7
Merge pull request #288 from martinsumner/mas-i287-memory 
Mas i287 memory
 2019-07-24 12:25:39 +01:00
Martin Sumner
90909e7c17 Delete current_counterexample.eqc 2019-07-22 12:16:03 +01:00
Martin Sumner
5bef21d971 Add unit test to prove binary/copy issue 
Need to understand why the binary:copy is necessary - unit test now shows this.
 2019-07-22 10:35:55 +01:00
Martin Sumner
c9c577259e Need to binary copy the header 
Otherwise the whole binary is kept in memory ... and the SST memory footprint is much bigger.
 2019-07-19 13:37:27 +01:00
Martin Sumner
da1ecc144a Tidy-up GC on compaction 
Make sure we hibernate any CDB files after we score them, as they may not be used for sometime, and there may be garbage binary references present.
 2019-07-19 13:30:53 +01:00
Martin Sumner
7862a6c523 Change page cache loading by lookup/no_lookup 
By default load the first 4 levels of the ledger into the page cache of lookup is to be supported, but just levels 0 and 1 otherwise.
 2019-07-18 14:00:19 +01:00