umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1606 commits 1 branch 0 tags 25 MiB
Commit graph

1606 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
60e29f2ff0 (slightly) less random reads on journal compaction 2020-03-06 11:29:25 +00:00
Martin Sumner
5b11462054
Merge pull request #305 from martinsumner/mas-i304-otp22 
Mas i304 otp22
 2020-02-24 12:49:25 +00:00
Martin Sumner
12e84ad628 Don't add later OTP until we can test it 2020-02-24 12:42:40 +00:00
Martin Sumner
009abdd599 Build and test on OTP 22 2020-02-24 09:55:05 +00:00
Martin Sumner
02155558df
Tag for Riak 2.9.1 2020-02-12 09:26:53 +00:00
Martin Sumner
681f243e87 Merge branch 'master' into develop-3.0 2019-11-20 10:34:03 +00:00
Martin Sumner
4d550ef2a1
Bump version for new release 2019-11-20 10:33:51 +00:00
Martin Sumner
b3de5d46c0 Merge branch 'master' into develop-2.9 2019-11-20 09:51:24 +00:00
Martin Sumner
b8976531b1
Merge pull request #302 from martinsumner/mas-i301-saferename 
Mas i301 saferename
 2019-11-19 21:47:24 +00:00
Martin Sumner
bcf10c9709 Fixup comments 2019-11-19 16:36:57 +00:00
Martin Sumner
693defb6d3 Use the same file write/sync/rename path where needed 
When we want to be sure a file has been written before proceeding - we need a safer (that `file:write_file/2`) mechanism to be sure that it is written before proceeding.

This will:
open, write, sync, rename and then optionally read-back.

Changed so that manifest writing uses the safest form (including read back), and that sst writing uses a slightly looser form (with no read back to avoid performance issues).
 2019-11-19 15:50:59 +00:00
Martin Sumner
65bdb40277 Merge branch 'master' into develop-2.9 2019-11-18 13:37:30 +00:00
Martin Sumner
b946271195 Merge branch 'master' into develop-3.0 2019-11-05 20:17:39 +00:00
Martin Sumner
e16b650823
Merge pull request #300 from martinsumner/travisci-test 
Travisci test
 2019-11-05 20:01:59 +00:00
Martin Sumner
8ce9f29c88 Add travis status 2019-11-05 19:54:11 +00:00
Martin Sumner
293b5abed1 Add latest rebar3 3.12.0 2019-11-05 19:42:39 +00:00
Martin Sumner
7bf4faed9a Create .travis.yml 2019-11-05 19:36:20 +00:00
Martin Sumner
e4a5ac8776 Merge branch 'master' into develop-3.0 2019-11-04 22:27:10 +00:00
Martin Sumner
4b967bee4a
Merge pull request #299 from martinsumner/mas-i298-corruptedblocks 
Mas i298 corruptedblocks
 2019-11-04 18:19:34 +00:00
Martin Sumner
96779e667e Add specs and further tests 
Prove corrupted blocks are handled as expected when detected from within check_blocks function.
 2019-11-04 15:32:18 +00:00
Martin Sumner
843b28313d Handle when corrupted blocks creates empty list 
So can't get nth or last.
 2019-11-04 12:49:56 +00:00
Martin Sumner
aa7bb78cab Update rebar.config 
Switch to develop-3.0 lz4
 2019-11-01 13:41:10 +00:00
Martin Sumner
8388a147f2
Merge pull request #297 from martinsumner/master 
Update from master
 2019-08-29 21:45:20 +01:00
Martin Sumner
432fe71bf0
Merge pull request #296 from martinsumner/mas-i293-cachemonitor 
Mas i293 cachemonitor
 2019-08-29 19:41:03 +01:00
Martin Sumner
78ef767a96 Merge branch 'master' into mas-i293-cachemonitor 2019-08-29 17:49:04 +01:00
Martin Sumner
d5c1d1e51e Log about cache ratio and object hit ratio 
If not a snapshot.
 2019-08-29 17:48:46 +01:00
Martin Sumner
b7eb2b08dc
Merge pull request #295 from martinsumner/mas-i294-syncintest 
Test improvements
 2019-08-29 15:44:18 +01:00
Martin Sumner
ecda13872a Add logging of cache ratio 
Two reasons for logging this:

- to assist in sizing the ledger cache;
- to resolve the mystery when there appear to be no fetches from the penciller (as the penciller does not report fetches from the ledger cache)
 2019-08-29 11:26:29 +01:00
Martin Sumner
0966ce9929 Test improvements 
Improve the speed of leveled_cdb tests by disabling sync on write.

Improve the strength of check of the correct behaviour when compacting with a reduced journal size.
 2019-08-29 10:32:07 +01:00
Martin Sumner
e3913a6c07
Merge pull request #292 from martinsumner/mas-i287-config 
Make page cache level configurable
 2019-07-26 21:43:58 +01:00
Martin Sumner
8587686783 Add testing to ensure keydeltas are compacted in test 2019-07-26 21:43:00 +01:00
Martin Sumner
57d73fc548 Make page cache level configurable 2019-07-25 12:23:10 +01:00
Martin Sumner
6cd898b731
Update leveled.app.src 2019-07-25 11:12:35 +01:00
Martin Sumner
02c1c57edf
Merge pull request #291 from martinsumner/mas-i289-config 
Mas i289 config
 2019-07-25 11:11:07 +01:00
Martin Sumner
e7c8dd7a78 Typo round-up 
Also reduce log noise when persisting new Journal files
 2019-07-25 10:24:40 +01:00
Martin Sumner
dab9652f6c Add ability to control journal size by object count 
This helps when there are files wiht large numbers of key deltas (and hence small values), where otherwise the object count may get out of control.
 2019-07-25 09:45:23 +01:00
Martin Sumner
0ba7b3347e
Merge pull request #290 from martinsumner/mas-i289-compactingkeydeltas 
Compaction of already compacted journals
 2019-07-24 19:41:38 +01:00
Martin Sumner
22e732841c Compaction of already compacted journals 
Ensure that journals with a large volume of key deltas do not erroneously get repeatedly compacted.
 2019-07-24 18:03:22 +01:00
Martin Sumner
21e0ce70e7
Merge pull request #288 from martinsumner/mas-i287-memory 
Mas i287 memory
 2019-07-24 12:25:39 +01:00
Martin Sumner
90909e7c17 Delete current_counterexample.eqc 2019-07-22 12:16:03 +01:00
Martin Sumner
5bef21d971 Add unit test to prove binary/copy issue 
Need to understand why the binary:copy is necessary - unit test now shows this.
 2019-07-22 10:35:55 +01:00
Martin Sumner
c9c577259e Need to binary copy the header 
Otherwise the whole binary is kept in memory ... and the SST memory footprint is much bigger.
 2019-07-19 13:37:27 +01:00
Martin Sumner
da1ecc144a Tidy-up GC on compaction 
Make sure we hibernate any CDB files after we score them, as they may not be used for sometime, and there may be garbage binary references present.
 2019-07-19 13:30:53 +01:00
Martin Sumner
7862a6c523 Change page cache loading by lookup/no_lookup 
By default load the first 4 levels of the ledger into the page cache of lookup is to be supported, but just levels 0 and 1 otherwise.
 2019-07-18 14:00:19 +01:00
Martin Sumner
85bfa7fbb4 Use hibernate not garbage_collect 
Use hibernation rather than manual garbage_collect calls as per standard recommendation.  Hibernate will be default gabage_collect anyway.  Maybe help with SST files that naturally go quiet.

Plus typos from previous commit in leveled_cdb.
 2019-07-18 13:21:38 +01:00
Martin Sumner
5a853ee44d Hibernate iclerk on completion of compaction 
Will be inactive for a period.  Will also force garbage collection.
 2019-07-18 13:10:11 +01:00
Martin Sumner
3c834afa08 Use hibernate on open or roll to read 
CDB files may be opened or rolled then left untouched for a period, so clean up any memory.  Been awoken from hibernate has a cost, but it is a rare event.
 2019-07-18 13:07:48 +01:00
Martin Sumner
478c5b6db0 Load ledger in reverse order 
Now that the SST files will fadvise on load (to force load into the page cache).  The load should take place in reverse order, so that if th eledger is > page_cache, it is the higher levels that will end up in the cache at the expense of the lower levels.
 2019-07-16 10:25:49 +01:00
Martin Sumner
f8b3101a3a Two memory management helpers 
Two helpers for memory management:

1 - a scan over the cdb file may lead to a lot of binary references being made.  So force a GC fater the scan.

2 - the penciller files contain slots that will be frequently read - so advice the page cache to pre-load them on startup.

This is in response to unexpected memory mangement issues in a potentially non-conventional setup - where the erlang VM held a lot of memory (that could be GC'd , in preference to the page cache - and consequently disk I/O and request latency were higher than expected.
 2019-07-15 13:44:39 +01:00
Martin Sumner
b2d4d766cd
Update leveled.app.src 
Ready for release of 0.9.16
 2019-06-19 10:32:12 +01:00