umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1492 commits 1 branch 0 tags 25 MiB
Commit graph

1492 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
65bdb40277 Merge branch 'master' into develop-2.9 2019-11-18 13:37:30 +00:00
Martin Sumner
e16b650823
Merge pull request #300 from martinsumner/travisci-test 
Travisci test
 2019-11-05 20:01:59 +00:00
Martin Sumner
8ce9f29c88 Add travis status 2019-11-05 19:54:11 +00:00
Martin Sumner
293b5abed1 Add latest rebar3 3.12.0 2019-11-05 19:42:39 +00:00
Martin Sumner
7bf4faed9a Create .travis.yml 2019-11-05 19:36:20 +00:00
Martin Sumner
4b967bee4a
Merge pull request #299 from martinsumner/mas-i298-corruptedblocks 
Mas i298 corruptedblocks
 2019-11-04 18:19:34 +00:00
Martin Sumner
96779e667e Add specs and further tests 
Prove corrupted blocks are handled as expected when detected from within check_blocks function.
 2019-11-04 15:32:18 +00:00
Martin Sumner
843b28313d Handle when corrupted blocks creates empty list 
So can't get nth or last.
 2019-11-04 12:49:56 +00:00
Martin Sumner
8388a147f2
Merge pull request #297 from martinsumner/master 
Update from master
 2019-08-29 21:45:20 +01:00
Martin Sumner
432fe71bf0
Merge pull request #296 from martinsumner/mas-i293-cachemonitor 
Mas i293 cachemonitor
 2019-08-29 19:41:03 +01:00
Martin Sumner
78ef767a96 Merge branch 'master' into mas-i293-cachemonitor 2019-08-29 17:49:04 +01:00
Martin Sumner
d5c1d1e51e Log about cache ratio and object hit ratio 
If not a snapshot.
 2019-08-29 17:48:46 +01:00
Martin Sumner
b7eb2b08dc
Merge pull request #295 from martinsumner/mas-i294-syncintest 
Test improvements
 2019-08-29 15:44:18 +01:00
Martin Sumner
ecda13872a Add logging of cache ratio 
Two reasons for logging this:

- to assist in sizing the ledger cache;
- to resolve the mystery when there appear to be no fetches from the penciller (as the penciller does not report fetches from the ledger cache)
 2019-08-29 11:26:29 +01:00
Martin Sumner
0966ce9929 Test improvements 
Improve the speed of leveled_cdb tests by disabling sync on write.

Improve the strength of check of the correct behaviour when compacting with a reduced journal size.
 2019-08-29 10:32:07 +01:00
Martin Sumner
e3913a6c07
Merge pull request #292 from martinsumner/mas-i287-config 
Make page cache level configurable
 2019-07-26 21:43:58 +01:00
Martin Sumner
8587686783 Add testing to ensure keydeltas are compacted in test 2019-07-26 21:43:00 +01:00
Martin Sumner
57d73fc548 Make page cache level configurable 2019-07-25 12:23:10 +01:00
Martin Sumner
6cd898b731
Update leveled.app.src 2019-07-25 11:12:35 +01:00
Martin Sumner
02c1c57edf
Merge pull request #291 from martinsumner/mas-i289-config 
Mas i289 config
 2019-07-25 11:11:07 +01:00
Martin Sumner
e7c8dd7a78 Typo round-up 
Also reduce log noise when persisting new Journal files
 2019-07-25 10:24:40 +01:00
Martin Sumner
dab9652f6c Add ability to control journal size by object count 
This helps when there are files wiht large numbers of key deltas (and hence small values), where otherwise the object count may get out of control.
 2019-07-25 09:45:23 +01:00
Martin Sumner
0ba7b3347e
Merge pull request #290 from martinsumner/mas-i289-compactingkeydeltas 
Compaction of already compacted journals
 2019-07-24 19:41:38 +01:00
Martin Sumner
22e732841c Compaction of already compacted journals 
Ensure that journals with a large volume of key deltas do not erroneously get repeatedly compacted.
 2019-07-24 18:03:22 +01:00
Martin Sumner
21e0ce70e7
Merge pull request #288 from martinsumner/mas-i287-memory 
Mas i287 memory
 2019-07-24 12:25:39 +01:00
Martin Sumner
90909e7c17 Delete current_counterexample.eqc 2019-07-22 12:16:03 +01:00
Martin Sumner
5bef21d971 Add unit test to prove binary/copy issue 
Need to understand why the binary:copy is necessary - unit test now shows this.
 2019-07-22 10:35:55 +01:00
Martin Sumner
c9c577259e Need to binary copy the header 
Otherwise the whole binary is kept in memory ... and the SST memory footprint is much bigger.
 2019-07-19 13:37:27 +01:00
Martin Sumner
da1ecc144a Tidy-up GC on compaction 
Make sure we hibernate any CDB files after we score them, as they may not be used for sometime, and there may be garbage binary references present.
 2019-07-19 13:30:53 +01:00
Martin Sumner
7862a6c523 Change page cache loading by lookup/no_lookup 
By default load the first 4 levels of the ledger into the page cache of lookup is to be supported, but just levels 0 and 1 otherwise.
 2019-07-18 14:00:19 +01:00
Martin Sumner
85bfa7fbb4 Use hibernate not garbage_collect 
Use hibernation rather than manual garbage_collect calls as per standard recommendation.  Hibernate will be default gabage_collect anyway.  Maybe help with SST files that naturally go quiet.

Plus typos from previous commit in leveled_cdb.
 2019-07-18 13:21:38 +01:00
Martin Sumner
5a853ee44d Hibernate iclerk on completion of compaction 
Will be inactive for a period.  Will also force garbage collection.
 2019-07-18 13:10:11 +01:00
Martin Sumner
3c834afa08 Use hibernate on open or roll to read 
CDB files may be opened or rolled then left untouched for a period, so clean up any memory.  Been awoken from hibernate has a cost, but it is a rare event.
 2019-07-18 13:07:48 +01:00
Martin Sumner
478c5b6db0 Load ledger in reverse order 
Now that the SST files will fadvise on load (to force load into the page cache).  The load should take place in reverse order, so that if th eledger is > page_cache, it is the higher levels that will end up in the cache at the expense of the lower levels.
 2019-07-16 10:25:49 +01:00
Martin Sumner
f8b3101a3a Two memory management helpers 
Two helpers for memory management:

1 - a scan over the cdb file may lead to a lot of binary references being made.  So force a GC fater the scan.

2 - the penciller files contain slots that will be frequently read - so advice the page cache to pre-load them on startup.

This is in response to unexpected memory mangement issues in a potentially non-conventional setup - where the erlang VM held a lot of memory (that could be GC'd , in preference to the page cache - and consequently disk I/O and request latency were higher than expected.
 2019-07-15 13:44:39 +01:00
Martin Sumner
b2d4d766cd
Update leveled.app.src 
Ready for release of 0.9.16
 2019-06-19 10:32:12 +01:00
Martin Sumner
bf828ee8c3
Merge pull request #286 from martinsumner/mas-i285-binarycopy 
Memory management
 2019-06-18 11:03:24 +01:00
Martin Sumner
952f088873 Memory management 
Extracting binary from within a binary leaves a reference to the whole of the original binary.

If there are a lot of very large objects received abck toback - this can explode the amount of memory the penciller appears to hold (and gc cannot resolve this).

To dereference from the larger binary, need to do a binary copy
 2019-06-15 17:23:06 +01:00
Martin Sumner
c6a873bc15
Merge pull request #284 from martinsumner/mas-i282-pcllogging 
Change default DB ID
 2019-06-14 19:00:38 +01:00
Martin Sumner
b5859ddde9 Change default DB ID 
An undefined database id will use 65536 not 0 (as 0 is commonly used when defining database ids in Riak)
 2019-06-14 11:19:37 +01:00
Martin Sumner
432945cbd8
Merge pull request #283 from martinsumner/mas-i282-pcllogging 
Mas i282 pcllogging
 2019-06-14 09:50:46 +01:00
Martin Sumner
876a023db1 Add database_id to options 
So that this can be recorded in logs
 2019-06-13 14:58:32 +01:00
Martin Sumner
c3a4f5118d Each merge log details of the level below 
Help with troubleshooting memory problems, and potential issues with GC
 2019-06-13 11:50:02 +01:00
Martin Sumner
da59901890
Update leveled.app.src 2019-05-23 12:19:32 +01:00
Martin Sumner
bc14a19754
Merge pull request #281 from martinsumner/mas-i280-manifesttidy 
GC manifest files when numbers skipped
 2019-05-23 12:16:20 +01:00
Martin Sumner
e360b97cfb GC manifest files when numbers skipped 
Otherwise list of old files perpetually grows
 2019-05-23 10:16:15 +01:00
Martin Sumner
9efd973cb4
Merge pull request #279 from martinsumner/mas-i278-ssttimeoutrace 
Handle timeout/message race
 2019-05-23 09:35:52 +01:00
Martin Sumner
744a521289 Handle timeout/message race 
When there is hevay PUT load, leveled_sst files could go into the delete-pending state befre the GC message is receieved - and the GC message would then interrupt the timeout cycle and lead ot the file not being GC'd until close.
 2019-05-23 09:34:54 +01:00
Martin Sumner
984a2aba5a
Merge pull request #276 from martinsumner/mas-i275-logging 
Mas i275 logging
 2019-05-11 16:00:33 +01:00
Martin Sumner
aedd515a5b Bump vsn for release 2019-05-11 15:59:52 +01:00