Commit graph

1423 commits

Author SHA1 Message Date
Martin Sumner
a589c9ca63 Update leveled_sst.erl
Handle deprecation warning
2019-02-26 12:31:34 +00:00
Martin Sumner
fffa257ffb Update leveled_sst.erl
Remove abritrarily reduced timings.  Can cause problems when testing with coverage enabled
2019-02-26 12:25:42 +00:00
Martin Sumner
7dd07080c7 Double-check safety of rolling memory
Make sure there is no change pending regardless of why maybe_roll_memory has been called.

Also, check that the manifest SQN has been incremented before accepting the change.

Conflict here would lead to data loss in the penciller, so extra safety is important.
2019-02-26 10:33:20 +00:00
Martin Sumner
fd2e0e870c Align cache with default
Stop L0 from growing too large i.e. 127 * 4K is more than 10 times bigger than the default.
2019-02-25 23:35:12 +00:00
Martin Sumner
aea20de4b9
Merge pull request #261 from martinsumner/mas-i260-journalsnap
Check SQN order fold does not fold beyond end of snapshot
2019-02-15 09:26:25 +00:00
Martin Sumner
db1486fa36 Check SQN order fold does not fold beyond end of snapshot
the Journla snapshot is not a true snapshot, in that the active file in the snapshot can still be taking appends.  So when getting a snapshot it is necessary to check if folding over the snapshot that the SQN is <= JournalSQN when the snapshot is taken.

Normally consistency of the snapshot is managed as the operation depends on the penciller, and the penciller *is* a snapshot.  Not in this case, as the penciller will return true on a sqn check if the pcl SQN is behind the Journal.  So the Journal folder, has been given an additionla check to stop at the JournalSQN.

This is perhaps a fault in the pcl check sqn, which should only return true on an exact match?  I'm nervous about changing this though, so we have a less pure fix for now.
2019-02-14 21:14:11 +00:00
Martin Sumner
5799f06452
Merge pull request #259 from martinsumner/mas-i258-riakmaketest
Mas i258 riakmaketest
2019-02-14 15:21:26 +00:00
Martin Sumner
1776220148 Log and coverage tidy
These logs duplicate information being received from other logs, so reduced to debug.

The long running test needs to change with the LONG_RUNNING macro
2019-02-14 15:20:54 +00:00
Martin Sumner
6c2fbcb80e Ugly test hack
This test (prior tot his commit) works fine.  However, something about running it with riak `make test` causes it to fail.  The process crashes when the file:delete(F2) is called.

As the test works in isolation on R16, and also because  aprocess crashing in the real world in this stage would not be the end of the world (this whole part was added as a way of dealing with some unlikely but possible tidy-up scenarios in eqc tests), losing the test in R16 is tolerable.

So In R16 tests (which includes riak make test), the delete will no longer be called in this test.
2019-02-14 15:17:07 +00:00
Martin Sumner
5a1cb70d76
Merge pull request #257 from martinsumner/mas-i256-loglevel
Mas i256 loglevel
2019-02-13 14:57:57 +00:00
Martin Sumner
10f250e4be Update leveled.schema
To set log level
2019-02-13 14:36:45 +00:00
Martin Sumner
31badfd601 Log rationalisation and format improvements 2019-02-13 13:06:28 +00:00
Martin Sumner
baa8353db7 Issue 243 2019-02-13 11:15:38 +00:00
Martin Sumner
2061ff85db Revert "Resolve Issue 243"
This reverts commit 0b0fde2f23.
2019-02-13 11:14:48 +00:00
Martin Sumner
0b0fde2f23 Resolve Issue 243 2019-02-13 11:10:56 +00:00
Martin Sumner
5563510da7
Merge pull request #253 from martinsumner/mas-i249-sstcloseraces
Mas i249 sstcloseraces
2019-01-29 15:32:06 +00:00
Martin Sumner
509d541c9f Allow for false to close not crash
If PID has gone away
2019-01-29 13:46:25 +00:00
Martin Sumner
be6e23f7de Change cache_size in sst tests
Makes results more predictable (with coin toss variations)
2019-01-29 13:40:55 +00:00
Martin Sumner
51a0260a60 Get new file to check initiater is alive
If no activity within timeout.  Make sure that the process has been orphaned by pclerk ending before manifest entry update made.
2019-01-29 13:18:39 +00:00
Martin Sumner
e5a5da35eb Level Zero constructor to get close
Allows for the L0 constructor to be closed (even though not yet in the manifest) on shutdown
2019-01-29 13:06:00 +00:00
Martin Sumner
b12b6e4c91
Merge pull request #252 from martinsumner/mas-i249-iclerkandconfig
Mas i249 iclerkandconfig
2019-01-28 08:56:36 +00:00
Martin Sumner
e3bd83179a Uncomment tests! 2019-01-27 23:31:44 +00:00
Martin Sumner
db0db67c45 Delete leveledjc_eqc.erl
Remove this for now, until issues with running tests without QC installed can be resolved.

Allow for changes to support QC to be merged into master.
2019-01-27 22:09:48 +00:00
Martin Sumner
8f6862a10b Test sst slot configuration change
Confirm it results in many more files, if the slot count reduced.  Has to handle the fact that Level 0 file has unlimited slots regardless of number of slots configured
2019-01-27 22:03:55 +00:00
Martin Sumner
ae9b03ab3c Fix unit tests - and make slot size configurable 2019-01-26 16:57:25 +00:00
Martin Sumner
f907fb5c97 Close in all cases
in leveled_imanifest
2019-01-25 19:27:42 +00:00
Martin Sumner
a04ed53855 Merge branch 'mas-qc-inkercompaction' of https://github.com/martinsumner/leveled into mas-qc-inkercompaction 2019-01-25 19:11:37 +00:00
Martin Sumner
f7022627e5 Check not pending before compacting
Also check for existence before deleting a CDB file
2019-01-25 19:11:34 +00:00
Martin Sumner
a41183b8cb
Merge pull request #251 from martinsumner/mas-i249-iclerkfsm
Mas i249 iclerkfsm
2019-01-25 15:16:08 +00:00
Martin Sumner
c9a955f2dd
Merge branch 'mas-qc-inkercompaction' into mas-i249-iclerkfsm 2019-01-25 15:15:29 +00:00
Martin Sumner
5b54affbf0 Have inker reopen compacted files
The inker cler will now close compacted files before prompting the inker to update the manifest.  The inker should reopen those files, so that the file processes are linked to it and not the clerk.

This also stops a stopped clerk leading to orphaned cdb files.
2019-01-25 14:32:41 +00:00
Martin Sumner
e349774167 Allow clerk to be stopped during compaction scoring
This will stop needless compaction work from being completed when the iclerk is sent a close at this stage.
2019-01-25 12:11:42 +00:00
Martin Sumner
5fab9e2d62 Update .gitignore 2019-01-25 10:25:55 +00:00
Martin Sumner
7801f16de9 Update .gitignore 2019-01-25 10:24:47 +00:00
Martin Sumner
2c1503b6b5
Merge pull request #250 from martinsumner/mas-i249-iclerkshutdown
Mas i249 iclerkshutdown
2019-01-25 09:54:14 +00:00
Martin Sumner
00a59f4f8f
Merge branch 'mas-qc-inkercompaction' into mas-i249-iclerkshutdown 2019-01-25 09:53:56 +00:00
Martin Sumner
0333604fd9 Change to cast in inker/iclerk interaction
This allows for leveled_iclerk:clerk_stop to be a sync call, so that files will only be closed once the iclerk has stopped.  This is designed ot prevent iclerk crashes during shutdowns when files it is depnding on are closed mid shutdown.
2019-01-24 21:32:54 +00:00
Martin Sumner
28d0aef5fe Make check that compaction not ongoing before accepting new compaction
Respond 'busy' if compaction is ongoing
2019-01-24 15:46:17 +00:00
Martin Sumner
c9bf43953b Expect TS in snapshot references from manifest
Add type to prevent re-occurence.  This is also detected by failure in eqc tests.
2019-01-24 14:32:01 +00:00
Martin Sumner
a13a6ae45f Updated model
This has inappropriate default parameter changes.
2019-01-22 12:53:31 +00:00
Martin Sumner
b713ce60a8 Initial eqc setup 2019-01-21 10:51:07 +00:00
Martin Sumner
67104c85c1
Merge pull request #248 from martinsumner/mas-i246-dotdotwhy
Change file references in unit tests
2019-01-17 21:03:20 +00:00
Martin Sumner
7f08fd5a68 Change file references in unit tests
Write into test folder within the repo, not outside of it.  Try and resolve issues wiht make test in riak
2019-01-17 21:02:29 +00:00
Martin Sumner
3e75eb3647
Merge pull request #247 from martinsumner/mas-i246-maketestfail
Remove pointless test
2019-01-17 11:31:17 +00:00
Martin Sumner
86128ddf56 Remove pointless test
Was testing a test admin function for coverage reasons
2019-01-17 11:30:50 +00:00
Martin Sumner
267e103e58
Merge pull request #245 from martinsumner/mas-i244-startupcachesizes
Mas i244 startupcachesizes
2019-01-14 18:48:39 +00:00
Martin Sumner
a4d89ad6d1 Add log of higher than expected ratio of cache sizes
Warn at startup if this ratio is high.  Not sure how snapshots will perform if there are a lot of ledger cache sin the list.  However, it should still work.  basic_SUITE/load_count test intended to demonstrate that a large ratio is still functional
2019-01-14 16:11:04 +00:00
Martin Sumner
c060c0e41d Handle L0 cache being full
A test thta will cause leveled to crash due to a low cache size being set - but protect against this (as well as the general scenario of the cache being full).

There could be a potential case where a L0 file present (post pending) without work backlog being set.  In this case we want to roll the level zero to memory, but don't accept the cache update if the L0 cache is already full.
2019-01-14 12:27:51 +00:00
Martin Sumner
d5a9f2e8b7
Merge pull request #242 from martinsumner/mas-i241-eunitfailure
Creating the file is unnecessary
2018-12-18 16:27:38 +00:00
Martin Sumner
e77451f60b Creating the file is unnecessary 2018-12-18 16:26:48 +00:00