Commit graph

752 commits

Author SHA1 Message Date
Martin Sumner
4cf3eea1eb Merge pull request #76 from martinsumner/mas-compactionconfig
Change default compaction settings.

The change is as a result of the 12h volume tests, which showed a need to do more compaction work per cycle.
2017-06-02 08:42:20 +01:00
Martin Sumner
15c52ae118 Change default compaction settings
Need to allow specific settings to be passed into unit tests.

Also, too much journal compaction may lead to intermittent failures on
the basic_SUITE space_clear_on_delete test.  think this is because
there are less “deletes” to reload in on startup to trigger the cascade
down and clear up?
2017-06-02 08:37:57 +01:00
Martin Sumner
39fd3e5c87 Merge pull request #75 from martinsumner/mas-arraytype-i73
Mas arraytype i73
2017-06-01 22:28:17 +01:00
martinsumner
569b498727 Resolve dialyzer warnings
Botched switch to leveled_log in list - so reoslved dialyzer warnings
2017-06-01 22:03:51 +01:00
martinsumner
32612dfe4a Yet another array type OTP16 issue 2017-06-01 21:39:01 +01:00
martinsumner
afbf918f2c Change from using array type
Won't compile in OTP16
2017-06-01 21:37:23 +01:00
Martin Sumner
1a051eab35 Merge pull request #73 from martinsumner/mas-logbase-i72
Change log to keyfind
2017-06-01 10:30:53 +01:00
martinsumner
204a5a8caa Change log to keyfind
Save doing dict:to_list each call to log
2017-05-31 10:56:48 +01:00
Martin Sumner
92275cfc2f Merge pull request #71 from martinsumner/mas-specs-i61b
Mas specs i61b
2017-05-30 20:34:25 +01:00
martinsumner
e66e326c2b Dialyzer help
Add specs and docs
2017-05-30 16:50:23 +01:00
martinsumner
e86d91dcf6 Give dialyzer a hand
add specs and docs
2017-05-30 16:25:55 +01:00
martinsumner
c073344163 Dialyzer helper
Add specs and docs to external API
2017-05-30 15:36:50 +01:00
martinsumner
fc4802c539 Dialyzer help
Add specs and docs
2017-05-30 11:37:10 +01:00
martinsumner
0459591fd7 spec and doc - leveled_pmanifest
Give the dialyzer some help
2017-05-26 15:11:33 +01:00
martinsumner
f23072fb54 Merge remote-tracking branch 'refs/remotes/origin/master' into mas-specs-i61b 2017-05-26 13:45:20 +01:00
Martin Sumner
041da54c62 Merge pull request #70 from martinsumner/mas-iclerkrelease-i69
Release penciller snapshot after journal compaction
2017-05-26 13:44:49 +01:00
martinsumner
c664176247 Release penciller snapshot after journal compaction
As otherwise memory consumption beocmes an issue, as they will take an
hour to timeout naturally.
2017-05-26 10:51:30 +01:00
Martin Sumner
5c5fa8ee8d Merge pull request #68 from martinsumner/mas-isempty-i67
Mas isempty i67
2017-05-23 15:54:47 +01:00
Martin Sumner
96a548e17a Change tests - binary keys
the new code requires bucket listing to be on binary keys not just
binary buckets.  As this is only intended for use within Riak (where
all keys are buckets are binaries), this constraint seems OK.

A test needed changing to ensure it had a binary key in the bucket.
2017-05-23 15:54:11 +01:00
martinsumner
0d8ab0899e Add test for is_empty
Bucket listing didn't care if keys were active - now does.
2017-05-23 11:59:44 +01:00
martinsumner
2738ee951c Add specs and docs for Penciller
Help the dialyzer
2017-05-22 18:09:12 +01:00
martinsumner
366253831a Add dpecs to inker
Give the dialyzer a hand
2017-05-22 15:31:42 +01:00
Martin Sumner
7ca672d7bc Merge pull request #66 from martinsumner/mas-cdbindex-i65
Switch IndexCache to tuple
2017-05-22 13:12:56 +01:00
martinsumner
dfcb30a33e Switch IndexCache to tuple
So that direct element fetch should be faster?
2017-05-22 10:24:49 +01:00
Martin Sumner
f45681610e Merge pull request #64 from martinsumner/mas-specs-i61a
Mas specs i61a
2017-05-22 10:03:07 +01:00
martinsumner
c90e0f824d Spelling error change 2017-05-22 10:00:34 +01:00
martinsumner
a81dd2839e Merge remote-tracking branch 'refs/remotes/origin/master' into mas-specs-i61a 2017-05-22 09:58:41 +01:00
Martin Sumner
8e334c0b5a Merge pull request #63 from martinsumner/mas-compactscore-i35
Mas compactscore i35
2017-05-22 09:57:32 +01:00
Martin Sumner
2bbb504c02 Up max file compactions per run
Try and do more compaction work for each run
2017-05-21 22:06:41 +01:00
martinsumner
695638f34c Loop for get_positions
The CDB FSM process can be blocked by get_positions for all positions,
so loop around the index outside of the FSM process to allow for other
messages to interleave.
2017-05-20 12:25:06 +01:00
martinsumner
319dc5f388 CDB changes to add timing log
Forgot to save before commit last time!
2017-05-19 14:28:55 +01:00
martinsumner
36a48c16e5 Add timing log to position scan
Position scan may be very expensive, add timing log to confirm
2017-05-19 13:59:57 +01:00
martinsumner
18a12ff9ff Improve comments 2017-05-18 14:09:45 +01:00
martinsumner
8b3ca78d49 spec help for SST file 2017-05-18 12:29:56 +01:00
martinsumner
28f45749d7 Add specs to exported API of CDB files
Time to give the dialyzer some help
2017-05-17 12:54:02 +01:00
Martin Sumner
e4c6dc7119 Update VOLUME.md
Minor edits for clarity
2017-04-24 11:56:11 +01:00
Martin Sumner
b0c91172f4 Merge pull request #60 from martinsumner/mas-sweeperfold-i59
Mas sweeperfold i59
2017-04-24 11:39:15 +01:00
martinsumner
d5860790be Update future branch descriptions 2017-04-24 11:33:05 +01:00
martinsumner
a052edaea0 Add volume test information
Volume testing with AAE rebuilds
2017-04-24 11:24:14 +01:00
martinsumner
fbb4879d81 Change fold_heads to do basic Journal presence check
This at least checks the file is present, and the Key exists in the
index of that file.  If the value is corrupt it will be removed by
compation, and then this will fail (unless the file is never compacted).

TODO: resolve issus of files which are corrupt - but never compacted
- a job for backup?
2017-04-21 15:55:03 +01:00
martinsumner
4d12dfe0ab Returning snapshots
If the clerk updates the manifest - it might not recognise changes to
the manifest made since the clerk took the manifest.  So the penciller
must merge its view of the snapshots back in to the updated manifest
2017-04-19 22:46:37 +01:00
Martin Sumner
fa9daf8696 Correct async fold
fold objects which snaps in the fold was implemented incorrectly - it
took information from the LedgeCache at the point of the request, not
at the point of the fold.  So the LedgerCache SQN may have been
surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
martinsumner
7cba182951 Merge remote-tracking branch 'origin/mas-sweeperfold-i59' into mas-sweeperfold-i59
# Conflicts:
#	src/leveled_bookie.erl
2017-04-17 15:07:35 +01:00
martinsumner
50d95ef6aa Move snapshot inside of the fold function
riak_kv_sweeper gets the async fold function, then determines if the
function can be called.  If the system is busy the fold may be queued,
and may never be acted upon.

This may cause issues with snapshot timeouts etc.
2017-04-17 15:03:03 +01:00
Martin Sumner
e01efe02f6 Long snaphsot timeout increase
Increase this to 90 minutes.  The first time all the snapshots are
rebuilt it may take a long time, but they all get scheduled together -
and queued until concurrency limits allow it to be completed.

currently the snapshot is made on initialisation, and only released
when completed (which may be after the queue).  so the last couple of
snapshots were over-shooting the 1 hour.
2017-04-13 22:43:29 +01:00
Martin Sumner
618d9cf53b FoldHeads to output binary
so that byte_size will work in sweeper
2017-04-11 11:17:27 +01:00
Martin Sumner
9badc8fbe7 Merge branch 'master' into mas-sweeperfold-i59 2017-04-10 21:49:08 +01:00
Martin Sumner
13a1574fa0 Merge pull request #57 from martinsumner/mas-metadatabin
Mas metadatabin
2017-04-10 21:47:45 +01:00
martinsumner
60b0b88226 Switch to binary response
Force header in fold_heads to be a binary ... as this is what KV expects
2017-04-07 17:08:40 +00:00
martinsumner
6d42abbc1a Add bybucket to fold_heads
give fold_heads equivalent functionality to fold_objects - both can now
be done for allkeys and bybucket
2017-04-07 14:56:28 +00:00