umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
880 commits 1 branch 0 tags 25 MiB
Commit graph

880 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
7ca672d7bc Merge pull request #66 from martinsumner/mas-cdbindex-i65 
Switch IndexCache to tuple
 2017-05-22 13:12:56 +01:00
martinsumner
dfcb30a33e Switch IndexCache to tuple 
So that direct element fetch should be faster?
 2017-05-22 10:24:49 +01:00
Martin Sumner
f45681610e Merge pull request #64 from martinsumner/mas-specs-i61a 
Mas specs i61a
 2017-05-22 10:03:07 +01:00
martinsumner
c90e0f824d Spelling error change 2017-05-22 10:00:34 +01:00
martinsumner
a81dd2839e Merge remote-tracking branch 'refs/remotes/origin/master' into mas-specs-i61a 2017-05-22 09:58:41 +01:00
Martin Sumner
8e334c0b5a Merge pull request #63 from martinsumner/mas-compactscore-i35 
Mas compactscore i35
 2017-05-22 09:57:32 +01:00
Martin Sumner
2bbb504c02 Up max file compactions per run 
Try and do more compaction work for each run
 2017-05-21 22:06:41 +01:00
martinsumner
695638f34c Loop for get_positions 
The CDB FSM process can be blocked by get_positions for all positions,
so loop around the index outside of the FSM process to allow for other
messages to interleave.
 2017-05-20 12:25:06 +01:00
martinsumner
319dc5f388 CDB changes to add timing log 
Forgot to save before commit last time!
 2017-05-19 14:28:55 +01:00
martinsumner
36a48c16e5 Add timing log to position scan 
Position scan may be very expensive, add timing log to confirm
 2017-05-19 13:59:57 +01:00
martinsumner
18a12ff9ff Improve comments 2017-05-18 14:09:45 +01:00
martinsumner
8b3ca78d49 spec help for SST file 2017-05-18 12:29:56 +01:00
martinsumner
28f45749d7 Add specs to exported API of CDB files 
Time to give the dialyzer some help
 2017-05-17 12:54:02 +01:00
Martin Sumner
e4c6dc7119 Update VOLUME.md 
Minor edits for clarity
 2017-04-24 11:56:11 +01:00
Martin Sumner
b0c91172f4 Merge pull request #60 from martinsumner/mas-sweeperfold-i59 
Mas sweeperfold i59
 2017-04-24 11:39:15 +01:00
martinsumner
d5860790be Update future branch descriptions 2017-04-24 11:33:05 +01:00
martinsumner
a052edaea0 Add volume test information 
Volume testing with AAE rebuilds
 2017-04-24 11:24:14 +01:00
martinsumner
fbb4879d81 Change fold_heads to do basic Journal presence check 
This at least checks the file is present, and the Key exists in the
index of that file.  If the value is corrupt it will be removed by
compation, and then this will fail (unless the file is never compacted).

TODO: resolve issus of files which are corrupt - but never compacted
- a job for backup?
 2017-04-21 15:55:03 +01:00
martinsumner
4d12dfe0ab Returning snapshots 
If the clerk updates the manifest - it might not recognise changes to
the manifest made since the clerk took the manifest.  So the penciller
must merge its view of the snapshots back in to the updated manifest
 2017-04-19 22:46:37 +01:00
Martin Sumner
fa9daf8696 Correct async fold 
fold objects which snaps in the fold was implemented incorrectly - it
took information from the LedgeCache at the point of the request, not
at the point of the fold.  So the LedgerCache SQN may have been
surpassed in the Penciller by the time the fold was called.
 2017-04-17 23:01:55 +01:00
martinsumner
7cba182951 Merge remote-tracking branch 'origin/mas-sweeperfold-i59' into mas-sweeperfold-i59 
# Conflicts:
#	src/leveled_bookie.erl
 2017-04-17 15:07:35 +01:00
martinsumner
50d95ef6aa Move snapshot inside of the fold function 
riak_kv_sweeper gets the async fold function, then determines if the
function can be called.  If the system is busy the fold may be queued,
and may never be acted upon.

This may cause issues with snapshot timeouts etc.
 2017-04-17 15:03:03 +01:00
Martin Sumner
e01efe02f6 Long snaphsot timeout increase 
Increase this to 90 minutes.  The first time all the snapshots are
rebuilt it may take a long time, but they all get scheduled together -
and queued until concurrency limits allow it to be completed.

currently the snapshot is made on initialisation, and only released
when completed (which may be after the queue).  so the last couple of
snapshots were over-shooting the 1 hour.
 2017-04-13 22:43:29 +01:00
Martin Sumner
618d9cf53b FoldHeads to output binary 
so that byte_size will work in sweeper
 2017-04-11 11:17:27 +01:00
Martin Sumner
9badc8fbe7 Merge branch 'master' into mas-sweeperfold-i59 2017-04-10 21:49:08 +01:00
Martin Sumner
13a1574fa0 Merge pull request #57 from martinsumner/mas-metadatabin 
Mas metadatabin
 2017-04-10 21:47:45 +01:00
martinsumner
60b0b88226 Switch to binary response 
Force header in fold_heads to be a binary ... as this is what KV expects
 2017-04-07 17:08:40 +00:00
martinsumner
6d42abbc1a Add bybucket to fold_heads 
give fold_heads equivalent functionality to fold_objects - both can now
be done for allkeys and bybucket
 2017-04-07 14:56:28 +00:00
martinsumner
9375baf636 Add unit test for foldheads 
compare foldheads foldobjects and hahstree_query output
 2017-04-07 14:19:25 +00:00
martinsumner
b464e2e28c Extend foldobjects to support proxy object 
To allow for folds which probbaly don't need values to not always to
have to fetch the value
 2017-04-07 12:09:11 +00:00
Martin Sumner
4e9fa2a206 Timeout long-running snapshots 
Add logic to timeout long-running snapshots.
 2017-04-05 09:16:01 +01:00
martinsumner
400f65f557 Switch to binary metadata 
Trya nd maintain binary format when stored in Ledger so less
swapping/changing as added and removed.
 2017-04-04 10:02:35 +00:00
martinsumner
43bfbe3e0e Add in scheduler function 
To assist in scheduling compaction
 2017-03-30 15:46:37 +01:00
martinsumner
11ff3129f3 Reduce compaction targets 
Cmpaction is overly aggressive.  It is a lot of work to compact a run of
files for just 20% reduction in disk space, when disk space for the
Journal (i.e. low IOPS disk space should be relatively inexpensive).
Require at least a 40% reduction for a compaction job.
 2017-03-30 12:15:36 +01:00
Martin Sumner
1146e843bc Merge pull request #56 from martinsumner/mas-inkfetch 
Remove binary_to_term
 2017-03-30 11:29:26 +01:00
martinsumner
6143fcb664 Remove binary_to_term 
when fetching don't need to binary_to_term key changes
 2017-03-29 15:37:04 +01:00
Martin Sumner
512af7f41a Missing space 2017-03-24 16:11:20 +00:00
Martin Sumner
1335b11d82 Merge pull request #55 from martinsumner/mas-docs2i 
Mas docs2i
 2017-03-24 16:09:17 +00:00
martinsumner
f838ffda2c Typo 2017-03-24 15:41:33 +00:00
martinsumner
6de9740bda Typo 2017-03-24 15:31:19 +00:00
Martin Sumner
eeae245442 Merge pull request #54 from martinsumner/mas-sstfiveblocks 
Mas sstfiveblocks
 2017-03-24 15:27:09 +00:00
martinsumner
5e1482d71b 2i query test results 
Write-up of 2i query test results
 2017-03-24 15:20:03 +00:00
Martin Sumner
8db73917fb Need also to remove unused bits 2017-03-22 00:14:37 +00:00
Martin Sumner
15af4942ae Remove busy log 
Accounts for 60% of logs
 2017-03-22 00:11:17 +00:00
martinsumner
97312e1356 Test to reflect vclock not binary 2017-03-21 18:28:28 +00:00
martinsumner
e59585d733 Merge remote-tracking branch 'refs/remotes/origin/mas-etsmem-i52' into mas-sstfiveblocks 2017-03-21 18:25:18 +00:00
martinsumner
eef2199335 Up level for yield to 2 2017-03-21 18:24:11 +00:00
martinsumner
f108871691 Vclock metadata change 
Test performance ocntinues to be worse since the vlock metadata change.
Reversing out juts in case.
 2017-03-21 18:15:56 +00:00
martinsumner
756b46bb4d Return to merge scan width of 16 
This was reduced before the use of binary blocks was committed
 2017-03-21 17:53:34 +00:00
martinsumner
1fdcdf3b37 Midblock size - lookup 
No real reason for the midblock to be smaller in lookup slots - so give
the blocks a more consistent size
 2017-03-21 17:47:08 +00:00