umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
354 commits 1 branch 0 tags 25 MiB
Commit graph

354 commits

This branch
This branch
All branches
Author SHA1 Message Date
martinsumner
6e5f5d2d44 Alter ordering 
don't try the cache hit before checking for presence, only look in the
cache if protecting a lookup from the persisted part
 2016-12-24 18:13:55 +00:00
martinsumner
480820e466 Add hash to missing key test 2016-12-24 18:03:34 +00:00
martinsumner
8526106312 Test for missing keys 2016-12-24 17:59:07 +00:00
martinsumner
0d0ab32653 Some end-to-end testing 2016-12-24 17:48:31 +00:00
martinsumner
7a11e8b490 Some basic testing 2016-12-24 16:34:36 +00:00
martinsumner
58d8e60994 Some basic code layout work 2016-12-24 15:12:24 +00:00
martinsumner
cb654b1325 Build the table summary 
The table summary will be a skiplist, and this and the slot binary will
be CRC checked
 2016-12-24 01:23:40 +00:00
martinsumner
4f838f6f88 Settled on sizes 
Also removed length check due to warning in Erlang guidance about
non-constant time nature of this command.  Intend to remove lengths from
elsewhere (especially when used simply for logging).
 2016-12-24 00:41:50 +00:00
martinsumner
b1a3b4ad13 Switch slot to gb_trees and size of 128 2016-12-24 00:02:06 +00:00
martinsumner
0cea470b70 Share final timing test 2016-12-23 23:30:15 +00:00
martinsumner
2d08816445 Confirm timings 2016-12-23 18:08:22 +00:00
martinsumner
4466210ac8 Revert back to slot size of 256 
Changing the slot size higher has a significant impact on the fetch
time, although it allows for more constant time on write.  i.e. doubling
the size means 5 x cost of read, if only a 10% increase at write time.
 2016-12-23 17:07:05 +00:00
martinsumner
b1429a7330 Experiment with slot width of 512 2016-12-23 16:49:16 +00:00
martinsumner
60bddbc874 More timing - and changes slot width 2016-12-23 13:17:59 +00:00
martinsumner
b37f3acb1e Extra timings 2016-12-23 12:44:44 +00:00
martinsumner
90e587dcee Initial functions and unit tests 
Try to replace SFT files with one that more natively supports features
already in use (e.g. skiplist, tinybloom and magic_hash)
 2016-12-23 12:30:58 +00:00
martinsumner
05ddcadbf9 Merge pull request #13 from martinsumner/mas-staggerhashtreewrite 
Stopped unnecessary seek for last_key
 2016-12-22 21:34:06 +00:00
martinsumner
0ddaaf9ac3 Stopped unnecessary seek for last_key 
When rolling we already know the last_key - no need to seek for it on
startup.

The time it takes for this seek needs to be considered with regards to
startup time.  Can we do without knowing lastkey?
 2016-12-22 19:51:39 +00:00
martinsumner
44cf5788ab Merge pull request #12 from martinsumner/mas-puttiming 
Timing Points
 2016-12-22 18:07:28 +00:00
martinsumner
a131b99082 Randomising logging of PUT timings 2016-12-22 17:33:14 +00:00
martinsumner
353fb08e21 Randomise logging of GET/HEAD samples 2016-12-22 17:28:41 +00:00
martinsumner
ee534081c3 Reduce log levels 
Remove some log noise to debug level
 2016-12-22 17:15:42 +00:00
martinsumner
151dd3ab70 Sample only for HEAD/GET response times 
Report regularly but only on a sample
 2016-12-22 16:47:36 +00:00
Martin Sumner
676e8fa494 Add Get Timing 2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909 put-timing default 
Remove need for individual actors to know the defaults for put_timing
tuple
 2016-12-22 14:41:43 +00:00
Martin Sumner
e9e0a7b323 Set higher logpoint 
Expectation is for many HEAD requests - so only log every 100K
 2016-12-22 14:36:57 +00:00
martinsumner
df350e1e6f Add unit test for head timing 2016-12-22 14:09:17 +00:00
martinsumner
130fb36ddd Add head timings 
Include log breaking down timings of HEAD requests by result and level
 2016-12-22 14:03:31 +00:00
martinsumner
ea20fc07f4 Maybe not 2016-12-21 21:56:33 +00:00
martinsumner
39d634c95b And again 2016-12-21 21:49:08 +00:00
martinsumner
3de146043b Its not like there's more than two hard things 
D'oh
 2016-12-21 21:41:54 +00:00
martinsumner
b2835aeaec Improve fetching efficiency 
Experiment to see if parsing all keys in block can be avoided - and if
so does this make the range scan more efficient.

Unproven change.
 2016-12-21 18:28:14 +00:00
martinsumner
be775127e8 Improve logging of merge activity and slow GETs 
Look into speculation that collisions between fetch_rnage and fetch may
be an issue
 2016-12-21 12:45:27 +00:00
martinsumner
f3e16dcd10 Add long-running logs 2016-12-21 01:56:12 +00:00
martinsumner
c193962c92 Sort out different timestamps 2016-12-20 23:16:52 +00:00
martinsumner
060ce2e263 Add put timing points 2016-12-20 23:11:50 +00:00
martinsumner
299e8e6de3 Initial phash test 
phash does not appear to be a potential causer of delay
 2016-12-20 20:55:56 +00:00
martinsumner
2a8e1afe41 Merge pull request #11 from martinsumner/mas-journalcorruptfail 
Resolve failing recovery test
 2016-12-16 23:58:37 +00:00
martinsumner
9e28287231 Resolve failing recovery test 
Now passing consistently with a number of different corruptions catered
for (including corruption of the Tag in the Inker Key)
 2016-12-16 23:18:55 +00:00
martinsumner
1684f0a913 Merge pull request #10 from martinsumner/mas_riakmetadata 
Full Riak Metadata
 2016-12-15 11:59:14 +00:00
martinsumner
4798dc1148 Write block theory 
With riak metadata the sft files are about 2MB - so a group write coun
of 64 is trying to write in 500 KB chunks.  Is this too big?
 2016-12-14 22:03:38 +00:00
Martin Sumner
b92b511166 Revert "Experiment" 
This reverts commit fe907eb479.
 2016-12-14 18:49:47 +00:00
martinsumner
fe907eb479 Experiment 
Perhaps the results ar epaging related - change settings to hold less in
memory and see
 2016-12-14 11:38:42 +00:00
martinsumner
f4e2e274e0 Reintroduce riak metadata extraction 
The full riak metadata had been stripped from the Ledger update for
performance reasons.  However, the full metadata is required in order to
save a GET before a PUT.  Therefore we want to do isolated testing on
this change to establish the relative cost value in that cost saving.
 2016-12-14 10:27:11 +00:00
martinsumner
bb1221a918 Merge pull request #9 from martinsumner/mas-addpclbloom 
Mas addpclbloom
 2016-12-14 10:11:18 +00:00
Martin Sumner
5efed94a1e Try slightly larger cache 2016-12-13 22:29:55 +00:00
martinsumner
b2c36ae541 Merge remote-tracking branch 'refs/remotes/origin/master' into mas-addpclbloom 2016-12-13 21:04:15 +00:00
martinsumner
baf4ca252f Revert "Experiment with temporary us eof ETS table" 
This reverts commit 2a106d0dc5.
 2016-12-13 20:24:29 +00:00
martinsumner
2a106d0dc5 Experiment with temporary us eof ETS table 
Rather than expensive lists:ukeymerge, try use a temporary ETS table.
 2016-12-13 19:38:14 +00:00
martinsumner
bc5190a9bd Merge pull request #8 from martinsumner/mas-cdb-hashtree-refactor 
Mas cdb hashtree refactor
 2016-12-13 18:48:37 +00:00