martinsumner
e01b310d20
Handle production of empty file
2016-12-29 05:09:47 +00:00
martinsumner
55386622f7
Fixed issues
...
Two issues - when the key range falls in-between two marks in the
summary, we didn't pick up any mark. then when trimming both right and
left, the left trim was being discarded.
2016-12-29 04:37:49 +00:00
martinsumner
5b9e68df99
Add some crash protection for empty return from to_range
...
Not clear though why it would occur.
2016-12-29 03:04:10 +00:00
martinsumner
3f3b36597a
Add timer for SST creation
2016-12-29 02:55:28 +00:00
martinsumner
c3999110e2
Remove io:format from debugging
2016-12-29 02:47:21 +00:00
martinsumner
a665b8ea4f
Tidy-up unused variable
2016-12-29 02:41:02 +00:00
martinsumner
0c4d949c7f
State mixup in FSM
2016-12-29 02:40:09 +00:00
Martin Sumner
18f2b5660d
Fix to ensure directory structure created
2016-12-29 02:31:10 +00:00
martinsumner
dc28388c76
Removed SFT
...
Now moved over to SST on this branch
2016-12-29 02:07:14 +00:00
martinsumner
c664483f03
Add basic merge support
...
No generates KV list first, and then creates a new SST
2016-12-28 21:47:05 +00:00
martinsumner
3716de1c82
Revert back to sampling
...
timing logs should be based on a sample
2016-12-28 15:49:58 +00:00
martinsumner
cbad375373
Refactoring of skiplist ranges and support for sst ranges
...
the Skiplist range code was needlessly complicated. It may be faster
than the new code, but the complexity delta cannot be support for such a
small change.
This was incovered whilst troubleshooting the initial kv range test.
2016-12-28 15:48:04 +00:00
martinsumner
6e5f5d2d44
Alter ordering
...
don't try the cache hit before checking for presence, only look in the
cache if protecting a lookup from the persisted part
2016-12-24 18:13:55 +00:00
martinsumner
480820e466
Add hash to missing key test
2016-12-24 18:03:34 +00:00
martinsumner
8526106312
Test for missing keys
2016-12-24 17:59:07 +00:00
martinsumner
0d0ab32653
Some end-to-end testing
2016-12-24 17:48:31 +00:00
martinsumner
7a11e8b490
Some basic testing
2016-12-24 16:34:36 +00:00
martinsumner
58d8e60994
Some basic code layout work
2016-12-24 15:12:24 +00:00
martinsumner
cb654b1325
Build the table summary
...
The table summary will be a skiplist, and this and the slot binary will
be CRC checked
2016-12-24 01:23:40 +00:00
martinsumner
4f838f6f88
Settled on sizes
...
Also removed length check due to warning in Erlang guidance about
non-constant time nature of this command. Intend to remove lengths from
elsewhere (especially when used simply for logging).
2016-12-24 00:41:50 +00:00
martinsumner
b1a3b4ad13
Switch slot to gb_trees and size of 128
2016-12-24 00:02:06 +00:00
martinsumner
0cea470b70
Share final timing test
2016-12-23 23:30:15 +00:00
martinsumner
2d08816445
Confirm timings
2016-12-23 18:08:22 +00:00
martinsumner
4466210ac8
Revert back to slot size of 256
...
Changing the slot size higher has a significant impact on the fetch
time, although it allows for more constant time on write. i.e. doubling
the size means 5 x cost of read, if only a 10% increase at write time.
2016-12-23 17:07:05 +00:00
martinsumner
b1429a7330
Experiment with slot width of 512
2016-12-23 16:49:16 +00:00
martinsumner
60bddbc874
More timing - and changes slot width
2016-12-23 13:17:59 +00:00
martinsumner
b37f3acb1e
Extra timings
2016-12-23 12:44:44 +00:00
martinsumner
90e587dcee
Initial functions and unit tests
...
Try to replace SFT files with one that more natively supports features
already in use (e.g. skiplist, tinybloom and magic_hash)
2016-12-23 12:30:58 +00:00
martinsumner
05ddcadbf9
Merge pull request #13 from martinsumner/mas-staggerhashtreewrite
...
Stopped unnecessary seek for last_key
2016-12-22 21:34:06 +00:00
martinsumner
0ddaaf9ac3
Stopped unnecessary seek for last_key
...
When rolling we already know the last_key - no need to seek for it on
startup.
The time it takes for this seek needs to be considered with regards to
startup time. Can we do without knowing lastkey?
2016-12-22 19:51:39 +00:00
martinsumner
44cf5788ab
Merge pull request #12 from martinsumner/mas-puttiming
...
Timing Points
2016-12-22 18:07:28 +00:00
martinsumner
a131b99082
Randomising logging of PUT timings
2016-12-22 17:33:14 +00:00
martinsumner
353fb08e21
Randomise logging of GET/HEAD samples
2016-12-22 17:28:41 +00:00
martinsumner
ee534081c3
Reduce log levels
...
Remove some log noise to debug level
2016-12-22 17:15:42 +00:00
martinsumner
151dd3ab70
Sample only for HEAD/GET response times
...
Report regularly but only on a sample
2016-12-22 16:47:36 +00:00
Martin Sumner
676e8fa494
Add Get Timing
2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909
put-timing default
...
Remove need for individual actors to know the defaults for put_timing
tuple
2016-12-22 14:41:43 +00:00
Martin Sumner
e9e0a7b323
Set higher logpoint
...
Expectation is for many HEAD requests - so only log every 100K
2016-12-22 14:36:57 +00:00
martinsumner
df350e1e6f
Add unit test for head timing
2016-12-22 14:09:17 +00:00
martinsumner
130fb36ddd
Add head timings
...
Include log breaking down timings of HEAD requests by result and level
2016-12-22 14:03:31 +00:00
martinsumner
ea20fc07f4
Maybe not
2016-12-21 21:56:33 +00:00
martinsumner
39d634c95b
And again
2016-12-21 21:49:08 +00:00
martinsumner
3de146043b
Its not like there's more than two hard things
...
D'oh
2016-12-21 21:41:54 +00:00
martinsumner
b2835aeaec
Improve fetching efficiency
...
Experiment to see if parsing all keys in block can be avoided - and if
so does this make the range scan more efficient.
Unproven change.
2016-12-21 18:28:14 +00:00
martinsumner
be775127e8
Improve logging of merge activity and slow GETs
...
Look into speculation that collisions between fetch_rnage and fetch may
be an issue
2016-12-21 12:45:27 +00:00
martinsumner
f3e16dcd10
Add long-running logs
2016-12-21 01:56:12 +00:00
martinsumner
c193962c92
Sort out different timestamps
2016-12-20 23:16:52 +00:00
martinsumner
060ce2e263
Add put timing points
2016-12-20 23:11:50 +00:00
martinsumner
299e8e6de3
Initial phash test
...
phash does not appear to be a potential causer of delay
2016-12-20 20:55:56 +00:00
martinsumner
2a8e1afe41
Merge pull request #11 from martinsumner/mas-journalcorruptfail
...
Resolve failing recovery test
2016-12-16 23:58:37 +00:00
