Martin Sumner
cb8fe0c2d3
Update VOLUME.md
2018-02-26 15:54:49 +00:00
Martin Sumner
2a7783b01f
Merge pull request #121 from martinsumner/mas-nhshw-document
...
Add Phase 6 Volume overview
2018-02-26 15:41:52 +00:00
Martin Sumner
002f060808
Add Phase 6 Volume overview
...
Initial test from phase 6 - real hardware
2018-02-26 15:36:01 +00:00
Martin Sumner
80424ac108
Merge pull request #119 from DeadZen/patch-1
...
Update ANTI_ENTROPY.md
2018-02-19 20:22:01 +00:00
Pedram Nimreezi
aad21c3a34
Update ANTI_ENTROPY.md
...
Small typo
2018-02-19 14:21:27 -05:00
Martin Sumner
4d3151752f
Merge pull request #118 from martinsumner/mas-i117-factor4scale
...
Mas i117 factor4scale
2018-02-11 11:34:31 +00:00
Martin Sumner
834704a3ff
Merge branch 'mas-i117-factor4scale' of https://github.com/martinsumner/leveled into mas-i117-factor4scale
2018-02-10 08:10:32 +00:00
Martin Sumner
f748fc8611
Narrower still
...
Make the LSM tree more bottle shaped.
Experiment to judge performance impact
2018-02-10 08:10:24 +00:00
Martin Sumner
5673d8b558
Expand test to ensure coverage catch
2018-02-10 08:09:33 +00:00
Martin Sumner
8113aebdcf
Add timings for Level 3
...
Level 3 readings now relatively common - so time the separately
2018-02-09 08:59:21 +00:00
Martin Sumner
c7cea04aba
Correct maximum length
2018-02-08 15:31:35 +00:00
Martin Sumner
7e4c3db915
Alternate scale factor
...
Also had failed unit test - there was an issue with bit-flipping the position not being safely caught
2018-02-08 10:29:27 +00:00
Martin Sumner
7e29480bf2
Merge pull request #114 from martinsumner/mas-bloomtest
...
Mas bloomtest
2017-12-06 09:08:27 +00:00
Martin Sumner
bb39498ec5
Missed log line added back
...
.. and covered in test
2017-12-04 15:26:01 +00:00
Martin Sumner
fea87c21ec
Notes testing on non-compressed L1
2017-12-04 13:08:24 +00:00
Martin Sumner
f8ceedc9bb
Compress L0 only
...
Doing at L1 has a negative impact as tests draw on. Also improve head time tracking
2017-12-04 10:49:42 +00:00
Martin Sumner
1f5d5033a4
Revert "Revert "Disable compression L0 and L1""
...
This reverts commit 958d3f5e14
2017-12-04 09:30:27 +00:00
Martin Sumner
958d3f5e14
Revert "Disable compression L0 and L1"
...
This reverts commit b10c0cf895
2017-12-04 09:29:44 +00:00
Martin Sumner
b10c0cf895
Disable compression L0 and L1
2017-12-02 09:19:17 +00:00
Martin Sumner
6e589942b6
Cover bit flips in the slot header
2017-12-01 16:20:48 +00:00
Martin Sumner
9a6cfa6522
Typos
2017-12-01 14:24:00 +00:00
Martin Sumner
5bac389d0c
Switch to CRC check at Block Level
...
Previously done at Slot Level - but Blocks were still read from disk after the Slot CRC had been checked.
This seems safer. It requires an extra CRC check for every fetch. However, CRC chekcing smaller binaries during the buld process appears to be beneficial to performance.
Hoped this will be an enabler to turning off compression at Levels 0 and 1 to improve performance (wihtout having a compensating issues with reduced CRC performance)
2017-12-01 14:15:13 +00:00
Martin Sumner
7a99d060a3
Resolve OTP 19 compatibility
...
Dialyzer issues otherwise
2017-11-30 19:10:26 +00:00
Martin Sumner
9d9ad17d36
Typo
2017-11-30 16:29:10 +00:00
Martin Sumner
3b42bc28d1
Add build timing info to merge_list log
...
Help to determine what the expensive part of the operation is
2017-11-30 16:15:38 +00:00
Martin Sumner
e59f051c54
Further details on merge time
2017-11-30 13:59:32 +00:00
Martin Sumner
54e2e2344c
Further review
...
Looks like we have idnetified the real CPU burner. But how to address?
2017-11-30 13:41:12 +00:00
Martin Sumner
7581c4eba3
Update HeadTime charts
2017-11-29 14:46:24 +00:00
Martin Sumner
ad36578434
Add updated HeadTime charts
2017-11-29 14:31:07 +00:00
Martin Sumner
41c308c5fd
As used in lookup - will always be hash
2017-11-28 22:13:18 +00:00
Martin Sumner
676879a335
Start of write-up on latency
2017-11-28 19:03:06 +00:00
Martin Sumner
cb10a00845
Make test reliable
...
Might corrupt the wrong journal otherwise
2017-11-28 14:57:17 +00:00
Martin Sumner
eb90541a85
Add a small cache to SST file
...
so that a HEAD which folllows a HEAD (e.g. when a GET follows a HEAD) has a chance of avoiding the binary_to_term CPU load
2017-11-28 14:56:40 +00:00
Martin Sumner
7c4e1a5ad9
Typo
2017-11-28 11:49:50 +00:00
Martin Sumner
a655baf881
Resolve R16 dialyzer issue
2017-11-28 11:49:29 +00:00
Martin Sumner
5342e3a94f
Improve testing of bloom feature
...
In particular will blooms re-appear following startup
2017-11-28 11:43:46 +00:00
Martin Sumner
c2f19d8825
Switch to using bloom at penciller
...
Previouslythe tinybloom was used within the SST file as an extra check to remove false fetches.
However the SST already has a low FPR check in the slot_index. If the newebloom was used (which is no longer per slot, but per sst), this can be shared with the penciller and then the penciller could use it and avoid the message pass.
the message pass may be blocked by a 2i query or a slot fetch request for a merge. So this should make performance within the Penciller snappier.
This is as a result of taking sst_timings within a volume test - where there was an average of + 100microsecs for each level that was dropped down. Given the bloom/slot checks were < 20 microsecs - there seems to be some further delay.
The bloom is a binary of > 64 bytes - so passing it around should not require a copy.
2017-11-28 01:19:30 +00:00
Martin Sumner
467ad50cd1
Settle on hash approach
2017-11-27 15:29:01 +00:00
Martin Sumner
6b222ea8f9
Revert "Revert "Switch to 32 slots""
...
This reverts commit 99dad0c86f
2017-11-27 15:17:40 +00:00
Martin Sumner
99dad0c86f
Revert "Switch to 32 slots"
...
This reverts commit 7af6a3ba00
2017-11-27 15:16:18 +00:00
Martin Sumner
7af6a3ba00
Switch to 32 slots
2017-11-27 15:16:10 +00:00
Martin Sumner
5e6541dddb
Revert "Revert "Half size of each slot's bloom""
...
This reverts commit d37c5eab3f
2017-11-27 14:51:25 +00:00
Martin Sumner
34dc63a8f8
Change measurement
2017-11-27 14:51:03 +00:00
Martin Sumner
d37c5eab3f
Revert "Half size of each slot's bloom"
...
This reverts commit d83eea7c60
2017-11-27 14:49:43 +00:00
Martin Sumner
d83eea7c60
Half size of each slot's bloom
2017-11-27 14:48:51 +00:00
Martin Sumner
c65dfa31d8
Add alternative bloom
...
Bloom filter that can take largere keys but is still efficient to build. Allows bloom filter to be checked without first detemrining the slot. Also, as represents the whole SST - it could be sent to the penciller to remove the need for a message pass.
The bloom is smaller and has a worse fpr than leveled_tinybloom. Failing the bloom check isn't so bad - due to the slot index check being relatively fast and having a very low fpr.
2017-11-24 11:38:58 +00:00
Martin Sumner
0e51627539
Merge pull request #113 from martinsumner/mas-i111-ssttimings
...
Mas i111 ssttimings
2017-11-22 07:44:30 +00:00
Martin Sumner
f436cfd03e
Add consistent timing points
...
Now all timing points should be made in a consistent fashion
2017-11-21 23:13:24 +00:00
Martin Sumner
3ef550d9f8
Refactor timing point management
...
For Penciller and timing head requests.
2017-11-21 19:58:36 +00:00
Martin Sumner
58946a7f98
Amend SST Timing Capture
...
Use sampling mechansm from CDB timing capture. Do it less though - as far more SST fetches in comparison to CDB fetches.
2017-11-21 17:00:23 +00:00
