umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1056 commits 1 branch 0 tags 25 MiB
Commit graph

1056 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
75467be642 Initial re-wording to reflect low-level design changes 
This might change back again.  It is easier to make the KeyStore and TreeCaches to manage more directly the rebuild process.  The only issue is whther this would "lock up" thevnode should the vnode ever wait on a response for example for a tree root (when the TreeCache is finishing the load)
 2018-03-05 11:08:17 +00:00
Martin Sumner
4bf6d3e73d Fiddle with naming in query API 
Was easier in the calling applictaion to switch between using and not using a list of the Query format was consistent between those two cases.
 2018-03-02 10:20:43 +00:00
Martin Sumner
c6b3fe78f4 Add 24 Hour Test results 2018-03-02 09:47:34 +00:00
Martin Sumner
6f9d7e8fc1 Merge branch 'master' into mas-i115-simplettcache 2018-03-02 09:35:34 +00:00
Martin Sumner
5312806592 Stop Iterator re-use 
The IMM iterator should not be reused, as it has already been filtered for a query.  so if reused for a different query incorrect and unexpected results may occur.

This reuse had been stopped by a previous commit, and this cleans up subsequently unused code.
 2018-03-02 08:16:34 +00:00
Martin Sumner
d21d18dd82 Remove rogue log 2018-03-01 23:37:05 +00:00
Martin Sumner
861aa5a7db Support multi-query fold 
Allow a single snapshot to run query over multiple ranges.   Used initially to fold over multiple buckets.
 2018-03-01 23:19:52 +00:00
Martin Sumner
cb8fe0c2d3
Update VOLUME.md 2018-02-26 15:54:49 +00:00
Martin Sumner
2a7783b01f
Merge pull request #121 from martinsumner/mas-nhshw-document 
Add Phase 6 Volume overview
 2018-02-26 15:41:52 +00:00
Martin Sumner
002f060808 Add Phase 6 Volume overview 
Initial test from phase 6 - real hardware
 2018-02-26 15:36:01 +00:00
Martin Sumner
80424ac108
Merge pull request #119 from DeadZen/patch-1 
Update ANTI_ENTROPY.md
 2018-02-19 20:22:01 +00:00
Martin Sumner
10f4c4c5e5
Merge pull request #120 from DeadZen/patch-2 
Update GENERAL_TICTACAAE_FOR_RIAK.md
 2018-02-19 20:21:49 +00:00
Pedram Nimreezi
6e525de76b
Update GENERAL_TICTACAAE_FOR_RIAK.md 
Just a small typo
 2018-02-19 14:40:25 -05:00
Pedram Nimreezi
aad21c3a34
Update ANTI_ENTROPY.md 
Small typo
 2018-02-19 14:21:27 -05:00
Martin Sumner
a91726f990 More details on rebuilds/rehashes 2018-02-17 11:45:19 +00:00
Martin Sumner
fa532fbd27 Tidy set_status 2018-02-16 20:56:12 +00:00
Martin Sumner
090e414b23 Coverage issues 
Not making proxy object so get_size not required.

Extend tests to improve coverage
 2018-02-16 20:27:49 +00:00
Martin Sumner
70dfb77088 Optional lookup in head_only mode 
Allow decision to be made on startup whether ObjectSpecs can be looked up directly when running in head_only mode.
 2018-02-16 17:06:30 +00:00
Martin Sumner
7751be1431 Initial write-up 
... of what we're trying to achieve
 2018-02-16 15:09:03 +00:00
Martin Sumner
910ccb6072 Add lookup support in head_only mode 
Originally had disabled the ability to lookup individual values when running in head_only mode.  This is a saving of about 11% at PUT time (about 3 microseconds  per PUT) on a macbook.

Not sure this saving is sufficient enought to justify the extra work if this is used as an AAE Keystore with Bitcask and LWW (when we need to lookup the current value before adjusting).

So reverted to re-adding support for HEAD requests with these keys.
 2018-02-16 14:16:28 +00:00
Martin Sumner
2b6281b2b5 Initial head_only features 
Initial commit to add head_only mode to leveled.  This allows leveled to receive batches of object changes, but where those objects exist only in the Penciller's Ledger (once they have been persisted within the Ledger).

The aim is to reduce significantly the cost of compaction.  Also, the objects ar enot directly accessible (they can only be accessed through folds).  Again this makes life easier during merging in the LSM trees (as no bloom filters have to be created).
 2018-02-15 16:14:46 +00:00
Martin Sumner
4d3151752f
Merge pull request #118 from martinsumner/mas-i117-factor4scale 
Mas i117 factor4scale
 2018-02-11 11:34:31 +00:00
Martin Sumner
834704a3ff Merge branch 'mas-i117-factor4scale' of https://github.com/martinsumner/leveled into mas-i117-factor4scale 2018-02-10 08:10:32 +00:00
Martin Sumner
f748fc8611 Narrower still 
Make the LSM tree more bottle shaped.

Experiment to judge performance impact
 2018-02-10 08:10:24 +00:00
Martin Sumner
5673d8b558 Expand test to ensure coverage catch 2018-02-10 08:09:33 +00:00
Martin Sumner
8113aebdcf Add timings for Level 3 
Level 3 readings now relatively common - so time the separately
 2018-02-09 08:59:21 +00:00
Martin Sumner
c7cea04aba Correct maximum length 2018-02-08 15:31:35 +00:00
Martin Sumner
7e4c3db915 Alternate scale factor 
Also had failed unit test - there was an issue with bit-flipping the position not being safely caught
 2018-02-08 10:29:27 +00:00
Martin Sumner
7e29480bf2
Merge pull request #114 from martinsumner/mas-bloomtest 
Mas bloomtest
 2017-12-06 09:08:27 +00:00
Martin Sumner
bb39498ec5 Missed log line added back 
.. and covered in test
 2017-12-04 15:26:01 +00:00
Martin Sumner
fea87c21ec Notes testing on non-compressed L1 2017-12-04 13:08:24 +00:00
Martin Sumner
f8ceedc9bb Compress L0 only 
Doing at L1 has a negative impact as tests draw on.  Also improve head time tracking
 2017-12-04 10:49:42 +00:00
Martin Sumner
1f5d5033a4 Revert "Revert "Disable compression L0 and L1"" 
This reverts commit 958d3f5e14.
 2017-12-04 09:30:27 +00:00
Martin Sumner
958d3f5e14 Revert "Disable compression L0 and L1" 
This reverts commit b10c0cf895.
 2017-12-04 09:29:44 +00:00
Martin Sumner
b10c0cf895 Disable compression L0 and L1 2017-12-02 09:19:17 +00:00
Martin Sumner
6e589942b6 Cover bit flips in the slot header 2017-12-01 16:20:48 +00:00
Martin Sumner
9a6cfa6522 Typos 2017-12-01 14:24:00 +00:00
Martin Sumner
5bac389d0c Switch to CRC check at Block Level 
Previously done at Slot Level - but Blocks were still read from disk after the Slot CRC had been checked.

This seems safer.  It requires an extra CRC check for every fetch.  However, CRC chekcing smaller binaries during the buld process appears to be beneficial to performance.

Hoped this will be an enabler to turning off compression at Levels 0 and 1 to improve performance (wihtout having a compensating issues with reduced CRC performance)
 2017-12-01 14:15:13 +00:00
Martin Sumner
7a99d060a3 Resolve OTP 19 compatibility 
Dialyzer issues otherwise
 2017-11-30 19:10:26 +00:00
Martin Sumner
9d9ad17d36 Typo 2017-11-30 16:29:10 +00:00
Martin Sumner
3b42bc28d1 Add build timing info to merge_list log 
Help to determine what the expensive part of the operation is
 2017-11-30 16:15:38 +00:00
Martin Sumner
e59f051c54 Further details on merge time 2017-11-30 13:59:32 +00:00
Martin Sumner
54e2e2344c Further review 
Looks like we have idnetified the real CPU burner.  But how to address?
 2017-11-30 13:41:12 +00:00
Martin Sumner
7581c4eba3 Update HeadTime charts 2017-11-29 14:46:24 +00:00
Martin Sumner
ad36578434 Add updated HeadTime charts 2017-11-29 14:31:07 +00:00
Martin Sumner
41c308c5fd As used in lookup - will always be hash 2017-11-28 22:13:18 +00:00
Martin Sumner
676879a335 Start of write-up on latency 2017-11-28 19:03:06 +00:00
Martin Sumner
cb10a00845 Make test reliable 
Might corrupt the wrong journal otherwise
 2017-11-28 14:57:17 +00:00
Martin Sumner
eb90541a85 Add a small cache to SST file 
so that a HEAD which folllows a HEAD (e.g. when a GET follows a HEAD) has a chance of avoiding the binary_to_term CPU load
 2017-11-28 14:56:40 +00:00
Martin Sumner
7c4e1a5ad9 Typo 2017-11-28 11:49:50 +00:00