umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
674 commits 1 branch 0 tags 25 MiB
Commit graph

674 commits

This branch
This branch
All branches
Author SHA1 Message Date
martinsumner
f838ffda2c Typo 2017-03-24 15:41:33 +00:00
martinsumner
6de9740bda Typo 2017-03-24 15:31:19 +00:00
martinsumner
5e1482d71b 2i query test results 
Write-up of 2i query test results
 2017-03-24 15:20:03 +00:00
Martin Sumner
46759b945b Merge pull request #53 from martinsumner/mas-cdbcrc-i31 
Switch to binary vclock
 2017-03-20 22:44:59 +00:00
Martin Sumner
a2c164e1dd Switch to binary vclock 2017-03-20 22:44:19 +00:00
Martin Sumner
43c5f2fb07 Merge pull request #51 from martinsumner/mas-cdbcrc-i31 
Mas cdbcrc i31
 2017-03-20 22:18:40 +00:00
Martin Sumner
eec9d509f9 Add back hash performance tests 
Need to consider if magic hash is an issue
 2017-03-20 20:28:47 +00:00
martinsumner
7154815a2b Keep vclock as binary 
No obvious, need at present for vlock to be a term within leveled
 2017-03-20 20:28:02 +00:00
Martin Sumner
994d9758b0 Object now 4 bytes larger 
due to addition of key changes length
 2017-03-20 16:21:49 +00:00
martinsumner
f3ffa920af Trying to standardise binary manipulation of value 
Looking into theory that use of term_to_binary is imperfect.  Also may
be better to compress values only when they are compacted?
 2017-03-20 15:43:54 +00:00
Martin Sumner
0cdc0eb558 Merge pull request #50 from martinsumner/mas-sstbinarytrim-i42 
Mas sstbinarytrim i42
 2017-03-20 10:28:55 +00:00
martinsumner
5c662aeca1 Additional unit test 
Need to test scenario where the key list the SST file created from is an
exact multiple of the slot size
 2017-03-19 23:42:24 +00:00
martinsumner
431c2cee40 Remove unnecessary line 
Brnach cannot be reached as firts key is always discovered when it is a
no_loolup
 2017-03-19 23:37:50 +00:00
martinsumner
f20aba9c8b Curtail trimmed slot crazyness 
There was complicated and confusing code that achieved nothing for
effiency when trimming slots.  the expensive part (binary_to_term) was
still needed on every block, and it was hard to get code coverage and
make sense of what it was really trying to achieve.

This is now much simpler - and may set us up for potential further
indexing help.
 2017-03-19 21:47:22 +00:00
martinsumner
c203e2ee06 Range queries - pass out as binaries 
Avoid converting to erlang temr wihtin the FSM and then requiring a copy
outside of the FSM - pass out as a binary
 2017-03-17 10:47:20 +00:00
martinsumner
f287895db0 Pass out slots as a binary 
If we convetr firts to a list, then the list has to be copied - passing
out as binaries means the bulk can be passed as references
 2017-03-17 10:43:34 +00:00
Martin Sumner
2e1a5e7537 Merge pull request #49 from martinsumner/mas-treelog 
Debug excessive log
 2017-03-16 20:03:41 +00:00
martinsumner
80b62cbff2 Debug excessive log 
Logs excessively during 2i tests.  Set to debug for now, until can think
further about this
 2017-03-16 20:03:18 +00:00
Martin Sumner
7c835eefc3 Merge pull request #48 from martinsumner/mas-snapshotlog 
Remaining ledger snapshots log -> debug
 2017-03-16 19:38:36 +00:00
martinsumner
8b2091cef7 Remaining ledger snapshots log -> debug 
This log under 2i load appears thousands of times per second.  Not
sustainable as an info log.  Will need to think about how to manage
this, but setting back to debug for now
 2017-03-16 19:37:39 +00:00
Martin Sumner
7f50540120 Merge pull request #47 from martinsumner/mas-pushmem-i46 
Mas pushmem i46
 2017-03-16 09:28:44 +00:00
martinsumner
5dbd7a2bc2 Check query out of range 
It doesn't work - so protecting against it in fetch_range is pointless,
will blow up in lookup_slots
 2017-03-16 08:43:18 +00:00
martinsumner
6199a2c352 RTrim 
RTrim only worked in special case of key matching, that would never
occur in real world range query.  RTrim should really check for key
passing.

Returning empty list should not be possible - unless the query is
outside of the range entirely (and such a query should never go to this
SST).
 2017-03-16 08:37:36 +00:00
martinsumner
dde37566b9 Add unit test for more than one slot 2017-03-15 16:40:43 +00:00
martinsumner
c6d17b998e Additional unit tests for SST range fetches 
Resolve some of coverage issues
 2017-03-15 11:27:46 +00:00
martinsumner
508da0be45 Additional unit tests 2017-03-14 22:47:48 +00:00
martinsumner
8a5ed1e198 Confirm skip on unknowns when compacting journal 2017-03-14 17:26:39 +00:00
martinsumner
878ec41ffa Merge remote-tracking branch 'refs/remotes/origin/mas-sstblockv2-i42' into mas-pushmem-i46 
# Conflicts:
#	src/leveled_penciller.erl
 2017-03-14 15:43:35 +00:00
martinsumner
4b60c0e35b Scan width semi-reverted 
No evidence from valume test that the scan width has made a posiitve
difference - so reverting, but not fully as slots may now be twice as
big, so sticking to half previous value
 2017-03-14 01:18:50 +00:00
martinsumner
19bc838d90 Fix bad exit with no FK 2017-03-14 00:52:07 +00:00
martinsumner
a1c49b668a Fix empty file again 
No special definition of empty required, as now an empty list when empty
 2017-03-14 00:17:09 +00:00
martinsumner
2b0ec1d9cc Don't double-loop on slots 
Previous version built a list of slots, then iterated over it to build a
list of binaries.  This converts the slot to a binary before building
the list
 2017-03-13 23:51:48 +00:00
martinsumner
4f0622d2ac Merge remote-tracking branch 'refs/remotes/origin/mas-sstblock-i42' into mas-sstblockv2-i42 2017-03-13 21:09:13 +00:00
martinsumner
c5bb150f97 Drop some logs 
Not found to be interesting so far
 2017-03-13 20:30:33 +00:00
martinsumner
54534e725f Experiment with smaller scan width 
When testing with large numbers of 2i terms (and hence more Riak
Metadata), there is a surge in slow response times when there are
multiple concurrent merge events.

This could be veyr short term CPU starvation because of the merge
process.  Perhaps it is delays waiting for the scan to complete -
smaller scanwidth may mena more interleaving and less latency?
 2017-03-13 19:53:12 +00:00
martinsumner
5311a157d5 Merge remote-tracking branch 'refs/remotes/origin/mas-sstblock-i42' into mas-sstblockv2-i42 2017-03-13 19:22:41 +00:00
martinsumner
c787e0cd78 Handle corrupted Ledger Key when applying recovery strategy 
Otherwise may blow up in journal_compaction_bustedjournal test
 2017-03-13 14:32:46 +00:00
martinsumner
f3e962c43a Add level to SST slow fetch log 2017-03-13 12:16:36 +00:00
martinsumner
62c3ba8b6f Passing ETS reference not tree 
ETS reference gets converted to tree by Penciller
 2017-03-13 11:54:46 +00:00
martinsumner
f2cd9b3f33 Consistency of empty slotlist references 
Need to return an empty slotlist in a consistent way
 2017-03-11 13:04:55 +00:00
martinsumner
1f8de798bd Fix empty slot issue 2017-03-11 12:41:30 +00:00
martinsumner
a07770a3df Unit tets of lookup over-size issue 
A mistake meant resetting to lookup on a skipped key would cause issues
if the skipped key ocurred under a no_lookup slot after the ?SLOT_SIZE
had been reached.  This caused the slot to switch to lookup, but beyond
the maximum size
 2017-03-11 00:03:55 +00:00
martinsumner
4e4f498f20 Correctly set no_lookup on skip_key 
Otherwise could change to lookup after the size limit has been reached
 2017-03-10 23:48:17 +00:00
martinsumner
1813317121 Correctly identify empty slotlist 2017-03-10 22:49:00 +00:00
martinsumner
b2f3d882a9 Draft of branch to condense range_only keys 2017-03-10 20:43:37 +00:00
martinsumner
730ab2ec48 tidy out io:format 2017-03-10 11:10:15 +00:00
martinsumner
601f43de3d Merge remote-tracking branch 'refs/remotes/origin/master' into mas-sstblock-i42 2017-03-10 10:24:51 +00:00
Martin Sumner
121f410b8f Merge pull request #45 from martinsumner/mas-shortenfilepath 
Mas shortenfilepath
 2017-03-10 08:59:41 +00:00
martinsumner
d7eee2f9c9 Remove rogue log 2017-03-09 22:24:11 +00:00
martinsumner
39a005a8d0 Try and be consistent in flilename format 
i.e. ./filename.sst
 2017-03-09 21:52:29 +00:00