Commit graph

1115 commits

Author SHA1 Message Date
Martin Sumner
aedeb0c934 Add support for with_lookup head_only
head_only mode cna be run with_lookup - but there is no L0 index created in this case.

So the L0 index wasn't returning a potition list and the L0 cache wasn't being checked.

Code now checks every  position in the L0 cache, when  a lookup is attempted in head_only mode.
2018-06-23 15:15:49 +01:00
Martin Sumner
990e857ebe Add to log 2018-06-23 13:25:10 +01:00
Martin Sumner
ac14bbdf41 Add log to penciller 2018-06-23 13:18:32 +01:00
Martin Sumner
0dda129d3e Fix bad specs
There were some bad specs in '|' OR'd specs.  These were being falsely ignored in dialyzer until https://github.com/erlang/otp/pull/1722.

Running on OTP21 exposed these incomplete specs.
2018-06-21 15:46:42 +01:00
Martin Sumner
8a61e5af78 Add extra checks 2018-06-08 14:12:07 +01:00
Martin Sumner
022997b2e6 Coverage issue
The scan_table situation where the query needs to be start inclusive, was consistently getting coverage.  It was less likely to get coverage with smaller cache sizes.

It is not clear why this wasn't being triggered before.  Perhaps because of the erroneous jitter setting?

Multiple cache sizes now tested to try and make sure the test is always  in-line with expectations.
2018-06-08 12:34:10 +01:00
Martin Sumner
589d947c01 Jitter proportionate to size
Not not default size
2018-06-07 18:08:54 +01:00
Martin Sumner
7c1f34c94f Format
Reverse mysterious tab displacement
2018-06-07 14:55:05 +01:00
Martin Sumner
319c6b4ca7 Undefined typo
Interetsingly setting  max_pencillercachesize to a non-integer merely had the impact of making the penciller cache size infinite.

So a guard added to make sure it is an integer going forward.
2018-06-07 14:53:34 +01:00
Martin Sumner
11d72013d2 Add open_options type
Make it easier to discover all the options and the defaults from reaidng the code.

Note default sync option is now none.  This doesn't mean this is recommended
2018-06-07 12:40:12 +01:00
Martin Sumner
fc998d2971 GC Inker Manifest 2018-06-05 16:57:54 +01:00
Martin Sumner
93f16a6297 Unit test - GC
End up with right number of manifest files
2018-06-05 16:48:56 +01:00
Martin Sumner
a72d1cc941 Validate current manifest was written
Then delete a previous maniafest (from 5 manifests ago, if it is still there).
2018-06-05 15:36:39 +01:00
Martin Sumner
a14941a122 Fix unexported types
file:location not exported?
2018-06-04 10:57:37 +01:00
Martin Sumner
9f9da77398 Allow add_kv to output segment
Make it easier to spot clashes with dirty segments in kv_index_tictactree
2018-05-24 20:03:53 +01:00
Martin Sumner
ca2b857e36 Add replace segment function
This will be useful in kv_index_tictactree to do partial rebuilds after fetch_clock requests
2018-05-24 16:12:12 +01:00
Martin Sumner
274b1f5c6d Export types from leveled_tictac 2018-05-19 11:09:03 +01:00
Martin Sumner
afa9ddecd2 Merge branch 'mas-i31-cdccrckey' of https://github.com/martinsumner/leveled into mas-i31-cdccrckey 2018-05-18 15:48:39 +01:00
Martin Sumner
b8a642fd35 Spec update 2018-05-18 15:48:36 +01:00
Martin Sumner
8011a9cde1 Unit test should not pass with bad key
Unit test needed to capture a bad key scenario, which did not get returned false.  However, bad keys are now CRC checked and so willalways return false - and so the case clause for bad key can be dropped.
2018-05-18 15:23:43 +01:00
Martin Sumner
039b135f5f Ease timeout pressure in unit test 2018-05-18 14:36:47 +01:00
Martin Sumner
5e95e6a6b3 correct incorrect return value
required by leveled_codec:compact_inkerkvc/2
2018-05-18 13:44:51 +01:00
Martin Sumner
d468dd32a8 More consistency in extract value functions
Try to make this a little bit cleaner and more obvious.  Make sure the value passed is always a value and not a crc/value.
2018-05-18 13:29:19 +01:00
Martin Sumner
7cc5512a0a safe_read
There are now two cases of safe_read_next_key/2

the first case using the original function name will return just the Key when positive (not the Key and KeyBinary).  If the Key and KeyBinary is required (e.g. for CRC checking purposes), then use safe_read_next_keyint/2.
2018-05-18 11:49:19 +01:00
Martin Sumner
d56ff6efbc Make CRC check cover key and value
So that a corrupted key is detected through a CRC check without crashing leveled (e.g. because it binary_to_term/1 failes somewhere or it is a tuple of the wrong length).

Previously the CRC covered the value only.

Note if you only wish to extract the value, the key cannot be independently validated.  Perhaps a process extrating key only could hit issues.
2018-05-18 11:17:04 +01:00
Martin Sumner
989f23bca6 Add cache population for non-yielding range fetches
More likely to require caching at lower levels.
2018-05-17 16:38:52 +01:00
Martin Sumner
779ccd9c2a Add use of block index when not cached (for fetch range) 2018-05-17 14:56:15 +01:00
Martin Sumner
18aabb49ba Segment filter and multiple keys in slot
An issue was spotted.  If we use a segment filter in a query, and there are multiple matches within a given slot - only the first match is returned.

Tests didn't detect this.  Now they do, and the issue is resolved.
2018-05-16 17:24:23 +01:00
Martin Sumner
cbf6e26fc8 Revert "Accumulate keys in check_blocks"
This reverts commit 5414e18047.
2018-05-16 15:11:18 +01:00
Martin Sumner
5414e18047 Accumulate keys in check_blocks
Previously couldn't accumulate keys using check-blocks - so if a key was found in the first position, and there were other positions to check for other keys, those other positions wouldn't be checked.
2018-05-16 11:38:26 +01:00
Martin Sumner
2cc088fc64 More debug 2018-05-16 11:24:57 +01:00
Martin Sumner
a7cda7213f Debug logs 2018-05-16 11:06:51 +01:00
Martin Sumner
8b4adcccaf Debug logs 2018-05-16 10:51:59 +01:00
Martin Sumner
06fd9856b5 Add debug log 2018-05-16 10:34:47 +01:00
Martin Sumner
3d1c085af3 Revert log additions 2018-05-15 10:41:56 +01:00
Martin Sumner
15a5db5055 Sort tuned seg list 2018-05-15 10:36:07 +01:00
Martin Sumner
6160db3377 Further log 2018-05-15 10:30:09 +01:00
Martin Sumner
1e40648c83 Add initial log 2018-05-15 10:11:37 +01:00
Martin Sumner
eb5950a179 Merge branch 'master' into mas-i134-compileflag 2018-05-04 17:14:21 +01:00
Martin Sumner
96cee2a64c Add specific compile flag to avoid deprecation warning
Will see us through OTP 20 and 21.
2018-05-04 17:14:12 +01:00
Martin Sumner
6a20b2ce66 Use leveled_codec types
... and exporting them.

Previously types wer enot exported, and it appears dialyzer treated tham as any() when they were unexported types ??!!??
2018-05-04 15:24:08 +01:00
Martin Sumner
2063cacd8f More spec/doc work in leveled_codec
Note that at some stage KeyChanges got overloaded to mean {KeyChanges, TTL}, and the spec now tries to make this a bit clearer
2018-05-04 11:19:37 +01:00
Martin Sumner
aa34ffda5b Crash not skip on corrupted key 2018-05-03 20:14:36 +01:00
Martin Sumner
2cd20fcb47 Missed generate_uuid reference 2018-05-03 18:26:02 +01:00
Martin Sumner
f88f511df3 leveled_codec spec/doc
Try and make this code a little bit more organised andd easier to follow
2018-05-03 17:18:13 +01:00
Martin Sumner
416aa7c0d3 Reference startup writeup 2018-05-02 16:03:03 +01:00
Martin Sumner
ef9ac57c5e Add information on startup options 2018-05-02 15:48:16 +01:00
Martin Sumner
a9db4a0082 Spec and comment Inker 2018-05-02 10:57:27 +01:00
Martin Sumner
dd7b753688 Add spec and comments to penciller 2018-05-01 21:28:40 +01:00
Martin Sumner
ee5dc43cfd Add doc/spec to bookie
Improve the comments and dialyzer hints within the bookie.
2018-05-01 12:03:24 +01:00