Commit graph

1529 commits

Author SHA1 Message Date
Martin Sumner
881b93229b Isolate better changes needed to support changes to metadata extraction
More obvious how to extend the code as it is all in one module.

Also add a new field to the standard object metadata tuple that may hold in the future other object metadata base don user-defined functions.
2018-12-06 15:31:11 +00:00
Martin Sumner
bfddb53e31
Merge pull request #221 from martinsumner/mas-i219-tuplebuckets
Deprecate use of Bucket as a constraint in indexfold
2018-12-05 20:47:40 +00:00
Martin Sumner
835955adb9 Deprecate use of Bucket as a constraint in indexfold
Must also give a StartKey.  Otherwise there may be conflict when tuplebuckets are used
2018-12-05 20:46:17 +00:00
Martin Sumner
f415d22ef0
Merge pull request #220 from martinsumner/mas-i157-indexttl
Add test of index expiry
2018-12-05 15:42:43 +00:00
Martin Sumner
510994233e Add check that index disappears
Check I0 count goes down when that index is removed
2018-12-05 15:42:21 +00:00
Martin Sumner
cf1fcaeef2 Add test of index expiry
To show how this works, and prove that it does work thta way.

Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
Martin Sumner
150fcf15d6
Merge pull request #217 from martinsumner/mas-i216-nomod
Remove incorrect mod entry
2018-12-05 09:31:57 +00:00
Martin Sumner
d438c29200
Merge pull request #218 from martinsumner/mas-uw-configurableloglevel
Mas uw configurableloglevel
2018-12-05 09:31:31 +00:00
Martin Sumner
9735661d98 Typos as per PR comments 2018-12-05 09:31:03 +00:00
Martin Sumner
578a9f88e0 Support for log settings at startup
Both log level and forced_logs.  Allows for log_level to be changed at startup ad runtime.  Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
Martin Sumner
968be4c4be Remove incorrect mod entry 2018-12-04 20:15:29 +00:00
Ulf Wiger
88c233f911 dynamically check log_level 2018-11-30 10:38:03 +01:00
Martin Sumner
aacc2adc54
Merge pull request #212 from martinsumner/mas-i211-breakingfolds
Mas i211 breakingfolds
2018-11-24 10:33:50 +00:00
Martin Sumner
6d2d0694e3 Reverse necessary on bucket list
The function should see the buckets in order, so it accumulates in such a way to reverse the order - it makes sense that the outcome should be in reverse.
2018-11-23 19:03:24 +00:00
Martin Sumner
a9aa23bc9c Bucket list
update the docs to advertise throw capability.  Test it for bucket list (and fix ordering of bucket lists)
2018-11-23 18:56:30 +00:00
Martin Sumner
ef2a8c62af Add capability to exit a head or object fold with a throw
This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected.

This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception).  This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
2018-11-23 16:00:11 +00:00
Martin Sumner
da5abb4e4c
Merge pull request #210 from martinsumner/mas-i209-rangeissue
Mas i209 rangeissue
2018-11-15 08:56:25 +00:00
Martin Sumner
fab11bc2d2 Update comments to assist with clarity 2018-11-15 08:54:06 +00:00
Martin Sumner
a12931b430 Add comments 2018-11-15 01:06:37 +00:00
Martin Sumner
b571be9e43 Check that seglist-filtered keys are actually in range 2018-11-15 00:00:18 +00:00
Martin Sumner
3c02901ab3
Merge pull request #208 from martinsumner/mas-i207-althashkey
Mas i207 althashkey
2018-11-09 19:13:49 +00:00
Martin Sumner
0aab208e9b Additional export
Export changes required to support kv_index_tictactree.  This will call tictac_hahs, but also needs to know hash of key used in bxor - so that it can control alterations.
2018-11-09 16:05:48 +00:00
Martin Sumner
bcdf553223 Export function 2018-11-09 15:53:15 +00:00
Martin Sumner
a7773b148d Split hash - seperate key has for bxor with value 2018-11-09 14:51:38 +00:00
Martin Sumner
4756d402a0
Merge pull request #206 from martinsumner/mas-i204-largeseglists
Mas i204 largeseglists
2018-11-07 22:57:25 +00:00
Martin Sumner
2afb160a12 Add test - large seglist 2018-11-07 21:35:21 +00:00
Martin Sumner
ea7aa3086d Refactor membership check
To change to set membership when size beyond threshold
2018-11-07 17:43:26 +00:00
Martin Sumner
aaf10135a1
Merge pull request #205 from martinsumner/mas-xxx-timingtests
Mas xxx timingtests
2018-11-07 17:03:26 +00:00
Martin Sumner
787b02e4d8 Switch to leveled_rand for compatibility 2018-11-07 17:01:51 +00:00
Martin Sumner
6f4f1b9560 Add membership timing tests - sets vs lists 2018-11-07 16:57:00 +00:00
Martin Sumner
62f0351906
Merge pull request #203 from martinsumner/mas-i202-fixuptypes
Tidy up unexported types
2018-11-05 16:18:29 +00:00
Martin Sumner
174a40aab2 Tidy up unexported types
also re:mp may not be exported in R16
2018-11-05 16:02:19 +00:00
Martin Sumner
d3c9104336
Merge pull request #201 from martinsumner/mas-i189-tuplebuckets-take2
Mas i189 tuplebuckets take2
2018-11-05 12:22:36 +00:00
Martin Sumner
e9fb893ea0 Check segment is as expected with tuplebuckets
In head_only mode
2018-11-05 10:31:15 +00:00
Martin Sumner
e72a946f43 TupleBuckets in Riak objects
Adds support with test for tuplebuckets in Riak keys.

This exposed that there was no filter using the seglist on the in-mmemory keys.  This means that if there is no filter applied in the fold_function, many false positives may emerge.

This is probably not a big performance benefit (and indeed for performance it may be better to apply during the leveled_pmem:merge_trees).

Some thought still required as to what is more likely to contribute to future bugs: an extra location using the hash matching found in leveled_sst, or the extra results in the query.
2018-11-05 01:21:08 +00:00
Martin Sumner
37cdb22979
Merge pull request #200 from martinsumner/mas-i198-recentaae
Mas i198 recentaae
2018-11-04 20:51:13 +00:00
Martin Sumner
4b8f493fd6 Allow compariosn with empty binaries
If trees are non-existent, then fetch_root may return an empty binary (in tictac aae) - still need to compare
2018-11-02 17:34:58 +00:00
Martin Sumner
2eec8a5378 MaxCount monitoring and responding
Stop issue of {no_more_keys, Acc} being passed on fold over list of ranges to next range (and blowing up)
2018-11-01 23:40:28 +00:00
Martin Sumner
dc84eabe0c Revert "Temp log"
This reverts commit 2b57ff831c.
2018-11-01 20:16:08 +00:00
Martin Sumner
c126872fab Revert "More temp logs"
This reverts commit 7e959f1353.
2018-11-01 20:16:04 +00:00
Martin Sumner
7e959f1353 More temp logs 2018-11-01 20:04:22 +00:00
Martin Sumner
2b57ff831c Temp log 2018-11-01 19:58:32 +00:00
Martin Sumner
71fa1447e0 Allow for all keys head folds to used modifed range
This helps with kv_index_tictcatree with the leveled_so backend.  Now this cna do folds over ranges of keys with modified filters (as folds over ranges of keys must go over lal keys if the backend is segment_ordered)
2018-11-01 17:30:18 +00:00
Martin Sumner
aa123a80a7 Allow for backwards/forwards compatibility in specs 2018-11-01 12:40:24 +00:00
Martin Sumner
acd5db779a Merge branch 'master' into mas-i198-recentaae 2018-11-01 10:59:05 +00:00
Martin Sumner
8e752ce93c
Merge pull request #199 from martinsumner/mas-i196-expandseglist
Mas i196 expandseglist
2018-11-01 10:52:29 +00:00
Martin Sumner
f77dc8c3a5 Add object_spec type
Initial refactor to prepare to allow for a new version object_spec type that will support LMD being promoted as an accessible item.
2018-11-01 10:41:46 +00:00
Martin Sumner
376407fa66 Extend testing
In particular discover that after objects are modifed (outside of the LMD window) - they no longer appear in results for queries in the LMD window.
2018-10-31 21:37:53 +00:00
Martin Sumner
62f1302305 Add test comparing filter with non-filter
How do timings differ - does the SST filter accelerate?  The answer appears to be yes - the filtered version takes half the time.
2018-10-31 18:34:27 +00:00
Martin Sumner
19bfe48564 Initial ct test
Which exposed it wasn't working.  If there is no segment list passed - just a modification filter, you don't need to check the position list (as checking the position list returns an empty position so sipping all the matching results!)
2018-10-31 16:35:53 +00:00