Commit graph

1433 commits

Author SHA1 Message Date
Martin Sumner
cee5a60ceb Protect against bad journal keys in Ledger 2018-12-07 00:48:42 +00:00
Martin Sumner
3ff51c000c Typo 2018-12-06 22:55:00 +00:00
Martin Sumner
e0352414f2 iClerk refactor
the skip/retain/recalc handlign was confusing.  This removes the switcheroo between leveled_codec and leveled_iclerk when mkaing the decision.

Also now the building of the accumulator is handled efficiently (not using ++ on the list).

Tried to rmeove as much of ?HEAD tag handling from leveled_head - as we want leveled_head to be only concerned with the head manipulation for object tags (?STD, ?RIAK and user-defined).
2018-12-06 22:45:05 +00:00
Martin Sumner
8e687ee7c8 Add user-defined functions
To allow for extraction of metadata, and building of head responses - it should eb possible to dynamically and user-defined tags, and functions to treat them.

If no function is defined, revert to the behaviour of the ?STD tag.
2018-12-06 21:00:59 +00:00
Martin Sumner
881b93229b Isolate better changes needed to support changes to metadata extraction
More obvious how to extend the code as it is all in one module.

Also add a new field to the standard object metadata tuple that may hold in the future other object metadata base don user-defined functions.
2018-12-06 15:31:11 +00:00
Martin Sumner
bfddb53e31
Merge pull request #221 from martinsumner/mas-i219-tuplebuckets
Deprecate use of Bucket as a constraint in indexfold
2018-12-05 20:47:40 +00:00
Martin Sumner
835955adb9 Deprecate use of Bucket as a constraint in indexfold
Must also give a StartKey.  Otherwise there may be conflict when tuplebuckets are used
2018-12-05 20:46:17 +00:00
Martin Sumner
f415d22ef0
Merge pull request #220 from martinsumner/mas-i157-indexttl
Add test of index expiry
2018-12-05 15:42:43 +00:00
Martin Sumner
510994233e Add check that index disappears
Check I0 count goes down when that index is removed
2018-12-05 15:42:21 +00:00
Martin Sumner
cf1fcaeef2 Add test of index expiry
To show how this works, and prove that it does work thta way.

Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
Martin Sumner
150fcf15d6
Merge pull request #217 from martinsumner/mas-i216-nomod
Remove incorrect mod entry
2018-12-05 09:31:57 +00:00
Martin Sumner
d438c29200
Merge pull request #218 from martinsumner/mas-uw-configurableloglevel
Mas uw configurableloglevel
2018-12-05 09:31:31 +00:00
Martin Sumner
9735661d98 Typos as per PR comments 2018-12-05 09:31:03 +00:00
Martin Sumner
578a9f88e0 Support for log settings at startup
Both log level and forced_logs.  Allows for log_level to be changed at startup ad runtime.  Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
Martin Sumner
968be4c4be Remove incorrect mod entry 2018-12-04 20:15:29 +00:00
Ulf Wiger
88c233f911 dynamically check log_level 2018-11-30 10:38:03 +01:00
Martin Sumner
aacc2adc54
Merge pull request #212 from martinsumner/mas-i211-breakingfolds
Mas i211 breakingfolds
2018-11-24 10:33:50 +00:00
Martin Sumner
6d2d0694e3 Reverse necessary on bucket list
The function should see the buckets in order, so it accumulates in such a way to reverse the order - it makes sense that the outcome should be in reverse.
2018-11-23 19:03:24 +00:00
Martin Sumner
a9aa23bc9c Bucket list
update the docs to advertise throw capability.  Test it for bucket list (and fix ordering of bucket lists)
2018-11-23 18:56:30 +00:00
Martin Sumner
ef2a8c62af Add capability to exit a head or object fold with a throw
This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected.

This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception).  This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
2018-11-23 16:00:11 +00:00
Martin Sumner
da5abb4e4c
Merge pull request #210 from martinsumner/mas-i209-rangeissue
Mas i209 rangeissue
2018-11-15 08:56:25 +00:00
Martin Sumner
fab11bc2d2 Update comments to assist with clarity 2018-11-15 08:54:06 +00:00
Martin Sumner
a12931b430 Add comments 2018-11-15 01:06:37 +00:00
Martin Sumner
b571be9e43 Check that seglist-filtered keys are actually in range 2018-11-15 00:00:18 +00:00
Martin Sumner
3c02901ab3
Merge pull request #208 from martinsumner/mas-i207-althashkey
Mas i207 althashkey
2018-11-09 19:13:49 +00:00
Martin Sumner
0aab208e9b Additional export
Export changes required to support kv_index_tictactree.  This will call tictac_hahs, but also needs to know hash of key used in bxor - so that it can control alterations.
2018-11-09 16:05:48 +00:00
Martin Sumner
bcdf553223 Export function 2018-11-09 15:53:15 +00:00
Martin Sumner
a7773b148d Split hash - seperate key has for bxor with value 2018-11-09 14:51:38 +00:00
Martin Sumner
4756d402a0
Merge pull request #206 from martinsumner/mas-i204-largeseglists
Mas i204 largeseglists
2018-11-07 22:57:25 +00:00
Martin Sumner
2afb160a12 Add test - large seglist 2018-11-07 21:35:21 +00:00
Martin Sumner
ea7aa3086d Refactor membership check
To change to set membership when size beyond threshold
2018-11-07 17:43:26 +00:00
Martin Sumner
aaf10135a1
Merge pull request #205 from martinsumner/mas-xxx-timingtests
Mas xxx timingtests
2018-11-07 17:03:26 +00:00
Martin Sumner
787b02e4d8 Switch to leveled_rand for compatibility 2018-11-07 17:01:51 +00:00
Martin Sumner
6f4f1b9560 Add membership timing tests - sets vs lists 2018-11-07 16:57:00 +00:00
Martin Sumner
62f0351906
Merge pull request #203 from martinsumner/mas-i202-fixuptypes
Tidy up unexported types
2018-11-05 16:18:29 +00:00
Martin Sumner
174a40aab2 Tidy up unexported types
also re:mp may not be exported in R16
2018-11-05 16:02:19 +00:00
Martin Sumner
d3c9104336
Merge pull request #201 from martinsumner/mas-i189-tuplebuckets-take2
Mas i189 tuplebuckets take2
2018-11-05 12:22:36 +00:00
Martin Sumner
e9fb893ea0 Check segment is as expected with tuplebuckets
In head_only mode
2018-11-05 10:31:15 +00:00
Martin Sumner
e72a946f43 TupleBuckets in Riak objects
Adds support with test for tuplebuckets in Riak keys.

This exposed that there was no filter using the seglist on the in-mmemory keys.  This means that if there is no filter applied in the fold_function, many false positives may emerge.

This is probably not a big performance benefit (and indeed for performance it may be better to apply during the leveled_pmem:merge_trees).

Some thought still required as to what is more likely to contribute to future bugs: an extra location using the hash matching found in leveled_sst, or the extra results in the query.
2018-11-05 01:21:08 +00:00
Martin Sumner
37cdb22979
Merge pull request #200 from martinsumner/mas-i198-recentaae
Mas i198 recentaae
2018-11-04 20:51:13 +00:00
Martin Sumner
4b8f493fd6 Allow compariosn with empty binaries
If trees are non-existent, then fetch_root may return an empty binary (in tictac aae) - still need to compare
2018-11-02 17:34:58 +00:00
Martin Sumner
2eec8a5378 MaxCount monitoring and responding
Stop issue of {no_more_keys, Acc} being passed on fold over list of ranges to next range (and blowing up)
2018-11-01 23:40:28 +00:00
Martin Sumner
dc84eabe0c Revert "Temp log"
This reverts commit 2b57ff831c.
2018-11-01 20:16:08 +00:00
Martin Sumner
c126872fab Revert "More temp logs"
This reverts commit 7e959f1353.
2018-11-01 20:16:04 +00:00
Martin Sumner
7e959f1353 More temp logs 2018-11-01 20:04:22 +00:00
Martin Sumner
2b57ff831c Temp log 2018-11-01 19:58:32 +00:00
Martin Sumner
71fa1447e0 Allow for all keys head folds to used modifed range
This helps with kv_index_tictcatree with the leveled_so backend.  Now this cna do folds over ranges of keys with modified filters (as folds over ranges of keys must go over lal keys if the backend is segment_ordered)
2018-11-01 17:30:18 +00:00
Martin Sumner
aa123a80a7 Allow for backwards/forwards compatibility in specs 2018-11-01 12:40:24 +00:00
Martin Sumner
acd5db779a Merge branch 'master' into mas-i198-recentaae 2018-11-01 10:59:05 +00:00
Martin Sumner
8e752ce93c
Merge pull request #199 from martinsumner/mas-i196-expandseglist
Mas i196 expandseglist
2018-11-01 10:52:29 +00:00