Commit graph

1351 commits

Author SHA1 Message Date
Martin Sumner
f211e587f7 Make multi_schema mainly hidden
As wiht other schemas - make the multi_backend schema mainly hidden
2018-12-12 16:16:47 +00:00
Martin Sumner
52c523dbab Add multi_backend schema 2018-12-12 09:56:06 +00:00
Martin Sumner
4387837279
Merge pull request #230 from martinsumner/mas-i229-loadrepeatedkey
Mas i229 loadrepeatedkey
2018-12-10 14:08:24 +00:00
Martin Sumner
f274d2a63a Tighten acceptable duration
even with cover, passes in 30s.
2018-12-10 13:23:39 +00:00
Martin Sumner
b38d845c3a Update src/leveled_pmem.erl
Simple change to check for presence of objetc in list before adding it:

```Head check took 124416 microseconds checking list of length 5000

Head check took 130114 microseconds checking list of length 5000

Check for presence of repeated objects

Head check took 1725 microseconds checking list of length 5```
2018-12-10 12:01:11 +00:00
Martin Sumner
e73f48a18b Add failing test
Test fails as fetching repeated object is too slow.

```Head check took 124301 microseconds checking list of length 5000

Head check took 112286 microseconds checking list of length 5000

Head check took 1336512 microseconds checking list of length 5

2018-12-10T11:54:41.342 B0013 <0.2459.0> Long running task took 260788 microseconds with task of type pcl_head

2018-12-10T11:54:41.618 B0013 <0.2459.0> Long running task took 276508 microseconds with task of type pcl_head

2018-12-10T11:54:41.894 B0013 <0.2459.0> Long running task took 275225 microseconds with task of type pcl_head

2018-12-10T11:54:42.173 B0013 <0.2459.0> Long running task took 278836 microseconds with task of type pcl_head

2018-12-10T11:54:42.477 B0013 <0.2459.0> Long running task took 304524 microseconds with task of type pcl_head```

It taks twice as long to check for one repeated object as it does to check for 5K non-repeated objects
2018-12-10 11:58:21 +00:00
Martin Sumner
401228c84e
Merge pull request #228 from martinsumner/mas-i223-dialyzer
Fix dialyzer warnings
2018-12-07 14:36:42 +00:00
Martin Sumner
4b4b774c0d Fix dialyzer warnings
Dialyzer got smarter in OTP 21 and spotted that the output type was wrong from tune_seglist
2018-12-07 14:36:18 +00:00
Martin Sumner
bb0d088aae
Update leveled.app.src
Switch release reference
2018-12-07 12:50:24 +00:00
Martin Sumner
7821aed6c6
Merge pull request #227 from martinsumner/mas-lz4ref
Change lz4 ref
2018-12-07 12:48:19 +00:00
Martin Sumner
c1b3e7c793 Change lz4 ref
there were no local modifications - so use tagged version of szktty (which is in hex)
2018-12-07 12:47:48 +00:00
Martin Sumner
43033a5e19
Merge pull request #226 from martinsumner/mas-appsrc
Mas appsrc
2018-12-07 12:29:18 +00:00
Martin Sumner
1fc73f9092 Update src/leveled.app.src 2018-12-07 12:26:41 +00:00
Martin Sumner
019a7a02e9 Update src/leveled.app.src 2018-12-07 12:26:11 +00:00
Martin Sumner
b9a95978dd
Update README.md 2018-12-07 11:53:54 +00:00
Martin Sumner
d395b4fb48
Merge pull request #225 from martinsumner/mas-i222-extractmetadata
Mas i222 extractmetadata
2018-12-07 11:52:21 +00:00
Martin Sumner
fe177f306d Update docs 2018-12-07 11:42:59 +00:00
Martin Sumner
714e128df8 Tidy up protecting against corrupt Keys
this was previously not na issue as leveled_codec:segment_hash/1 would handle anyhting that could be hashed.  This now has to be a tuple, and one with a first element - so corrupted tuples are failing.

Add a guard chekcing for a corrupted tuple, but we only need this when doing journal compaction.

Change user_defined keys to be `retain` as a tag strategy
2018-12-07 09:07:22 +00:00
Martin Sumner
cee5a60ceb Protect against bad journal keys in Ledger 2018-12-07 00:48:42 +00:00
Martin Sumner
3ff51c000c Typo 2018-12-06 22:55:00 +00:00
Martin Sumner
e0352414f2 iClerk refactor
the skip/retain/recalc handlign was confusing.  This removes the switcheroo between leveled_codec and leveled_iclerk when mkaing the decision.

Also now the building of the accumulator is handled efficiently (not using ++ on the list).

Tried to rmeove as much of ?HEAD tag handling from leveled_head - as we want leveled_head to be only concerned with the head manipulation for object tags (?STD, ?RIAK and user-defined).
2018-12-06 22:45:05 +00:00
Martin Sumner
8e687ee7c8 Add user-defined functions
To allow for extraction of metadata, and building of head responses - it should eb possible to dynamically and user-defined tags, and functions to treat them.

If no function is defined, revert to the behaviour of the ?STD tag.
2018-12-06 21:00:59 +00:00
Martin Sumner
881b93229b Isolate better changes needed to support changes to metadata extraction
More obvious how to extend the code as it is all in one module.

Also add a new field to the standard object metadata tuple that may hold in the future other object metadata base don user-defined functions.
2018-12-06 15:31:11 +00:00
Martin Sumner
bfddb53e31
Merge pull request #221 from martinsumner/mas-i219-tuplebuckets
Deprecate use of Bucket as a constraint in indexfold
2018-12-05 20:47:40 +00:00
Martin Sumner
835955adb9 Deprecate use of Bucket as a constraint in indexfold
Must also give a StartKey.  Otherwise there may be conflict when tuplebuckets are used
2018-12-05 20:46:17 +00:00
Martin Sumner
f415d22ef0
Merge pull request #220 from martinsumner/mas-i157-indexttl
Add test of index expiry
2018-12-05 15:42:43 +00:00
Martin Sumner
510994233e Add check that index disappears
Check I0 count goes down when that index is removed
2018-12-05 15:42:21 +00:00
Martin Sumner
cf1fcaeef2 Add test of index expiry
To show how this works, and prove that it does work thta way.

Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
Martin Sumner
150fcf15d6
Merge pull request #217 from martinsumner/mas-i216-nomod
Remove incorrect mod entry
2018-12-05 09:31:57 +00:00
Martin Sumner
d438c29200
Merge pull request #218 from martinsumner/mas-uw-configurableloglevel
Mas uw configurableloglevel
2018-12-05 09:31:31 +00:00
Martin Sumner
9735661d98 Typos as per PR comments 2018-12-05 09:31:03 +00:00
Martin Sumner
578a9f88e0 Support for log settings at startup
Both log level and forced_logs.  Allows for log_level to be changed at startup ad runtime.  Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
Martin Sumner
968be4c4be Remove incorrect mod entry 2018-12-04 20:15:29 +00:00
Ulf Wiger
88c233f911 dynamically check log_level 2018-11-30 10:38:03 +01:00
Martin Sumner
aacc2adc54
Merge pull request #212 from martinsumner/mas-i211-breakingfolds
Mas i211 breakingfolds
2018-11-24 10:33:50 +00:00
Martin Sumner
6d2d0694e3 Reverse necessary on bucket list
The function should see the buckets in order, so it accumulates in such a way to reverse the order - it makes sense that the outcome should be in reverse.
2018-11-23 19:03:24 +00:00
Martin Sumner
a9aa23bc9c Bucket list
update the docs to advertise throw capability.  Test it for bucket list (and fix ordering of bucket lists)
2018-11-23 18:56:30 +00:00
Martin Sumner
ef2a8c62af Add capability to exit a head or object fold with a throw
This allows for all fold functions to throw an exception to exit out of a fold with all dependencies still closed down as expected.

This was previously available for key folds, which was necessary for the folds to work in Riak (as max_results in index queries depends one xiting the fold with an exception).  This change now adds a ct test, and adds support for head folds, object folds (key order) and object folds (sqn order)
2018-11-23 16:00:11 +00:00
Martin Sumner
da5abb4e4c
Merge pull request #210 from martinsumner/mas-i209-rangeissue
Mas i209 rangeissue
2018-11-15 08:56:25 +00:00
Martin Sumner
fab11bc2d2 Update comments to assist with clarity 2018-11-15 08:54:06 +00:00
Martin Sumner
a12931b430 Add comments 2018-11-15 01:06:37 +00:00
Martin Sumner
b571be9e43 Check that seglist-filtered keys are actually in range 2018-11-15 00:00:18 +00:00
Martin Sumner
3c02901ab3
Merge pull request #208 from martinsumner/mas-i207-althashkey
Mas i207 althashkey
2018-11-09 19:13:49 +00:00
Martin Sumner
0aab208e9b Additional export
Export changes required to support kv_index_tictactree.  This will call tictac_hahs, but also needs to know hash of key used in bxor - so that it can control alterations.
2018-11-09 16:05:48 +00:00
Martin Sumner
bcdf553223 Export function 2018-11-09 15:53:15 +00:00
Martin Sumner
a7773b148d Split hash - seperate key has for bxor with value 2018-11-09 14:51:38 +00:00
Martin Sumner
4756d402a0
Merge pull request #206 from martinsumner/mas-i204-largeseglists
Mas i204 largeseglists
2018-11-07 22:57:25 +00:00
Martin Sumner
2afb160a12 Add test - large seglist 2018-11-07 21:35:21 +00:00
Martin Sumner
ea7aa3086d Refactor membership check
To change to set membership when size beyond threshold
2018-11-07 17:43:26 +00:00
Martin Sumner
aaf10135a1
Merge pull request #205 from martinsumner/mas-xxx-timingtests
Mas xxx timingtests
2018-11-07 17:03:26 +00:00