Commit graph

1471 commits

Author SHA1 Message Date
Martin Sumner
420322c14f
Merge pull request #238 from martinsumner/mas-innn-schemamixup
Typo
2018-12-14 22:43:46 +00:00
Martin Sumner
8ccb985775 Typo 2018-12-14 22:43:21 +00:00
Martin Sumner
d246fd363e
Merge pull request #237 from martinsumner/mas-i233-multibackend
Mas i233 multibackend
2018-12-14 22:00:03 +00:00
Martin Sumner
5b578ea2bb Add new snapshot timeout to multi-backend 2018-12-14 19:57:32 +00:00
Martin Sumner
e6d868f8cd Merge branch 'master' into mas-i233-multibackend 2018-12-14 19:55:34 +00:00
Martin Sumner
534fcdb1b6
Merge pull request #236 from martinsumner/mas-i234-snapshottimeout
Mas i234 snapshottimeout
2018-12-14 19:49:06 +00:00
Martin Sumner
ceea196cc0 Update priv/leveled.schema
The snapshottimeouts would not normally eb changed - so make them hidden
2018-12-14 14:27:44 +00:00
Martin Sumner
ef068326a0 Update priv/leveled.schema 2018-12-14 14:13:44 +00:00
Martin Sumner
2741c46daa Add timeout for inker snapshots
So that they can also be released if they silently crash without closing neatly.
2018-12-14 13:53:36 +00:00
Martin Sumner
8bf36214e1 Make snapshot timeout configurable
Also increase defaults.  In riak snapshots may be used to fold over all objects, and with delays at the receiving end, this could take significant time
2018-12-14 11:23:04 +00:00
Martin Sumner
da17d6c9a4 Merge branch 'master' into mas-i233-multibackend 2018-12-14 10:01:52 +00:00
Martin Sumner
b344b7d827
Merge pull request #232 from martinsumner/mas-pr231-review
Mas pr231 review
2018-12-14 10:01:16 +00:00
Martin Sumner
45f62ffa56 Make all multi schema hidden 2018-12-13 16:24:33 +00:00
Martin Sumner
f211e587f7 Make multi_schema mainly hidden
As wiht other schemas - make the multi_backend schema mainly hidden
2018-12-12 16:16:47 +00:00
Martin Sumner
52c523dbab Add multi_backend schema 2018-12-12 09:56:06 +00:00
Martin Sumner
7a876066e2 Resolve coverage gaps
logs should always at least have a log_level defined which is in the ?LOG_LEVELS - so the final case of

`-is_active_level([]   , _, _) -> false.`

is unnecessary (even if a bad log level is passed in).

correct Bucket and Key definition to hit expected eunit test cases in leveled_pmem
2018-12-11 22:25:37 +00:00
Martin Sumner
672cfd4fcd Allow for run-time changes to log_level and forced_logs
Will not lead to immediate run time changes in SST or CDB logs.  These log settings will only change once the new files are re-written.

To completely change the log level - a restart of the store is necessary with new startup options.
2018-12-11 21:59:57 +00:00
Martin Sumner
6677f2e5c6 Push log update through to cdb/sst
Using the cdb_options and sst_options records
2018-12-11 20:42:00 +00:00
Martin Sumner
9ca6b499e1 Remove use of string rather than straddle OTP version
string functions were used in unit tetss only, and were replaceable with io_lib:format
2018-12-11 15:44:37 +00:00
Martin Sumner
90574122c9 Merge remote-tracking branch 'aeternity/uw-avoid-set_env' into mas-pr231-review 2018-12-10 18:33:23 +00:00
Ulf Wiger
d30ca16a17 store log opts in pdict + inherit at proc start 2018-12-10 16:09:11 +01:00
Martin Sumner
4387837279
Merge pull request #230 from martinsumner/mas-i229-loadrepeatedkey
Mas i229 loadrepeatedkey
2018-12-10 14:08:24 +00:00
Martin Sumner
f274d2a63a Tighten acceptable duration
even with cover, passes in 30s.
2018-12-10 13:23:39 +00:00
Martin Sumner
b38d845c3a Update src/leveled_pmem.erl
Simple change to check for presence of objetc in list before adding it:

```Head check took 124416 microseconds checking list of length 5000

Head check took 130114 microseconds checking list of length 5000

Check for presence of repeated objects

Head check took 1725 microseconds checking list of length 5```
2018-12-10 12:01:11 +00:00
Martin Sumner
e73f48a18b Add failing test
Test fails as fetching repeated object is too slow.

```Head check took 124301 microseconds checking list of length 5000

Head check took 112286 microseconds checking list of length 5000

Head check took 1336512 microseconds checking list of length 5

2018-12-10T11:54:41.342 B0013 <0.2459.0> Long running task took 260788 microseconds with task of type pcl_head

2018-12-10T11:54:41.618 B0013 <0.2459.0> Long running task took 276508 microseconds with task of type pcl_head

2018-12-10T11:54:41.894 B0013 <0.2459.0> Long running task took 275225 microseconds with task of type pcl_head

2018-12-10T11:54:42.173 B0013 <0.2459.0> Long running task took 278836 microseconds with task of type pcl_head

2018-12-10T11:54:42.477 B0013 <0.2459.0> Long running task took 304524 microseconds with task of type pcl_head```

It taks twice as long to check for one repeated object as it does to check for 5K non-repeated objects
2018-12-10 11:58:21 +00:00
Martin Sumner
401228c84e
Merge pull request #228 from martinsumner/mas-i223-dialyzer
Fix dialyzer warnings
2018-12-07 14:36:42 +00:00
Martin Sumner
4b4b774c0d Fix dialyzer warnings
Dialyzer got smarter in OTP 21 and spotted that the output type was wrong from tune_seglist
2018-12-07 14:36:18 +00:00
Martin Sumner
bb0d088aae
Update leveled.app.src
Switch release reference
2018-12-07 12:50:24 +00:00
Martin Sumner
7821aed6c6
Merge pull request #227 from martinsumner/mas-lz4ref
Change lz4 ref
2018-12-07 12:48:19 +00:00
Martin Sumner
c1b3e7c793 Change lz4 ref
there were no local modifications - so use tagged version of szktty (which is in hex)
2018-12-07 12:47:48 +00:00
Martin Sumner
43033a5e19
Merge pull request #226 from martinsumner/mas-appsrc
Mas appsrc
2018-12-07 12:29:18 +00:00
Martin Sumner
1fc73f9092 Update src/leveled.app.src 2018-12-07 12:26:41 +00:00
Martin Sumner
019a7a02e9 Update src/leveled.app.src 2018-12-07 12:26:11 +00:00
Martin Sumner
b9a95978dd
Update README.md 2018-12-07 11:53:54 +00:00
Martin Sumner
d395b4fb48
Merge pull request #225 from martinsumner/mas-i222-extractmetadata
Mas i222 extractmetadata
2018-12-07 11:52:21 +00:00
Martin Sumner
fe177f306d Update docs 2018-12-07 11:42:59 +00:00
Martin Sumner
714e128df8 Tidy up protecting against corrupt Keys
this was previously not na issue as leveled_codec:segment_hash/1 would handle anyhting that could be hashed.  This now has to be a tuple, and one with a first element - so corrupted tuples are failing.

Add a guard chekcing for a corrupted tuple, but we only need this when doing journal compaction.

Change user_defined keys to be `retain` as a tag strategy
2018-12-07 09:07:22 +00:00
Ulf Wiger
fbe200f1ca warning-free vsns of string:str/2 & string:right/3 2018-12-07 08:39:44 +01:00
Martin Sumner
cee5a60ceb Protect against bad journal keys in Ledger 2018-12-07 00:48:42 +00:00
Martin Sumner
3ff51c000c Typo 2018-12-06 22:55:00 +00:00
Martin Sumner
e0352414f2 iClerk refactor
the skip/retain/recalc handlign was confusing.  This removes the switcheroo between leveled_codec and leveled_iclerk when mkaing the decision.

Also now the building of the accumulator is handled efficiently (not using ++ on the list).

Tried to rmeove as much of ?HEAD tag handling from leveled_head - as we want leveled_head to be only concerned with the head manipulation for object tags (?STD, ?RIAK and user-defined).
2018-12-06 22:45:05 +00:00
Martin Sumner
8e687ee7c8 Add user-defined functions
To allow for extraction of metadata, and building of head responses - it should eb possible to dynamically and user-defined tags, and functions to treat them.

If no function is defined, revert to the behaviour of the ?STD tag.
2018-12-06 21:00:59 +00:00
Martin Sumner
881b93229b Isolate better changes needed to support changes to metadata extraction
More obvious how to extend the code as it is all in one module.

Also add a new field to the standard object metadata tuple that may hold in the future other object metadata base don user-defined functions.
2018-12-06 15:31:11 +00:00
Martin Sumner
bfddb53e31
Merge pull request #221 from martinsumner/mas-i219-tuplebuckets
Deprecate use of Bucket as a constraint in indexfold
2018-12-05 20:47:40 +00:00
Martin Sumner
835955adb9 Deprecate use of Bucket as a constraint in indexfold
Must also give a StartKey.  Otherwise there may be conflict when tuplebuckets are used
2018-12-05 20:46:17 +00:00
Martin Sumner
f415d22ef0
Merge pull request #220 from martinsumner/mas-i157-indexttl
Add test of index expiry
2018-12-05 15:42:43 +00:00
Martin Sumner
510994233e Add check that index disappears
Check I0 count goes down when that index is removed
2018-12-05 15:42:21 +00:00
Martin Sumner
cf1fcaeef2 Add test of index expiry
To show how this works, and prove that it does work thta way.

Test may require adjusting if tested on a slow node (e.g. reduce KeyCount or increase TTL)
2018-12-05 15:18:20 +00:00
Martin Sumner
150fcf15d6
Merge pull request #217 from martinsumner/mas-i216-nomod
Remove incorrect mod entry
2018-12-05 09:31:57 +00:00
Martin Sumner
d438c29200
Merge pull request #218 from martinsumner/mas-uw-configurableloglevel
Mas uw configurableloglevel
2018-12-05 09:31:31 +00:00