Commit graph

80 commits

Author SHA1 Message Date
Martin Sumner
e360b97cfb GC manifest files when numbers skipped
Otherwise list of old files perpetually grows
2019-05-23 10:16:15 +01:00
Martin Sumner
14e1f577c9 Test default tag 2019-03-14 00:08:01 +00:00
Martin Sumner
01f0dadbb3 Add access to SQN
Use book_sqn/3 or book_sqn/4 to get the SQN of an object in the store.
2019-03-13 16:21:03 +00:00
Martin Sumner
be6e23f7de Change cache_size in sst tests
Makes results more predictable (with coin toss variations)
2019-01-29 13:40:55 +00:00
Martin Sumner
e3bd83179a Uncomment tests! 2019-01-27 23:31:44 +00:00
Martin Sumner
8f6862a10b Test sst slot configuration change
Confirm it results in many more files, if the slot count reduced.  Has to handle the fact that Level 0 file has unlimited slots regardless of number of slots configured
2019-01-27 22:03:55 +00:00
Martin Sumner
28d0aef5fe Make check that compaction not ongoing before accepting new compaction
Respond 'busy' if compaction is ongoing
2019-01-24 15:46:17 +00:00
Martin Sumner
c060c0e41d Handle L0 cache being full
A test thta will cause leveled to crash due to a low cache size being set - but protect against this (as well as the general scenario of the cache being full).

There could be a potential case where a L0 file present (post pending) without work backlog being set.  In this case we want to roll the level zero to memory, but don't accept the cache update if the L0 cache is already full.
2019-01-14 12:27:51 +00:00
Martin Sumner
672cfd4fcd Allow for run-time changes to log_level and forced_logs
Will not lead to immediate run time changes in SST or CDB logs.  These log settings will only change once the new files are re-written.

To completely change the log level - a restart of the store is necessary with new startup options.
2018-12-11 21:59:57 +00:00
Martin Sumner
6677f2e5c6 Push log update through to cdb/sst
Using the cdb_options and sst_options records
2018-12-11 20:42:00 +00:00
Martin Sumner
881b93229b Isolate better changes needed to support changes to metadata extraction
More obvious how to extend the code as it is all in one module.

Also add a new field to the standard object metadata tuple that may hold in the future other object metadata base don user-defined functions.
2018-12-06 15:31:11 +00:00
Martin Sumner
578a9f88e0 Support for log settings at startup
Both log level and forced_logs.  Allows for log_level to be changed at startup ad runtime.  Also allow for a list of forced logs, so if log_level is set > info, individual info logs can be forced to be seen (such as to see stats logs).
2018-12-05 00:17:39 +00:00
Martin Sumner
0bbaf1f25e Handle missing journal files without data loss
Make sure data added after the journal has been lost will not be lost (even though lost data may stay lost)
2018-10-03 18:38:56 +01:00
Martin Sumner
9a0082db4e Test to expose startup issue
See #194
2018-10-03 18:29:20 +01:00
Martin Sumner
39e366a31b Add ct test
Which passes
2018-09-27 11:32:42 +01:00
Martin Sumner
051fcd672c Add test of switching from a big journal to a little journal
And sending objects over the journal size
2018-09-25 16:56:24 +01:00
Russell Brown
b7bd65d11f Provide a top level API for folds
As the fold functions have been added to get_runner in an ad hoc way,
naturally, given the ongoing development of levelEd to support Riak,
it was difficult for a new user (in this case Quviq) to see what folds
are supported, and with what arguments, and expectations.

This PR is for discussion. It is one of many ways to group, spec, and
document the fold functions.

A test is also added for coverage of range queries.
2018-09-06 15:01:54 +01:00
Martin Sumner
50967438d3 Switch from binary_bucketlist
Allow for bucket listing of non-binary buckets (integer buckets, buckets with ascii strings)
2018-09-01 10:39:23 +01:00
Martin Sumner
eb5d9596b2 Add comments to test
Also loads some more objects to potentially make results clearer
2018-05-17 18:20:16 +01:00
Martin Sumner
82a1449447 Fix bad accumulator in test
Test takes a long time due to sleep (still need to work on that), but also FoldKeysFun uses ++ rather than [{B, K}|Acc] to extend the list.  Order of magnitude speed-up for these queries by changing the way this accumulates
2018-05-01 22:30:28 +01:00
Martin Sumner
faa5ef82aa Test logging of samples
To prompt the log the journal size needs to be reduced
2017-11-20 15:31:31 +00:00
Martin Sumner
29ffb74a5b Add coverage for CDB delete_pending with snapshot
CDB file in delete_pending state, and times out but the Inker has a snapshot open - and so the file is not deleted.
2017-11-09 12:02:03 +00:00
Martin Sumner
7de4dccbd9 Extend journal compaction test
to cover with and without  waste retention.  Also makes sure that CDB files in a restarted store will respect the wast retention period set.
2017-11-08 16:18:48 +00:00
Martin Sumner
1d475235d1 Improve test coverage
Make compress on receipt/compaction configurable
2017-11-06 18:44:08 +00:00
Martin Sumner
61b7be5039 Make compression algorithm an option
Compression can be switched between LZ4 and zlib (native).

The setting to determine if compression should happen on receipt is now a macro definition in leveled_codec.
2017-11-06 15:54:58 +00:00
Martin Sumner
36264eb416 Search range failure
Discovered a bug with search ranges in leveled_tree - this was uncovered by an intermittently fialing 19.3 test.

Test case added and bug fixed.  It was due to a fialure to use end_key passed causing issues with particular manifests and full bucket ranges.
2017-10-24 13:19:30 +01:00
Martin Sumner
bfaed921e6 Split code for folders - introduce runner actor
Introduce a dedicated module for all the different fold types.  Also simplify the list of folders by deprecating those folds that should eb achieveable by fold_heads/fold_objects type folds but with smarter functions.

Makes sure that the fold functiosn also have better spec coverage, and are dialyzer checked.
2017-10-17 20:39:11 +01:00
Martin Sumner
3950942da3 Roll in fix for intermittently failing test
As descibed in https://github.com/martinsumner/leveled/issues/92

Only the first fix was made.

Just to eb safe - archiving means renaming to another file with a different extension.  Assumption is that renamed files cna be manually reaped if necessary.
2017-09-27 23:52:49 +01:00
Martin Sumner
869e799b41 Fix tests
Obviously got totally messed up and confused when testing previous
commits.

Multiple tests were failing for a change which got merged in as the
tests were not reflecting the required API.
2017-09-15 10:33:16 +01:00
Martin Sumner
fd84e4f608 Test timeouts
So that coverage testing will run.
2017-07-02 22:23:02 +01:00
Martin Sumner
15c52ae118 Change default compaction settings
Need to allow specific settings to be passed into unit tests.

Also, too much journal compaction may lead to intermittent failures on
the basic_SUITE space_clear_on_delete test.  think this is because
there are less “deletes” to reload in on startup to trigger the cascade
down and clear up?
2017-06-02 08:37:57 +01:00
martinsumner
0d8ab0899e Add test for is_empty
Bucket listing didn't care if keys were active - now does.
2017-05-23 11:59:44 +01:00
Martin Sumner
fa9daf8696 Correct async fold
fold objects which snaps in the fold was implemented incorrectly - it
took information from the LedgeCache at the point of the request, not
at the point of the fold.  So the LedgerCache SQN may have been
surpassed in the Penciller by the time the fold was called.
2017-04-17 23:01:55 +01:00
Martin Sumner
994d9758b0 Object now 4 bytes larger
due to addition of key changes length
2017-03-20 16:21:49 +00:00
martinsumner
32cdf26ffb Correct to catch new response to snapshot request 2017-03-06 19:37:43 +00:00
martinsumner
41e5686a66 Test coverage - missing 1% 2017-02-26 20:52:40 +00:00
martinsumner
0ddaaf9ac3 Stopped unnecessary seek for last_key
When rolling we already know the last_key - no need to seek for it on
startup.

The time it takes for this seek needs to be considered with regards to
startup time.  Can we do without knowing lastkey?
2016-12-22 19:51:39 +00:00
martinsumner
e8c1d39df9 Switch to binary format Riak object
Initial change to try and test assuming that leveled received the binary
format of Riak objects (and parses that for metadata).
2016-11-28 22:26:09 +00:00
martinsumner
45f7fcb41a Datasync added for riak_sync strategy 2016-11-27 00:16:49 +00:00
martinsumner
196c807b5e Pass through sync_strategy
Allow to switch for Riak to use o_sync as the sync flag rather than sync
2016-11-25 17:41:08 +00:00
martinsumner
0f7e421371 Add destruction
Allow a store to be cleared out and destroyed
2016-11-21 12:34:40 +00:00
martinsumner
57727e74f9 D'oh 2016-11-18 17:14:01 +00:00
martinsumner
f5944f2466 Change test foler to return (Bucket, Key} as expeceted 2016-11-18 16:52:53 +00:00
martinsumner
6684e8e1d3 Refine query to accept fold functions
Need to be able to pass external fold functions into different queries,
to work as a Riak backend
2016-11-18 15:53:22 +00:00
martinsumner
630f802780 Inker Close nastiness
Try to stop some of the potential deadlocking around Inker close and
prove that snapshots at higher Manifest SQNs can be ignored
2016-11-14 19:34:11 +00:00
martinsumner
44738f7c75 Deferred Deletion of Journals
This allows for deleted journals to be retained for a period (the
waste_retnetion_period).  The idea being that a backup strategy can
ensure that all journals are backed up, even ones created and removed
from within a backup period - so that any restore pont is possible.

This is also a pre-cursor to removing some of the PromptDelete
complexity from the Inker Clerk - all compactions can prompt deletion as
deletion is now deferred.
2016-11-14 11:17:14 +00:00
martinsumner
37c23a5b38 Shift pause out of leveled
Leveled will now signal the need for a pause due to back-pressure, but
not actually pause itself.  The hope is that in a riak implementation
this pause can be managed by the put_fsm, and so not lock the store.
2016-11-07 10:27:38 +00:00
martinsumner
4583460328 Clean API of Riak-specific Methods
Clena the API of Riak specific methods, and also resolve timing issue in
simple_server unit test.  Previously this would end up with missing data
(and a lower sequence number after start) because of the penciller_clerk
timeout being relatively large in the context of this test.  Now the
timeout has bene reduced the L0 slot is cleared by the time of the
close.  To make sure an extra sleep has been added as a precaution to
avoid any intermittent issues.
2016-11-07 10:11:57 +00:00
martinsumner
479dc3ac80 Registering and releasing of Journal snapshots
Added a test of journal compaction with a registered snapshot and it
showed that the deleting of files did not correctly check the list of
registerd snapshots.  Corrected.
2016-11-04 15:56:57 +00:00
martinsumner
2f28ae86e4 Journal compaction test to cover deleted objects 2016-11-03 12:31:00 +00:00