martinsumner
a251f3eab0
Speed up query count test
...
Less individual querys to make count will speed up this taste, without
changing the nature of it
2016-11-04 18:20:00 +00:00
martinsumner
2b8a37439d
Log refinement - logging process IDs
2016-11-04 17:28:04 +00:00
martinsumner
c306d49a85
CDB Logging
...
Setup CDB logging to use log base
2016-11-04 16:11:11 +00:00
martinsumner
479dc3ac80
Registering and releasing of Journal snapshots
...
Added a test of journal compaction with a registered snapshot and it
showed that the deleting of files did not correctly check the list of
registerd snapshots. Corrected.
2016-11-04 15:56:57 +00:00
martinsumner
9abc1d643a
Logging update
...
Add SFT logs to logbase
2016-11-04 14:53:19 +00:00
martinsumner
171baefc0c
SFT Background Failure
...
Let it crash approach - stop trying to catch and propgate failure of
write
2016-11-04 14:31:19 +00:00
martinsumner
eeeee07081
Fold Objects - Check values test
...
Test that summed values in fold objects before and after restart
2016-11-04 14:23:37 +00:00
martinsumner
ba628c2f40
Test Rolling of CDB to two files trhough compaction
...
This exposed a potential issue with not opening readers in binary_mode -
so now defaults to binary mode. Will add test using object filder to
confirm values remain readable in rolled journals after
shutdown/startup.
2016-11-04 12:22:15 +00:00
martinsumner
68b17c71b3
Expand fold objects support
...
Fold over bucket and fold over index added
2016-11-04 11:01:37 +00:00
martinsumner
9ea74836ee
Alter impossible clause
...
Starts with {infinity, 0} so can never be {0.0}
2016-11-03 21:30:55 +00:00
martinsumner
41f00ba6fa
Filename nonsense
2016-11-03 20:48:23 +00:00
martinsumner
dd99d624b1
Tangling with filenames
...
filename join does not work as expected
2016-11-03 20:46:56 +00:00
martinsumner
c3a6489b93
Ensure manifest dir when starting Penciller
...
Otherwise may fail based on test ordering
2016-11-03 20:09:38 +00:00
martinsumner
d5ac4d412d
Use filename join
...
Potentiall to avoid *nix vs windows differences
2016-11-03 20:06:30 +00:00
martinsumner
ad9c886b65
Inker test for pending manifest
...
Should ignore the corrupted pending manifest file
2016-11-03 20:00:55 +00:00
martinsumner
72b9b35dac
Dump test utility - does not need to cope with CRC failure
...
Dump is used in tests only
2016-11-03 19:55:19 +00:00
martinsumner
49eed55735
Remove unnecessary clause
2016-11-03 19:39:23 +00:00
martinsumner
341e245c09
Remove unnecessary no match condition
2016-11-03 19:34:54 +00:00
martinsumner
2716d912ea
Timeout and close race
...
Race condition presvented in test - but still not handled nicely.
Perhaps need to consider making it a FSM and handling close differently
when L0 pending - i.e. don't close immediately, but set a timeout to
close on if we don't get the last fetch_levelzero
2016-11-03 19:02:50 +00:00
martinsumner
f41c788bff
Minor quibbles
...
Move legacy CDB code used only in unit tests into test area. Fix column
width in pmem and comment out the unused case statement (in healthy
tests) from the penciller test code
2016-11-03 16:46:25 +00:00
martinsumner
37e20ccdfe
L0 cache size counter improved
...
This is now accurate save for hash collisions. It may now be a small
under-estimate, whereas previously it could be a large over-estimate.
2016-11-03 16:22:51 +00:00
martinsumner
4e46c9735d
Log improvements
...
Continuation of log review and conversion to using central log function.
Fixup of convoluted shutdown process between Bookie, Inker and Inker's
Clerk
2016-11-03 16:05:43 +00:00
martinsumner
c6fc8d1768
Fix log P0005
2016-11-03 13:37:49 +00:00
martinsumner
ee39b48313
Messed up test fixed
2016-11-03 13:34:06 +00:00
martinsumner
2f28ae86e4
Journal compaction test to cover deleted objects
2016-11-03 12:31:00 +00:00
martinsumner
a752074148
Undo commenting
2016-11-03 12:12:56 +00:00
martinsumner
37e78dcdc9
Expanded AAE tests to include busted hashtable
...
Busted the hashtable in a Journal file, and demonstrated it can be fixed
by changing the extension name (no need to recover from backup if only
the hashtable is bust)
2016-11-03 12:11:50 +00:00
martinsumner
e8a7888397
Experiment with new cache size algorithm
...
Remove the jitter probability and make it a smooth function heading
towards the max ache size
2016-11-03 09:19:02 +00:00
martinsumner
94436d8dfd
Set timing rounding correctly
2016-11-02 18:56:36 +00:00
martinsumner
7147ec0470
Logging - Phase 1
...
Abstract out logging and introduce a logbase
2016-11-02 18:14:46 +00:00
martinsumner
0572f43b8a
Uncomment test
2016-11-02 15:40:22 +00:00
martinsumner
898f86a08d
Add Fold Object by KeyList support
2016-11-02 15:38:51 +00:00
martinsumner
8601e219d5
Revert test commenting
...
Commented out some tests - bring back in
2016-11-02 13:34:34 +00:00
martinsumner
a56ed18ba9
Test timing
...
Look to see if test timing related to intermittent failure
2016-11-02 13:27:16 +00:00
martinsumner
e7506c3c1f
Startup work - baffled
...
Changes the stratup otpions to a prolist to make it easier to get
environment variables as default.
Tried application:start - and completely baffled as to how to get this
to work.
2016-11-02 12:58:27 +00:00
martinsumner
ce34235f2f
Revert commented out test
...
Unintentional commenting
2016-11-01 01:38:48 +00:00
martinsumner
84a92b5f95
Further testing of compaction
...
Check we avoid crashing in challenging compaction scenarios
2016-11-01 00:46:14 +00:00
martinsumner
7d35ef7126
Lame AAE hashtree query test
...
Corruption of the values wihtin the journal doesn't get detected by the
hashtree query
2016-10-31 22:17:29 +00:00
martinsumner
73004328e1
Recovery Tests
...
Some initial entropy tests showing loss of data from a corrupted CDB
file.
2016-10-31 20:58:19 +00:00
martinsumner
b18f7fd1c1
Check presence in Journal on hashtree query
...
Basic happy day unit test added to demonstrate checking presence (with a
set probability) of the item in the hashtree query within the Journal.
2016-10-31 18:51:23 +00:00
martinsumner
7d3a04428b
Refactor snapshot
...
Better reuse snapshotting fucntions in the Bookie, and use it to support
doing Inker clone checks
2016-10-31 17:26:28 +00:00
martinsumner
bd6c44e9b0
Correct is_active
...
Firts part of adding support for scanning for Keys and Hashes. as part
of this discovered TTL support did the opposite (only fetched things in
the past!).
2016-10-31 16:02:32 +00:00
martinsumner
2607792d1f
Adjust setting
...
If cache size is too small then we're more likely to be not ready to
evict a L0 file
2016-10-31 15:18:21 +00:00
martinsumner
6b5b51412e
Improve TTL unit test
...
Add support for different type of index queries
2016-10-31 15:13:11 +00:00
martinsumner
9bef57a78d
Get Positions - when rolling
...
CT test was call get_positions hilst the sile was rolling - don't want
the file to be checked in this state, so just return an empty list.
2016-10-31 14:01:09 +00:00
martinsumner
3b05874b8a
Add initial timestamp support
...
Covered only by basic unit test at present.
2016-10-31 12:12:06 +00:00
martinsumner
4cffecf2ca
Handle gen_server:cast slowness
...
There was some unpredictable performance in tests, that was related to
the amount of time it took the sft gen_server to accept a cast whihc
passed the levelzero_cache.
The response time looked to be broadly proportional to the size of the
cache - so it appeared to be an issue with passing the large object to
the process queue.
To avoid this, the penciller now instructs the SFT gen_server to
callback to the server for each tree in the cache in turn as it is
building the list from the cache. Each of these requests should be
reltaively short, and the processing in-between should space out the
requests so the Pencille ris not blocked from answering queries when
pompting a L0 write.
2016-10-31 01:33:33 +00:00
martinsumner
311179964a
Quality review
...
Minor test fix-up and quality changes
2016-10-30 22:06:44 +00:00
martinsumner
0e6ee486f8
Make tets less pointless
...
Journla compaction test wouldn't actually cause compaction
2016-10-30 20:14:11 +00:00
martinsumner
89b5748062
Remove unnecessary clause
2016-10-30 19:49:01 +00:00
