umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
923 commits 1 branch 0 tags 25 MiB
Commit graph

923 commits

This branch
This branch
All branches
Author SHA1 Message Date
Martin Sumner
2f9afa1469 Add support for performing a magic hash on a binary 
Ignore unnecessray term_to_binary if already binary.  This will be useful when we use magic_hash in tictac_trees we wish to be exportable.
 2017-09-26 16:32:59 +01:00
Martin Sumner
f50a2a19d3 File should not be pushed 2017-09-25 15:57:06 +01:00
Martin Sumner
dfab33e8da Add smaller trees 
The "small" tree will serialise to 1.5MB - which seems large.  Much smaller trees seem to be more suitable for things like recently modified aae indexes.
 2017-09-25 13:07:08 +01:00
Martin Sumner
69ed945e58 Merge pull request #90 from martinsumner/mas-riakaae-impl-2 
Mas riakaae impl 2
 2017-09-22 14:19:33 +01:00
Martin Sumner
9730816c38 Merge branch 'master' into mas-riakaae-impl-2 2017-09-22 09:39:32 +01:00
Martin Sumner
95ecfd901a Merge pull request #89 from martinsumner/mas-i83-otp16tests 
Make tests compatible with OTP 16
 2017-09-15 18:58:07 +01:00
Martin Sumner
eba21f49fa Make tests compatible with OTP 16 
this required a switch to change the sync strategy based on rebar parameter.

However tests could be slow on macbook with OTP16 and sync - so timeouts added in unit tests, and ct tests sync_startegy changed to not sync for OTP16.
 2017-09-15 15:10:04 +01:00
Martin Sumner
605a8b1e73 Merge pull request #88 from martinsumner/mas-testfailure-01 
Fix tests
 2017-09-15 10:34:50 +01:00
Martin Sumner
869e799b41 Fix tests 
Obviously got totally messed up and confused when testing previous
commits.

Multiple tests were failing for a change which got merged in as the
tests were not reflecting the required API.
 2017-09-15 10:33:16 +01:00
Martin Sumner
ff6a662bed Merge pull request #87 from martinsumner/mas-foldheads-unittest 
Unit tests use wrong query format
 2017-09-14 18:07:26 +01:00
Martin Sumner
856a64c4d4 Unit tests use wrong query format 
Not sure how this happened.  Bad merge?  Just plain sloppiness on my part?  Anyhow, the unit tests were not working ..
 2017-09-14 18:06:50 +01:00
Martin Sumner
ed56ef17a1 Make export mochijson friendly 2017-09-13 23:45:48 +01:00
Martin Sumner
d8ca0274f6 Export the import 2017-09-13 22:02:44 +01:00
Martin Sumner
d98534b8f3 Add struct to help with mochijson2 
Try and make this more mochijson2 friendly
 2017-09-13 21:56:28 +01:00
Martin Sumner
9f97c82d0d Add import/export support 
Also fix for fold_heads unit tests to reflect new booleans required by changes to support there use in MapFolds
 2017-08-16 16:58:38 +01:00
Martin Sumner
9aaa144f7a Major mixup! 
Got my leveled and leveldb results the wrong way round.
 2017-08-11 11:13:43 +01:00
Martin Sumner
5f3d44ffca Merge pull request #86 from martinsumner/mas-aae-riakimplnotes 
Mas aae riakimplnotes
 2017-08-10 17:16:40 +01:00
Martin Sumner
bfb77eacf7 Clarify on AAE locks 2017-08-10 17:15:27 +01:00
Martin Sumner
e91a3034df Update volume chart 
Looks better this way
 2017-08-10 16:46:56 +01:00
Martin Sumner
107c28b385 Add comparison of just adding AAE 
Provide some actual measurement of the cost of turning on active AAE.  It is higher than I expected!
 2017-08-10 16:41:32 +01:00
Martin Sumner
53ddc8950b Add tests using fold_heads 
Comparing the inbuilt tictac_tree fold, to using "proper" abstraction and achieving the same thing through fold_heads.

The fold_heads method is slower (a lot more manipulation required in the fold) - expect it to require > 2 x CPU.

However, this does give the flexibility to change the hash algorithm.  This would allow for a fold over a database of AAE trees (where the hash has been pre-computed using sha) to be compared with a fold over a database of leveled backends.

Also can vary whether the fold_heads checks for presence of the object in the Inker.  So normally we can get the speed advantage of not checking the Journal for presence, but periodically we can.
 2017-08-07 10:45:41 +01:00
Martin Sumner
dd20132892 Add test with fold_heads 
Build the AAE tree equally using fold_heads.  This is a pre-cursor to running this within Riak.

In part this leans on some of the work done to improve standard Riak AAE with leveled.  When rebuilding the standard AAE store only the head is required, and so this process was switched in riak_kv_sweeper to make a fold_heads request if supported by the backend.

The head response is a proxy object, which when loaded into a riak_object will allow for access to object metadata, but will use the passed function if access to object contents is requested.
 2017-08-05 16:43:03 +01:00
Martin Sumner
22709abcd6 Buit what if build initiated during fold 2017-08-04 18:35:16 +01:00
Martin Sumner
bea40ffc72 More info on hashtree locks 2017-08-04 18:30:29 +01:00
Martin Sumner
76692ae589 The problem of the lock 
Still need to investigate behaviour under get_lock()
 2017-08-04 16:30:25 +01:00
Martin Sumner
b99539fc8a Add most recent thoughts on implementation 
Especially how to manage alternate backends to leveled
 2017-08-04 16:19:40 +01:00
Martin Sumner
ff5e85700d Update gitignore 2017-08-04 15:32:25 +01:00
Martin Sumner
f9b61fb5e7 Minor edit 2017-08-01 14:01:41 +01:00
Martin Sumner
82a1619846 Mini edit 2017-08-01 13:25:50 +01:00
Martin Sumner
a046666e4a Add further stats 2017-08-01 13:23:10 +01:00
Martin Sumner
676bf12dc3 Add figures on rate of list_keys operations 2017-08-01 13:16:49 +01:00
Martin Sumner
6107ac1748 Add response time comparison for listkeys 2017-08-01 13:04:05 +01:00
Martin Sumner
66d85efe62 Switch back to pure markdown 
But image still appears to have lost clarity
 2017-08-01 12:57:10 +01:00
Martin Sumner
48fd2c76f0 Use screenshot for clarity 
Detail lost when saving image direct from google sheets
 2017-08-01 12:55:09 +01:00
Martin Sumner
b9139b3116 Sizing experiment 2017-08-01 12:50:53 +01:00
Martin Sumner
43547cff41 Experiment with image insertion and sizing 2017-08-01 12:50:02 +01:00
Martin Sumner
a8c3403a9d Experiment with image sizing 2017-08-01 12:47:52 +01:00
Martin Sumner
89b5c43fd3 Add volume chart 2017-08-01 12:45:01 +01:00
Martin Sumner
a85c853fe0 Merge branch 'master' into mas-aae-riakimplnotes 2017-08-01 12:28:37 +01:00
Martin Sumner
714adf279a Merge pull request #81 from project-fifo/random 
Add compatibility for old and new random / rand functions
 2017-08-01 12:24:23 +01:00
Martin Sumner
968161a36d Initial start to write-up 
Charts etc still to add
 2017-08-01 11:19:21 +01:00
Heinz N. Gies
b319386210 Fix strong_rand to rand_bytes 2017-08-01 11:55:05 +02:00
Heinz N. Gies
bbe763514b Remove uniform_s/2 from old random code 2017-08-01 11:37:18 +02:00
Heinz N. Gies
38e9b0e80a Add missing uniform/0 function 2017-08-01 11:24:12 +02:00
Heinz N. Gies
25389893cf Add compatibility for old and new random / rand functions 2017-08-01 11:24:12 +02:00
Martin Sumner
a49d3bbfec Merge pull request #84 from martinsumner/mas-aaeexpiry-i82 
Add extra second to sleep
 2017-08-01 00:14:48 +01:00
Martin Sumner
8748fef28c Add extra second to sleep 
Sleep for just one more second to resolve intermittent failure
 2017-08-01 00:14:31 +01:00
Martin Sumner
7f6f51efcd Merge pull request #80 from project-fifo/dialyzer 
Fix dialyzer warnings for newer erlang releases
 2017-07-31 21:42:50 +01:00
Heinz N. Gies
379e33ba84 Cleanup dialyzer errrors in leveled_bookie 2017-07-31 19:58:56 +02:00
Heinz N. Gies
8717d42ffe Cleanup dialyzer errrors in leveled_cdb 2017-07-31 19:55:09 +02:00