umgeher/leveled
0
Fork 0
Code Issues Pull requests Releases Activity Actions
1471 commits 1 branch 0 tags 25 MiB
Commit graph

278 commits

Author SHA1 Message Date
Martin Sumner
9badc8fbe7 Merge branch 'master' into mas-sweeperfold-i59 2017-04-10 21:49:08 +01:00
martinsumner
60b0b88226 Switch to binary response 
Force header in fold_heads to be a binary ... as this is what KV expects
 2017-04-07 17:08:40 +00:00
martinsumner
6d42abbc1a Add bybucket to fold_heads 
give fold_heads equivalent functionality to fold_objects - both can now
be done for allkeys and bybucket
 2017-04-07 14:56:28 +00:00
martinsumner
9375baf636 Add unit test for foldheads 
compare foldheads foldobjects and hahstree_query output
 2017-04-07 14:19:25 +00:00
martinsumner
b464e2e28c Extend foldobjects to support proxy object 
To allow for folds which probbaly don't need values to not always to
have to fetch the value
 2017-04-07 12:09:11 +00:00
Martin Sumner
4e9fa2a206 Timeout long-running snapshots 
Add logic to timeout long-running snapshots.
 2017-04-05 09:16:01 +01:00
Martin Sumner
8db73917fb Need also to remove unused bits 2017-03-22 00:14:37 +00:00
Martin Sumner
15af4942ae Remove busy log 
Accounts for 60% of logs
 2017-03-22 00:11:17 +00:00
martinsumner
682dfc4d59 Revert "Revert "ETS - delete table not objects"" 
This reverts commit c46377584f.
 2017-03-21 12:02:22 +00:00
martinsumner
c46377584f Revert "ETS - delete table not objects" 
This reverts commit 7dc4913d5a.
 2017-03-21 01:32:41 +00:00
martinsumner
7dc4913d5a ETS - delete table not objects 
Try and delete the table not just the objects in the table - will this
improve memory leak?
 2017-03-20 22:43:22 +00:00
martinsumner
62c3ba8b6f Passing ETS reference not tree 
ETS reference gets converted to tree by Penciller
 2017-03-13 11:54:46 +00:00
martinsumner
9ad6969b0d Seed randomnes at Actor startup 2017-03-06 21:35:02 +00:00
martinsumner
c92107e4b4 2i order of events 
When running a load of mainly 2i queries, there is a huge cost in the
previous snapshot code.  The time taken to create a clone of the
Penciller (duplicating all the LoopState) varied between 1 and 200ms
depedning on the size of the LoopState.

For 2i queries, most of that LoopState was then being thrown away after
running the query against the levelzero_cache.  This was taking < 1ms on
average.  It would be better to avoid the o(100)ms of CPU  burning and
block for o(1)ms - so th eorder of events have been changed ot filter
first so only the small part of the LoopState actually required is
copied to the clone.
 2017-03-06 18:42:32 +00:00
martinsumner
eb6f668fcd Use log at random 
Easy way to sample frequent things - especially when they'r ein ocverag
equeries
 2017-03-06 10:34:56 +00:00
martinsumner
94a39e4c4d Fix double logging of B0004 2017-03-06 10:27:48 +00:00
martinsumner
5c2f05858d Alter logging to help understand performance factors 
Change logging of the snapshots to better understand performance
 2017-03-06 10:17:51 +00:00
martinsumner
426f5df75c Fix dialyzer warnings 2017-03-02 21:24:40 +00:00
martinsumner
30e23fe514 Add no_lookup feature to snapshots 2017-03-02 18:23:47 +00:00
martinsumner
b01f7d23df Add pre-filter for 2i queries 2017-03-02 17:49:43 +00:00
martinsumner
35b6e16590 Clear dialyzer warnings 2017-02-27 20:23:36 +00:00
martinsumner
fce7cbd2ef Add sync_strategy comments 2017-02-27 17:32:38 +00:00
martinsumner
2db28c3028 Tidy line length 2017-02-27 12:24:26 +00:00
martinsumner
8d3f4bd4dc Comment API on Bookie 
Make it easier to understand how to work the store
 2017-02-27 11:58:16 +00:00
martinsumner
aba385869f Test long running clause 2017-02-26 22:41:25 +00:00
martinsumner
ed96d0ca7a Change so that type of cache is parameterised 
The Tree doesn't seem to be better than the skiplist.  Get ready to make
this switchable
 2017-01-21 11:38:26 +00:00
martinsumner
3d99036093 Switch the LM1 cache to be a tree 
Use a tree of lists not a skiplist
 2017-01-20 16:36:20 +00:00
martinsumner
a617f8fb66 Fix snapshot_store - add index to clone 
Clone was not getting the updated index
 2017-01-05 22:17:30 +00:00
martinsumner
5a88565c08 Switch to binary index in pmem 
Remove the ets index in pmem and use a binary index instead.  This may
be slower, but avoids the bulk upload to ets, and means that matches
know of position (so only skiplists with a match need be tried).

Also stops the discrepancy between snapshots and non-snapshots - as
previously the snapshots were always slowed by not having access to the
ETS table.
 2017-01-05 21:58:33 +00:00
martinsumner
1d3fb18df7 Resolve snapshotting issue 
Need to make sure the extract from ets happens at the point the snapshot
is taken.
 2017-01-05 18:43:55 +00:00
martinsumner
2c828b8eca Fix snapshot issue 2017-01-05 17:55:27 +00:00
martinsumner
e6270d288f Half-way to ets for Bookie mem 
A half-way implementation with use of ETS as the bookie's memory
 2017-01-05 17:00:12 +00:00
martinsumner
dc28388c76 Removed SFT 
Now moved over to SST on this branch
 2016-12-29 02:07:14 +00:00
Martin Sumner
676e8fa494 Add Get Timing 2016-12-22 15:45:38 +00:00
Martin Sumner
7a0cf22909 put-timing default 
Remove need for individual actors to know the defaults for put_timing
tuple
 2016-12-22 14:41:43 +00:00
martinsumner
130fb36ddd Add head timings 
Include log breaking down timings of HEAD requests by result and level
 2016-12-22 14:03:31 +00:00
martinsumner
f3e16dcd10 Add long-running logs 2016-12-21 01:56:12 +00:00
martinsumner
060ce2e263 Add put timing points 2016-12-20 23:11:50 +00:00
Martin Sumner
5efed94a1e Try slightly larger cache 2016-12-13 22:29:55 +00:00
martinsumner
ccc993383d Stop second hash on fetch_head 
The bookie should magic_hash for fetch_head, and now passes the hash to
the Penciller so second hash not required.
 2016-12-11 01:21:53 +00:00
martinsumner
2d3a40e6f1 Magic Hash - and no L0 Index 
Move to using the DJ Bernstein Magic Hash consistently, and trying to
make sure we only hash once for each operation (as the hash is more
expensive than phash2).

The improved lookup time for missing keys should allow for the L0 index
to be removed, and hence speed up the completion time for push_mem
operations.

It is expected there will be a second stage of creating a tinybloom as
part of the SFT creation process, and then adding that tinybloom to the
manifest.  This will then reduce the message passing required for a GET
not in the cache or higher levels
 2016-12-11 01:02:56 +00:00
martinsumner
626a8e63f9 Experiment converting CDB to use skiplist not gb_tree 
Might insertion time be faster?
 2016-12-10 10:55:35 +00:00
martinsumner
a3f60e3609 OTP version shenanigans 2016-12-09 18:55:13 +00:00
martinsumner
d2bd01eaf1 Add fast fail to skiplist 
Add a bloom filter to the skiplist, to make it faster at returning not
found.  The SkipList is now encapsulated within a dict().
 2016-12-09 18:30:40 +00:00
martinsumner
f0db730f07 Adjust jitter settings 2016-12-09 16:34:15 +00:00
martinsumner
82cb49638a Attempt at performance improvement 
Try to add some extra jitter in to the process of L0 writes, and also
make L0 writes delayed to help with bufferring
 2016-12-09 14:36:03 +00:00
martinsumner
349d194a7c Increase jitter slightly 2016-12-09 09:52:31 +00:00
martinsumner
2f4013f430 Set Jitter correctly this time 2016-12-08 21:02:39 +00:00
martinsumner
f07ab85a81 Oops 2016-12-08 18:36:57 +00:00
martinsumner
34e7ce170a Jitter co-ordination issues 
Experiment to try and reoslve jitter co-orsination issue
 2016-12-08 18:35:20 +00:00