Add more volume test details

docs/FUTURE.md

@@ -55,7 +55,7 @@ The leveled backend has been implemented with some basic manual functional tests
 
 All standard features should (in theory) be supportable (e.g. TTL, M/R, term_regex etc) but aren't subject to end-to_end testing at present.
 
-There is a uses_r_object capability that leveled should in theory be able to support (making it unnecessary to convert to/from binary form).  However, given that only the yessir backend currently supports this capability, this hasn't been implemented in case there are potential issues in Riak with this capability.
+There is a uses_r_object capability that leveled should in theory be able to support (making it unnecessary to convert to/from binary form).  However, given that only the yessir backend currently supports this, the feature hasn't been implemented in case there are potential issues lurking within Riak when using this capability.
 
 ### Fast List-Buckets
 
@@ -90,6 +90,6 @@ So rather than doing three Key/Metadata/Body backend lookups for every request,
 - It is more likely to detect out-of-date objects in slow nodes (as the n-r responses may still be received and added to the result set when waiting for the GET response in the second loop).
 - The database is in-theory much less likely to have [TCP Incast](http://www.snookles.com/slf-blog/2012/01/05/tcp-incast-what-is-it/) issues, hence reducing the cost of network infrastructure, and reducing the risk of running Riak in shared infrastructure environments.
 
-The feature will not at present work with legacy vclocks. 
+The feature will not at present work safely with legacy vclocks. This branch generally relies on vector clock comparison only for equality checking, and removes some of the relatively expensive whole body equality tests (either as a result of set:from_list/1 or riak_object:equal/2), which are believed to be a legacy of issues with pre-dvv clocks.
 
 In tests, the benefit of this may not be that significant - as the primary resource saved is disk/network, and so if these are not the resources under pressure, the gain may not be significant.  In tests bound by CPU not disks, only a 10% improvement has so far been measured with this feature.

docs/VOLUME.md

@@ -14,7 +14,7 @@ The First test on a Riak Cluster has been based on the following configuration:
 - Using i2.2xlarge EC2 nodes with mirrored SSD drives (for data partition only)
 - noop scheduler, transparent huge pages disabled, ext4 partition
 - A 64 vnode ring-size
-- 45 concurrent basho_bench threads (basho_bench run on separate disks) running at max
+- 45 concurrent basho_bench threads (basho_bench run on a separate machine) running at max
 - AAE set to passive
 - sync writes enabled (on both backends)
 - An object size of 8KB
@@ -72,9 +72,54 @@ This indicates that without sync_on_writes, there remains potential value in usi
 
 ## Riak Cluster Test - 3
 
-to be completed ..
+Testing with optimised GET FSM (which checks HEAD requests from r nodes, and only GET request from one node if no sibling resolution required).
 
-Testing with optimised GET FSM (which checks HEAD requests from r nodes, and only GET request from one node if no sibling resolution required)
+The test has been based on the following configuration:
+
+- A 5 node cluster
+- Using i2.2xlarge EC2 nodes with mirrored SSD drives (for data partition only)
+- deadline scheduler, transparent huge pages disabled, ext4 partition
+- A 64 vnode ring-size
+- 100 concurrent basho_bench threads (basho_bench run on a separate machine) running at max
+- AAE set to passive
+- sync writes disabled (on both backends)
+- An object size of 8KB
+- A pareto distribution of requests with a keyspace of 50M keys
+- 5 GETs for each UPDATE
+- 6 hour test run
+
+Note the changes from the first cluster test: 
+
+- The recommended disk scheduler was used (the change was mistakenly omitted in the previous test).
+- Sync_writes were disabled (for comparison purposes with the previous test, as Basho have stated that they don't expect customers to use this feature - although some customers do use this feature).
+- More basho_bench threads were used to push the extra capacity created by removing the syncing to disk.
+- The test run was extended to six hours to confirm a change in the divergence between leveled and leveldb which occurred as the test progressed.
+
+This test showed a <b>12.5%</b> improvement in throughput when using LevelEd, across the whole test.  The difference between the backends expands as the test progresses though, so in the last hour of the test there was a <b>29.5%</b> improvement in throughput when using LevelEd. 
+
+There was again a significant improvement in variance in tail latency, although not as large a difference as seen when testing with writing to disks enabled.  Through the course of the test the average of the maximum response times (in each 10s period) were
+
+leveled GET mean(max)           | eleveldb GET mean(max)
+:-------------------------:|:-------------------------:
+250.3ms | 1,090.1ms
+
+leveled PUT mean(max)           | eleveldb PUT mean(max)
+:-------------------------:|:-------------------------:
+369.0ms | 1,817.3ms
+
+The difference in volatility is clear when graphing the test performance:
+
+leveled Results           |  eleveldb Results
+:-------------------------:|:-------------------------:
+![](../test/volume/cluster_three/output/summary_leveled_5n_100t_nosync.png "LevelEd")  |  ![](../test/volume/cluster_three/output/summary_leveldb_5n_100t_nosync.png "LevelDB")
+
+### Lies, damned lies etc
+
+Monitoring the resource utilisation between the two tests makes the difference between the two backends clearer.  Throughout the test run with the leveled backend, each node is almost exclusively constrained by CPU, with disk utilisation staying well within limits.  
+
+The leveldb backend can achieve higher throughput when it is constrained by CPU compared to leveled, but as the test runs it becomes constrained by disk busyness with increasing frequency.  When leveldb is constrained by disk throughput can still become significantly constrained even without disk syncing enabled (i.e. throughput can halve/double in adjoining 20s measurement periods).
+
+Leveled is consistently constrained by one factor, giving more predicable throughput. Leveldb performs well when constrained by CPU availability, but is much more demanding on disk, and when disk becomes the constraint throughput becomes volatile.
 
 ## Riak Cluster Test - 4