2i query test results

Write-up of 2i query test results

README.md

@@ -53,6 +53,7 @@ Test Description                  | Hardware     | Duration |Avg TPS    | TPS De
 8KB value, 50 workers, no_sync    | 5 x d2.2x    | 4 hr     | 10,400.29 | <b>+  8.37%</b>  | <b>+ 23.51%</b>
 4KB value, 100 workers, no_sync   | 5 x i2.2x    | 6 hr     | 14,993.95 | - 10.44%  | - 4.48%
 16KB value, 60 workers, no_sync   | 5 x i2.2x    | 6 hr     | 11,167.44 | <b>+ 80.48%</b>  | <b>+ 113.55%</b>
+8KB value, 80, workers, no_sync, 1% 2i | 5 x i2.2x | 6 hr | 9,855.96 | <b>+ 4.48%</b> | <b>+ 22.36%</b>
 
 Tests generally show a 5:1 improvement in tail latency for leveled.
 

docs/VOLUME.md

@@ -155,9 +155,46 @@ Testing during a journal compaction window
 
 ## Riak Cluster Test - Phase 4
 
-to be completed ..
+Testing with secondary indexes provides some new challenges:
+
+- Including index entries on updates accelerates the rate of compaction of the merge tree, as each update into the merge tree is now multiple changes;
+- The 2i query needs an efficient snapshot to be taken of the store, and multiple such snapshots to exist in parallel;
+- The coverage nature of the query means that the query operation is now as slow as the slowest of the vnodes (out of 22 nodes with a ring size of 64), rather than being as slow as the second slowest of three vnodes.  Coverage queries are reactive to local resource pressure, whereas standard GET flows can work around such pressure.
+
+The secondary index test was built on a test which sent
+
+- 100 GET requests;
+- 10 update requests (i.e. GET then PUT) for 8KB fixed sized objects;
+- 10 update requests (i.e. GET then PUT) for 8KB fixed sized objects with four index entries each (leading to 4 index changes for a new request, 8 index changes for an update request);
+- 1 2i range query on the "date of birth" index with return_terms enabled;
+- 1 2i range query on the "postcode index" with return_terms enabled.
+
+The query load is relatively light compared to GET/PUT load in-line with Basho recommendations (decline from 350 queries per second to 120 queries per second through the test).  The queries
+return o(1000) results maximum towards the tail of the test and o(1) results at the start of the test.
+
+Riak + leveled           |  Riak + leveldb
+:-------------------------:|:-------------------------:
+![](../test/volume/cluster_2i/output/summary_leveled_5n_80t_i2_nosync_2i.png "LevelEd")  |  ![](../test/volume/cluster_2i/output/summary_leveldb_5n_80t_i2_nosync_2i.png "LevelDB")
+
+The results are similar as to previous tests.  Although the test is on infrastructure with optimised disk throughput (and with no flushing to disk on write from Riak to minimise direct pressure from Riak), when running the tests with leveldb disk busyness rapidly becomes a constraining factor - and the reaction to that is volatility in throughput.  Riak combined with leveldb is capable in short bursts of greater throughput than Riak + leveled, however when throttled within the cluster by a node or nodes with busy disks, the reaction is extreme.
+
+The average throughputs hour by hour are:
+
+Hour      |Riak + leveled             |  Riak + leveldb           | Delta
+:--------|-------------------------:|-------------------------:|--------
+1 | 14,465.64 | 16,235.83 | - 10.90%
+2 | 10,600.16 | 11,813.14 | - 10.27%
+3 |  8,605.89 |  7,895.84 | <b>+ 18.63%</b>
+4 |  9,366,19 |  7,225.25 | <b>+ 19.11%</b>
+5 |  8,180.88 |  6,826.37 | <b>+ 19.84%</b>
+6 |  7,916.19 |  6,469.73 | <b>+ 22.36%</b>
+
+The point of inflection when leveled-backed Riak begins to out-perform leveledb-backed Riak is related to the total size of the database going beyond the size of the in-memory cache.  The need to go to disk for reads as well as writes amplifies the disk busyness problem associated with write amplification.
+
+As before, increasing the available CPU will increase the potential throughput of a leveled-backed cluster.  Increasing the disk IOPS rate will increase the potential throughput of a leveldb-backed cluster.  Increasing object sizes, will extend the throughput advantage of leveled and decreasing sizes will improve the performance of leveldb.  Increasing the size of the database relative to cache may increase the throughput advantage of leveled - but this may not hold true for all work profiles (for example those frequently accessing recently updated objects).
+
+In the 2i query test there is the same significant reduction in tail latency when comparing leveled with leveldb.  For the 2i queries there is also a significant reduction in mean latency - a <b> 57.96%</b> reduction over the course of the test.  However this reduction is not a sign that leveled can resolve 2i queries faster than leveldb - it is related to the problem of tail latency.  The minimum response time for a 2i query in leveldb drifts from 4ms to 14ms as the test progresses - whereas the minimum response time for Riak and leveled fluctuates from 5ms through to 20ms at the end of the test.  Outside of resource pressure leveldb will give a faster response - but resource pressure is much more likely with leveldb.
 
-Testing for load including 2i queries
 
 ## Riak Cluster Test - Phase 5