Background ^
In the previous article I covered how, in order to get decent performance out of lvmcache with a packaged Debian kernel, you’d have to use the 4.12.2-1~exp1 kernel from experimental. The kernels packaged in sid, testing (buster) and stable (stretch) aren’t new enough.
I decided to bisect the Linux kernel upstream git repository to find out exactly which commit(s) fixed things.
Results ^
Here’s a graph showing the IOPS over time for baseline SSD and lvmcache with a full cache under several different kernel versions. As in previous articles, the lines are actually Bezier curves fitted to the data which is scattered all over the place from 500ms averages.
What we can see here is that performance starts to improve with commit 4d44ec5ab751 authored by Joe Thornber:
dm cache policy smq: put newly promoted entries at the top of the multiqueue
This stops entries bouncing in and out of the cache quickly.
This is part of a set of commits authored by Joe Thornber on the
The first of Joe Thornber’s commits on that day in the dm-cache area was 072792dcdfc8 and stepping back to the commit immediately prior to that one (2ea659a9ef48) we get a kernel representing the moment that Linus designated the v4.12-rc1 tag. Joe’s commits went into -rc1, and without them the performance of lvmcache under these test conditions isn’t much better than baseline HDD.
It seems like some of Joe’s changes helped a lot and then the last one really provided the long term performance.
git bisect procedure ^
Normally when you do a git bisect you’re starting with something that works and you’re looking for the commit that introduced a bug. In this case I was starting off with a known-good state and was interested in which commit(s) got me there. The normal bisect key words of “good” and “bad” in this case would be backwards to what I wanted. Dominic gave me the tip that I could alias the terms in order to reduce my confusion:
$ git bisect start --term-old broken --term-new fixed |
From here on, when I encountered a test run that produced poor results I would issue:
$ git bisect broken |
and when I had a test run with good results I would issue:
$ git bisect fixed |
As I knew that the tag v4.13-rc1 produced a good run and v4.11 was bad, I could start off with:
$ git bisect reset v4.13-rc1 $ git bisect fixed $ git bisect broken v4.11 |
git would then keep bisecting the search space of commits until I would find the one(s) that resulted in the high performance I was looking for.
Good and bad? ^
As before I’m using fio to conduct the testing, with the same job specification:
ioengine=libaio
direct=1
iodepth=8
numjobs=2
readwrite=randread
random_distribution=zipf:1.2
bs=4k
size=2g
unlink=1
runtime=15m
time_based=1
per_job_logs=1
log_avg_msec=500
write_iops_log=/var/tmp/fio-${FIOTEST} |
The only difference from the other articles was that the run time was reduced to 15 minutes as all of the interesting behaviour happened within the first 11 minutes.
To recap, this fio job specification lays out two 2GiB files of random data and then starts two processes that perform 4kiB-sized reads against the files. Direct IO is used, in order to bypass the page cache.
A Zipfian distribution with a factor of 1.2 is used; this gives a 90/10 split where about 90% of the reads should come from about 10% of the data. The purpose of this is to simulate the hot spots of popular data that occur in real life. If the access pattern were to be perfectly and uniformly random then caching would not be effective.
In previous tests we had observed that dramatically different performance would be seen on the first run against an empty cache device compared to all other subsequent runs against what would be a full cache device. In the tests using kernels with the fix present the IOPS achieved would converge towards baseline SSD performance, whereas in kernels without the fix the performance would remain down near the level of baseline HDD. Therefore the fio tests were carried out twice.
Where to next? ^
I think I am going to see what happens when the cache device is pretty small in comparison to the working data.
All of the tests so far have used a 4GiB cache with 4GiB of data, so if everything got promoted it would entirely fit in cache. Not only that but the Zipf distribution makes most of the hits come from 10% of the data, so it’s actually just ~400MiB of hot data. I think it would be interesting to see what happens when the hot 10% is bigger than the cache device.
git bisect progress and test output ^
Unless you are particularly interested in the fio output and why I considered each one to be either fixed or broken, you probably want to stop reading now.