Errata for RAID gone Green
I noted in RAID gone Green that my read speed was lower than write speeds and provided some commentary on why that might be. I have a further explanation to add to this – the stripe size of the RAID array.
When I was initially setting up the volumes on the array I was following advice I found online (on the basis I was experimenting and could change it later without harm) to set the stripe size to be the same or close to the block size used in VMFS. Since I was targeting 2TB VMFS volumes and that needed an 8MB block, I opted for the maximum that the Dell PERC supported – 1MB. In hindsight, this may not be the optimum setting.
The default read-ahead size for Linux is something like 512 sectors» which means 256KB. With a stripe size of four times this a noteworthy result is that most of the time we will be reading from only one device. Consequently the performance we will observe will generally look like that of a single device, all the time – we won’t see any benefit from the stripe of the array.
When I tweaked the read-ahead, I started at 4096 sectors and ended up at 16384 and each time the read performance increased almost proportionately. How much does a read-ahead of 16384 equate to? 8MB. With a stripe of 1MB, how many disks does that read-ahead span? Eight. How many disks in my array? Eight. In effect, I was forcing it to read the data from all eight drives in parallel with a single I/O operation. I have not tried it but I suspect the linear gains will end at this value and any further improvements would be marginal at best. Another interesting thought is that if these drives reduce their spin rate aggressively, this also forces all of the drives up to speed a little earlier than it would otherwise and keeps them busy the entire time.
This is not the only factor at work and more tuning may be necessary later. In particular, I doubt I will keep a 1MB stripe size – I believe it will become a hindrance once my workload on this machine becomes much more random in nature, and I suspect an 8MB read-ahead in this particular virtual-machine will have similar problems. For the sake of testing the raw transfer ability of the drives I think it was a worthwhile setup, however, and it does feel very zippy with interactive use.