Our likely iSCSI parameter tuning
April 30, 2011
Now that I have some idea of what the various iSCSI parameters do and control, I can think about how we might want to change them. The necessary disclaimer is that at the moment, all of this is theoretical, not tested and validated and useful.
My general belief is that our IO load is mostly reads and is somewhere between truly random and short sequential reads (ie, sequential reads of short files). I expect that most of our writes are asynchronous, but some of them are synchronous as ZFS commits transaction groups.
(I have not actually verified this belief, partly because measuring this stuff is hard. Please do not suggest DTrace as the obvious easy answer unless you also have a DTrace script that gives good answers to these questions, at scale.)
Given this, my first overall conclusion is that tuning iSCSI parameters probably isn't that important for us, provided that they are sane to start with. Bandwidth is not really an issue in general and the major latency tuning you can do is for writes, which are not that important for us. Dedicated tuning could modestly lower the protocol overhead for reads and theoretically this slightly reduces the latency, but.
This doesn't mean that there's nothing to tune, though. Here's what I think we'll want:
Since we can set InitialR2T to no, I don't think we really care about ImmediateData. If it naturally defaults to 'yes' there's no reason to change it, but if it doesn't then there's not much reason to bother changing it. The exception would be if it turns out to be harmless to set the maximum PDU size very large (because neither side does any fixed buffer allocations based on it), large enough that typical writes easily fit into a single PDU.
* * *
Atom feeds are available; see the bottom of most pages.