2010-10-15
How big our fileserver environment is (part 2)
Yesterday I counted up (or ran down) the hardware side of how big our fileserver environment is. Today I'll cover the amount of disk space we have.
Our raw disk space is simple to compute, impressive sounding, and highly misleading. With four backends of 750 GB disks and five backends of 1 TB disks, we have 96 TB (of disk vendor terabytes) of raw space available, or 60 TB for the production backends only. However, all of our space is mirrored, so that is really 30 TB (actually slightly less because we have a 750 GB four-way mirror), and then you take out a disk from each backend for spares, and the resulting ~27 TB number is still misleading.
How we really think about disk space is in terms of 'chunks', by which we generally mean mirrored pairs of our standard-sized LUNs, roughly 250 GB each; there are three chunks on a 750 GB disk (or a pair of them) and four on a 1 TB disk. At the ZFS pool level, all of our space is handled in terms of chunks, as they are what we add to pools and so on.
After the dust settles, we have:
- 110 chunks total across all of the production backends; after various overheads, this would be about 25 TB of space as ZFS sees it.
- 70 chunks allocated to our various ZFS pools, for a total of 15.6 TB of space as ZFS sees it.
- 9.3 TB of space actually used by data (as counted by ZFS).
So we have a bit over half our potential space actually allocated, and a bit over half of our allocated space actually being used.
The allocation and space usage is spread very unevenly among our ZFS pools. The smallest size pool is one chunk and the largest is 12 chunks; the least used pool has only 27.5 GB used, the most used pool has 1.8 TB used (not coincidentally, it is also the 12-chunk pool).
Pools and filesystems: we have 25 ZFS pools in total, with 174 different filesystems between them (we have more NFS mounts than ZFS-level filesystems for reasons that don't fit in the margins of this entry). We have so many filesystems for complicated reasons, but part of it is how we've chosen to structure and organize filesystems here. ZFS makes filesystems cheap, so this is fine.
(The hot spare backend has no space allocated or used on it. The test fileserver and backends have a fluctuating amount of their space allocated and used, but how much is irrelevant.)