My NVMe versus SSD uncertainty (and hesitation)
One of the things I sometimes consider for my current home machine is adding a mirrored pair of NVMe drives (to go with my mirrored pair of SSDs and mirrored pair of HDs). So far I haven't really gotten very serious about this, though, and one of the reasons is that I'm not sure how much faster NVMe drives are going to be for me in practice and thus whether it is worth getting higher end NVMe drives or not.
(Another reason for not doing anything is that the cost per GB keeps dropping, so my inertia keeps being rewarded with more potential storage for the same potential cost.)
NVMe drives clearly have the possibility of being dramatically faster than SATA SSDs can be; their performance numbers generally cite large bandwidth, high number of IOPs, and a low latency (sometimes implicitly, sometimes explicitly). Assuming that these are relatively real, there are two obstacles to achieving them in practice; the OS driver stack and your (my) actual usage.
In order to delivery high bandwidth, many IOPs, and low latency for disk IO, the OS needs to have a highly efficient block IO system and good drivers. Extremely high-speed (and extremely parallel) 'disk' IO devices upend various traditional OS assumptions about how disk IO should be organized and optimized, and I'm not sure what the state of this is today for Linux. NVMe drives are now widely used, especially on high performance servers, so in theory things should be reasonably good here.
(Of course, one answer is that OS performance is only going to improve from here even with today's NVMe drives. But then my inertia whispers in my ear that I might as well wait for that to happen before I get some NVMe drives.)
The actual usage issue is that if you don't use your drives, how well they perform is mostly irrelevant. I have divided thoughts here. On the one hand, I don't really do much IO on my home machine these days, and much of what I do is not high priority (I may compile Firefox from source periodically, but I generally don't care if that's a bit faster or slower since I'm not actually waiting for it). On the other hand, I might well notice a significant improvement in latency even for infrequent disk IO activity, because it's activity that I'm waiting on (whether or not I realize it). Modern systems and programs go to disk for all sorts of reasons, often without making it clear to you that they are.
(In the days of loud HDs you could hear your disks working away, but SSDs are silent.)
Even given this, I do sometimes do IO intensive activities that I would really quite like to go fast. One periodic case of this is upgrading Fedora from release to release, which requires updating most of what is now over 5,000 RPM packages (with over 760,000 files and directories). This is a disk intensive process and I saw a real speed difference going from HDs to SSDs; I wouldn't mind seeing another one with a move to NVMe, assuming that NVMe would or could actually deliver one here.
PS: Since my major use of any new space will be for a ZFS pool, another question is how well ZFS itself is prepared for the performance characteristics of NVMe drives (well, ZFS on Linux). See, for example, this recent ZoL issue.