Good article, Chris Ramseyer.
"Real-world software rarely pushes fast storage devices to their limits"
Chris, I am curious about what all holds back NVMe SSD's from getting their full potential? What all needs to come together to reach their full potential? Will Kaby Lake and the new 200-Series Chipset Union Point motherboards help to get better performance out of the new NVMe & Optane SSD's? I've heard we need a far bigger BUS too. I am holding out for an NVMe SSD that will actually reach the claimed 32 Gb/s or close to it - minus overhead.
This is my semi-educated guess.
1) The storage chips need to be faster, but they are pretty fast.
2) Controllers need to be faster. Less complicated overhead, better command concurrency, etc
3) There is a latency vs throughput issue. If most programs are making one request at a time and waiting for the response for that request, then you need really low latency to have high bandwidth. On the other hand, if a program makes many concurrent requests, then it just multiplied its theoretical peak bandwidth.
Similar issue with why TCP has a transmit window. Waiting for a response over high latency slows you down. The main difference is TCP pushes data. Reading from the harddrive pulls data.