Something that has been confusing a number of people is the performance of Intel’s 3D XPoint-based SSDs. Why are they so slow?
Let me back up a little – they’re not really slow. When Intel compared its standard NAND flash based PCIe SSD to a similar SSD based on 3D XPoint memory, the XPoint model ran 7-8 times faster, which is very impressive. Intel demonstrated that at the Intel Developer Forum (IDF) last August and several times since then.
But Intel and Micron have been boasting since its introduction that 3D XPoint Memory is 1,000 times as fast as NAND flash. How do you get from a 1,000 times speed advantage down to a speed improvement of only 7-8 times?
That’s what the graphic in this post will explain. The small rendition above is just a portion of the larger chart shown below. I’ll explain what this graph tells us, and then will show how that 1,000 times speed advantage shrinks to only 7-8 times in an SSD.
The Storage Networking Industry Association (SNIA) uses this graph to illustrate the sources of latencies (delays) between an SSD and the software that requested the SSD’s data. I have removed a number of other bars that represent HDDs and SATA SSDs, leaving in only the two that represent the Intel NAND SSD and a similar SSD based on “Future NVM”, a broad base of possibilities, one of which would be 3D XPoint Memory.
The X-axis tells us how long it takes to perform a typical data read for both of these SSDs: about 90 microseconds for the NAND-based SSD and about 20 microseconds for the NVM SSD.
Each bar is broken into six components:
- NVM Tread, which probably should have been written tREAD. This is the “read” time of the NAND flash or NVM: How long does it take, from the time that an address is applied to the chip, for valid data to appear at the output of the NAND flash or NVM chip?
- NVM xfer: The time required to transfer (“xfer”) the data from the NAND flash or NVM to the SSD’s I/O pins.
- Misc SSD: This is to account for delays that stem from internal SSD housekeeping like address translation or the average time lost to garbage collection.
- Link Xfer: The time needed to transfer the data across the PCIe interface.
- Platform + adapter: Delays from the host computer’s PCIe port to the arrival of the data at the processor pins.
- Software: The amount of time the operating system’s I/O stack requires to perform a disk read operation
The first three components are significantly smaller for the NVM-based SSD than they are for the NAND-based SSD. The last three components don’t change at all. Those last three components are the source of the problem.
Those last three components, link transfer (xfer), platform and adapter, and software, account for about 15 microseconds of delay. If you were to use a magical memory that had zero delays, then its bar would never get any smaller than 15 microseconds.
In comparison, the upper bar, the one representing the NAND-based SSD, has combined latencies of about 90 microseconds, or six times as long.
These charts are imprecise, and that imprecision probably accounts for the difference between a 6-times speed improvement and the actual XPoint SSD’s speed of 7-8 times that of its NAND-based counterpart.
Since so much of 3D XPoint’s speed advantage is lost to these delays The SSD Guy expects for the PCIe interface to contribute very little to long-term 3D XPoint Memory revenues. Intel plans to offer another implementation, shipping 3D XPoint Memory on DDR4 DIMMs. A DIMM version will have much smaller delays of this sort, allowing 3D XPoint memory to provide a much more significant speed advantage. Over the product’s lifetime DIMMs should make up the majority of 3D XPoint Memory’s revenues.
My company, Objective Analysis, has written a report explaining 3D XPoint Memory and its market potential. Anyone interested in understanding the market for this product is welcome to click the following link to learn more: A Close Look at the Micron/Intel 3D XPoint Memory
So 3D Xpoint SSD does not offer any speed improvement over NAND SSD?
and the 7-8 times improvement is base on using NVMe technology?
so if we compere a DDR4-DIMM using NAND and a DDR4-DIMM using 3D Xpoint, which woukd be faster?
Thanks.
FBI Fido,
I may have been unclear. The 7-8X speed advantage is a comparison of Intel’s fastest NVMe/PCIe NAND flash SSD, and an NVMe/PCIe 3D XPoint SSD.
PCIe is significantly slower than the memory bus (DDR4) but PCIe is fast enough for NAND flash. When you move to DDR4 the change doesn’t improve NAND flash performance much, but it will benefit 3D XPoint tremendously.
Which is faster, & at what speed?
1) DDR4-DIMM RAM
2) DDR4-DIMM NAND SSD
3) DDR4-DIMM 3D Xpoint MEM & SSD
4) NVMe 3D Xpoint SSD
5) NVMe NAND SSD
6) M.2 NAND SSD (Samsung 2TB read-3.5GB/s and write-2.1GB/s)
thanks.
FBI Fido, thanks for the comment.
Please note that there are two m.2 interfaces: SATA and PCIe/NVMe. If your #6 is an NVMe m.2 SSD then its speed will be the same as #5.
I will assume that your #3 was meant to simply be XPoint on a DIMM, since the SSD part doesn’t make sense to me.
I’ll also assume that your #2 is an NVDIMM-F (like Diablo’s Memory Channel Storage) rather than an NVDIMM-N (from AgigA, Micron, Netlist, SMART, or Viking). NVDIMM-N is just as fast as DRAM. NVDIMM-F isn’t much faster than an SSD.
Ranking from fastest to slowest, then, you would have 1<3<2<4<5=6.
With the proper software 2 & 4 may be able to change place with each other, but the software would need to remove the software delays in this blog post's graphics.
Jim
SSDGuy: I assume these numbers are from an internally installed Optane drive? I believe there is a market for externally attached Optane, but the current NVMeoF proposals MAY dilute the value prop of Optane. We are based in Folsom CA, and have networked to Optane in the past. We cannot publish our results yet due to the copious number of NDA’s with Intel. However, it would be interesting to overlay the performance assumptions above with the RDMA based proposals of the NVMe over Fabrics group. The “goal” is 10us of induced latency, but we have not seen any actual numbers yet.
Jeff,
The Intel demos have used internally-installed Optane 3D XPoint NVME/PCIe SSDs. The numbers in the charts are all for internally-attached storage.
There’s a lot of interest in NVMeoF (NVMe over fabric) but this post is only about internal SSDs. It’s easy to understand how network latencies may be so much larger than XPoint latencies to the point where the advantage XPoint has over flash may be lost. Overlaying the performance, as you suggest, could indeed prove interesting.
Jim