The folks at the PCI-SIG technical work groups are working hard to agree upon a specification for PCIe Gen. 8 by 2028. In a recent announcement the group said that the specification’s draft 0.5 has been released for member review ahead of schedule, and that the released specification remains on track for full release by 2028.
As with earlier PCIe generations, the goals of the new specification are to provide greater bandwidth and lower latency. The specification’s objectives are:
-
-
- To deliver a raw speed of 256.0 GT/s, supporting up to 1.0 TB/s on a 16-lane bidirectional configuration
- Use protocol enhancements to improve bandwidth
- Achieve targets for
- Latency
- Forward Error Correction (FEC)
- Reliability
- Maintain backwards compatibility with all prior PCIe generations
- Reduce power
- Evaluate new connector technologies
-
That 256GT/s number is very impressive. In fact, it’s so fast that The Memory Guy decided that the appropriate graphic for this post would be a Figure-8 racetrack, an odd design created to increase the likelihood of collisions. PCIe’s bandwidth has doubled with each generation, and Gen 8 will be fast enough to cause its own unique set of issues.
During the Silicon Valley IMAPS Memory Summit on May 15, Shigenori Yanagi, Assistant Director of Kioxia’s Frontier Technology R&D Institute, gave a compelling keynote presentation to explain what this high transfer rate means to NAND flash chips. In a brief, PCIe Gen 8 is too fast for today’s NAND flash chips. Even PCIe Gen. 6 has caused issues with very high stacks of NAND dice. When chips are stacked 16-high or 32-high the bonding wires are long enough to impact timing. Dr. Yanagi used this image of a 32-layer NAND stack’s cross section to illustrate just how long those bonding wires are:
The long yellow wires plus the high capacitive loading of all these chips’ bonding pads cause signals to be skewed. Kioxia’s solution is to integrate a Switch Chip to manage the NAND interface.
But PCIe Gen. 8 will have 4 times the transfer rate of Gen. 6, and that makes Gen. 8 significantly harder for NAND chips to support. The NAND chips in a PCIe Gen. 8 SSD need to have an I/O bandwidth of 11.1Gb/s.
Dr. Yanagi explained that wire bonding would no longer be an option, and that the NAND chips would need either to be stacked with TSVs, or might even use hybrid bonding. Meanwhile, the trace lengths between the flash chips and the controller would need to be shortened too, leading to the option of co-packaging the NAND stack with the controller.
Optical interconnects may become necessary for the system to communicate with the SSD, otherwise, too many retimers would be necessary. Yanagi showed a chart that projected that if copper wires were used to connect via the PCIe Gen. 8 interface, a 256GT/s signal would need to connect with traces below 1μm in length!
For this reason, even PCIe Gen. 6 incorporated optical communication standards in its rev. 4 iteration. While this is encouraging, it seems that it won’t necessarily be enough. Co-packaged optics may be required to place the optical interface chip close enough to the SSD controller that the two chips can communicate fast enough to support all of Gen. 8’s speed.
Several concepts are being investigated, one of which uses a TSV-connected stack of NAND, similar to an HBM DRAM, perhaps even stacking it above the SSD controller chip. This approach, while costly, would help a lot to reduce latency and increase bandwidth. There are even considerations of using liquid cooling on these stacks.
It’s hard for the Memory Guy to even think in terms of hundreds of gigatransfers per second! This kind of frequency requires a lot of engineering just to make sure the signals perform well in the typical controlled environment of a server. A microwave oven runs at 1/100th that rate, and even 5G cell phones’ maximum frequency is only 39GHz. The wavelength of a 256GHz signal is only 1.17mm.
But like all the nearly-impossible feats that are regularly tackled by the great minds in this industry, a solution is sure to be found and it will be put into mass production which will make it commonplace.
And when it does, our Objective Analysis clients will be ready with plans that we helped them create to take advantage of all that this new PCIe standard will offer. Please contact us to learn how we can help your own company develop its best strategic plans.