SSDs are often used to either minimize a server’s DRAM requirements or to reduce the number of servers required in a data center, a fact that The SSD Guy likes to bring up whenever he can. These attributes were the main reason that SSDs gained such rapid acceptance in hyperscale datacenters about 20 years ago, when the world’s largest datacenters began to radically change the structure of their systems and embrace the widespread use of SSDs.
With this in mind, it’s unsurprising that a similar approach is now being promoted to lower the cost and power consumption of AI training. Phison is leading this campaign with a hardware + software solution they call aiDAPTIV+.
More SSDs, Less DRAM
I frequently try to draw attention to the poorly-understood fact that SSDs can be used to reduce a system’s DRAM requirements, and this approach may even reduce the number of servers needed to achieve a certain performance level. This quite naturally leads to lower installation costs and minimized power consumption with a negligible speed penalty, if there is any penalty at all.
One example of this appears in an earlier Memory Guy blog post that was published way back in 2018. We can go back even further to a report that I published in 2012 that used a series of benchmarks to prove that a PC of a given cost performed better with a small DRAM and a large NAND flash than a similar system that cost the same but allocated all of its memory budget to maximize the DRAM’s size.
Well the folks at Phison decided to apply this principal to AI. Their aiDAPTIV+ design uses specialized SSDs to reduce the amount of HBM DRAM required in an LLM training system.
Phison argues that the number of GPUs needed in a system is determined by the size of the HBM DRAM the problem requires. Their argument is that if a GPU has 80GB of HBM, and it’s running a model that needs 800GB of parameters, then you need to use ten GPUs.
While I don’t know enough to weigh in on that argument, I understand their point that if a GPU that is limited by its memory size then it can provide greatly-improved performance if data from a terabyte SSD is effectively used to swap data into and out of the GPU’s HBM. Phison tells me that a 70TB LLM consumes 1.4TB of data on disk. Without an SSD offload like that provided by aiDAPTIV+, once you run out of GPU memory you simply get an out-of-memory error and the work stops.
This is the basic principal behind any conventional processor’s virtual memory system as well as all of its caches, whether the blazing fast SRAM caches in the processor chip itself, or much slower SSD caches in front of an HDD. In all of these implementations, a slower large memory or storage device holds a large quantity of data that is automatically swapped into and out of a smaller, faster memory or storage device to achieve nearly the same performance as the system would get if the faster device were as large as the slower device.
So why not use the same approach to swap data into and out of a GPU’s HBM DRAM? The engineers at Phison decided to try this approach, and have had very good results, as we will see shortly.
AI Training Needs a Special SSD
One issue with AI training is that it generates a lot of write traffic, and writes are a particular problem with SSDs for two reasons: SSD writes are slow, and they cause the SSD to wear out.
aiDAPTIV+ needs SSDs with very low read and write latencies, and very good endurance. Phison, who as an SSD controller maker deeply understands how to manage flash chips, designed a special SSD for aiDAPTIV+ that outperforms others by taking the following steps:
-
-
- SLC flash is pretty fast, while MLC, TLC, and QLC get successively slower in order to store more bits onto a single flash cell. The new SSD uses TLC flash in pseudo SLC mode (pSLC). Ordinarily, SLC flash is much more costly than these others, but internally it’s the same as MLC, TLC, or QLC as long as you bypass the process of converting multiple bits into varying voltages to store on a single cell. The Phison engineers understand this so they implement SLC by starting with TLC NAND and then bypassing the conversion.
- SLC also has significantly greater endurance than multi-bit technologies, so the drive gets an endurance boost of about 20X by using TLC flash as SLC.
- The design is based on an existing 8TB TLC SSD, and that 8TB would be reduced to 2.73TB when used as pSLC.
- Overprovisioning is used in nearly all SSDs to speed up their internal operation and improve their endurance. It’s explained in depth in another SSD Guy blog post, but in brief I’ll say that overprovisioning involves using more flash than the specified capacity of the drive. It’s expensive, though – the more you overprovision, the more expensive the SSD. But the more you overprovision, the faster the SSD, and the more writes it can withstand. Phison’s SSD for aiDAPTIV+ is a 2TB SSD – the drive’s overprovisioning includes not only the overprovisioning that was already in the original 8TB TLC SSD, but it also adds in 0.73TB of that 2.73TB mentioned above. This provides an unusually large amount of overprovisioning.
- NAND flash loses bits as it ages. In older SLC and MLC SSDs this could be managed with simpler algorithms, but TLC and QLC require better error correction, so the industry converted to a more expensive LDPC approach for them. Designers can improve a controller’s LDPC error correction quality at an added cost. The SSD for aiDAPTIV+ uses very aggressive LDPC on SLC flash which has fewer errors to begin with, and this provides the SSD with a significant additional endurance boost.
- SLC flash is pretty fast, while MLC, TLC, and QLC get successively slower in order to store more bits onto a single flash cell. The new SSD uses TLC flash in pseudo SLC mode (pSLC). Ordinarily, SLC flash is much more costly than these others, but internally it’s the same as MLC, TLC, or QLC as long as you bypass the process of converting multiple bits into varying voltages to store on a single cell. The Phison engineers understand this so they implement SLC by starting with TLC NAND and then bypassing the conversion.
-
All of this is used to produce a 2TB SSD that boasts an endurance of 100 DWPD, while operating at a sustained sequential write bandwidth of 6GB/s per drive. Loading and storing HBM tends to be highly sequential, so this is a good match!
Phison explains that the aiDAPTIV+ software manages the data ingress and egress to the SSDs for offload, and has been tuned to maximize the offload’s data flow. They tell me that it’s not unusual to see 7GB/s reads and 6GB/s writes per drive, and in larger systems, two drives run with LVM RAID0 to achieve 14GB/s reads and 6GB/s writes. The aiDAPTIV+ software was optimized to take advantage of PCIe and NVMe bidirectional communications when swapping data.
These techniques all add a lot of cost to the SSD, but GPUs themselves are very expensive, costing tens of thousands of dollars, so an SSD with all these cost increases can still provide a solid cost saving at the system level.
Phison calls their AI SSD the aiDAPTIVCache, selling it as one of their Pascari brand of high-end enterprise SSDs. Other capacities are in the works.
The End Result
The aiDAPTIV+ solution combines this very special SSD with software that manages data into and out of the HBM on GPUs. Pytorch is the that are most popular framework, so aiDAPTIV+ software was originally set up to work with it, but today more models are also supported, including various versions of Llama, Mistral, Resnet, and many others.
Phison tells me that aiDAPTIV+ allows a 4-GPU system that would normally handle no more than a 7 billion parameter model can be used with aiDAPTIV+ to train models ten times that size, as it illustrated below.
This chart and the numbers at the bottom state that a 70 billion parameter model would normally need 30 GPUs to train (specifically, GPUs with 48GB of HBM, like the L40S or the RTX 6000 ada), while the same job could be performed with only 4 of these GPUs equipped with the aiDAPTIV+ software and SSDs. Phison explains that a 30 GPU system would cost around $288K, but a system with four RTX 6000 ada GPUs with aiDAPTIV+ should only cost about $36K.
There’s also a decided power advantage to using a system with fewer GPUs. With this approach, a 70B-parameter LLM workstation can be built with 4 GPUs and 1-2 CPUs, plus 2 of Phison’s aiDAPTIV+ SSDs. Phison’s team tells me that this particular workstation design consumes less than 15 Amps, allowing it to be air cooled while operating from the power available in a normal household.
Phison has told me that the offload mechanism of aiDAPTIV+ is so strong that you could even train the same 70B model on a very modest system with a single RTX 4000 ada, which only has 20GB of GDDR. Completing a 70B perimeter model with such extremely low GPU resources would come at a cost of time, though, since the GPU compute would be the limiting factor.
All in all, I find this to be an intriguing approach to saving money on AI training, which is usually a very costly process. If you’d like to learn more, Phison has a web page that you can consult.
And please keep in mind that we at Objective Analysis focus on deeply understanding technologies and issues that could change the market, so that we can build these into our forecasts and strategic planning services. Please reach out to us to explore ways that we might work together to optimize your own company’s performance.
[NOTE: After publishing the original on May 9, the folks at Phison noted some misunderstandings which The SSD Guy corrected on May 14. While these updates improve the post’s accuracy, nothing has changed regarding the basic concept as originally described.]