Why Does AI Need Different SSDs?

AI is on everyone’s minds these days, and SSD makers have lately been touting certain models, explaining that they have been designed for AI applications.  Why would that be?  Is there a difference between AI SSDs and those used for conventional applications?

The answers to both questions would be “Yes,” and in this post The SSD Guy will attempt to explain why in a clear and straightforward way.

Most SSDs Focus on Random IOPS

When SSDs first started to be used in enterprise applications they were promoted as a way to improve storage access times, which were already pretty chaotic.  It seems that server virtualization had recently become important, and storage was being moved out of the server (Direct-Attached Storage or DAS) into storage arrays on the network (Network-Attached Storage, or NAS).  This resulted in more varied I/O patterns into and out of storage, since a number of servers, each doing a different job, were accessing the shared storage at the same time, which combined several relatively orderly I/O streams into a single relatively jumbled stream.

On top of this, the virtualization software, which allowed any one server to run several programs simultaneously, no longer accessed storage in the orderly manner that servers had previously done when each server ran a single piece of software (e-mail, or web portal, or accounting software, or EDA…) but now every server in the data center was jumping from one program to another to create its own jumbled I/O stream.

This phenomenon was given a name by my colleague Robin Harris of Storage Mojo who dubbed it “The Storage Blender.”

HDDs had trouble keeping up with these jumbled random accesses thanks to their mechanical approach to storage, since there were rotational latencies and head-movement latencies, and this allowed SSDs to fill a gap between HDDs and DRAM in the memory/storage hierarchy.  SSDs serviced jumbled accesses so well that users started to choose them based on their random IOPS (Inputs or Outputs Per Second) specifications.  Suppliers found that they could command higher prices for SSDs with higher IOPS, and they helped increase the customers’ focus on random IOPS.

AI Workloads are Sequential

AI presents fundamentally different requirements on SSDs.  You have doubtlessly heard of the enormous databases that are managed by today’s Large Language Models (LLMs) for generative AI.  From a math perspective, these systems take a huge number of inputs as an input vector, multiply these inputs by a two-dimensional matrix of equal width and some enormous depth, to produce an enormous output vector.

(I would have made the matrix roughly as high as it is wide, maybe 10^75 or something, but the subscripts got too ugly, so I only gave the output vector 99 elements.  The intent is to underscore the phenomenal size of these datasets.)

Depending on the number of layers in the system, the output vector may be processed again as the input to a new matrix a number of times, once per layer.  Fortunately, the number of layers is usually far less than ten.

Ideally, all of the input values would be stored in DRAM.  So would the matrix weights, the output vector, and any other matrices and intermediate vectors that needed to be generated during the process.  At some point this becomes prohibitively expensive, although you would think that nothing could be too costly for a hyperscale datacenter, given that the major hyperscale datacenters’ combined annual capital spending runs in the hundreds of billions of dollars.

But even with that level of spending their money runs out, and that means that they won’t be able to keep all of the variables for these huge datasets and their weights in DRAM for the entire duration of a problem.  Instead, they must swap DRAM data into and out of fast storage, and that storage is SSDs.

But the traffic in this case is different from the traffic presented by virtualized networked servers.  When a super-large input vector or a matrix that is many times that size is swapped into or out of DRAM, that data stream is serial — everything is transferred in a very long sequential stream to the SSD.  So the SSDs that handle this data stream need not have the best IOPS specifications, but they do need to be able to service sequential reads and writes as fast as possible.

That changes the way that AI SSDs are designed.

Requirements for AI SSDs

Here’s what SSD designers consider when designing an SSD aimed at AI data centers.

First of all, the data’s structure is a better match for the internal workings of NAND flash chips, which are designed to read and write data sequentially, one page at a time.  When SSD designs are optimized to maximize IOPS the SSD controller must juggle around data flows to match the system’s highly-random data streams to NAND’s sequential streams.  With an AI workload very little juggling is required, and the SSD’s performance is more determined by the maximum speed at which the NAND flash chips can accept or provide data.  This means that an AI SSD must be designed to maximize sequential bandwidth for both reads and writes.

Second, AI’s very large data sets are better matched to higher SSD capacities than to lower ones.  This is mainly due to the fact that the data sets are very large, but the data centers that they go into have limited space and power budgets.  High-capacity SSDs reduce both the power and the size of the storage array, perhaps at the expense of random IOPS, but AI systems are less seriously impacted by random IOPS than by power and space.

Third, it takes time to start writing a data set into one SSD and then switch to a second SSD when the first one is filled.  With super-high-capacity SSDs this slow transition will be required a smaller number of times, so that you don’t use twenty SSDs to hold the data that you could put into five.  SSDs designed for AI data centers must provide the highest-possible capacities within a single slot.

About a year ago the SSD Guy blog published a post called: New Interest in Monster SSDs.  This post puzzled over the fact that SSD makers were introducing huge SSDs behind a narrow interface.  This interface constrained the SSD’s enormous amount of internal flash capacity, which could provide phenomenal random bandwidth, behind an interface that couldn’t take advantage of all of that bandwidth.  At that time I didn’t fully appreciate the problem that the SSD manufacturers were trying to solve.  With my new understanding of AI workloads I see that random IOPS provide little value to an AI workload, while very high-capacity SSDs provide a very good solution to the AI data center’s most pressing problems.

My company, Objective Analysis, takes pains to fully understand the problems that confront the industry as technologies grow and change.  Your company can use that understanding to provide itself a competitive advantage.  Please contact us to explore ways that we can work together to your company’s benefit.