Historically, high-performance computing (HPC) was used for a very narrow set of problems: fluid dynamics (especially for nuclear weapon simulation and design verification), weather prediction and modeling, aerodynamic simulation, and particle physics modeling. Over the last decade, the problem space that HPC can successfully and economically address has grown considerably, fueled primarily by increases in the capabilities of both servers and GPGPUs. However, as data sets have grown, the new bottleneck has become the movement of data from storage systems and storage devices to the servers and CPUs for processing. As we stated in our last blog, this is an issue that computational storage can address for specific HPC problems.
Generally speaking, the problems that computational storage can address in HPC are those that utilize petabyte-scale data sets, are read-intensive, and involve parallel operations on the data sets. Even better are those applications that perform significant searching of the data sets to find vector similarities. Problems that computational storage is not well positioned for are problems that are highly scalar in nature or are primarily write-intensive (complex data transformations fall into this category). Finally, some computational storage solutions provide acceleration for some problem sets such as artificial intelligence (AI) and encryption/decryption.
For the NGD Systems Newport and Catalina-2 computational storage platforms, workloads that match the attributes above include TensorFlow-based HPC applications such as the Facebook Artificial Intelligence Similarity Search (FAISS), biological workloads such as BLAST, unstructured databases like Apache HBASE, Redis and Aerospike, and applications like Hadoop MapReduce. For these and other similar applications, computational storage can significantly reduce the amount of data moved between storage and CPU DRAM, increasing performance and the number of jobs that can be run on a given hardware footprint. Find out how NGD Systems computational storage can help your HPC workloads at www.ngdsystems.com.