Crusher supercomputing system launched with AMD CPUs and MI250X GPUs
The U.S. Department of Energy’s (DOE) Oak Ridge National Laboratory (ORNL) is building an ExaFLOP-class supercomputer called Frontier, a $600 million project. Before the official operation of Frontier, Crusher supercomputing will be temporarily replaced as a test platform. According to TomsHardware reports, the Crusher supercomputing has come online recently.
Crusher uses the same architecture and components as Frontier, and each HPE Cray EX node includes an AMD 64-core EPYC “Trento” 7A53 processor, 512GB of DDR4 memory, and four Instinct MI250X compute cards. Crusher has a total of 192 nodes, which are loaded into two cabinets, one with 128 nodes and the other with 64 nodes. However, the total footprint is only one-tenth of the previous Cray XK7 Titan supercomputer, but it provides higher computing performance. The Cray XK7 Titan was once one of the fastest supercomputers in the world, powered by AMD Opteron processors and Nvidia Tesla computing cards, serving hundreds of scientific studies between 2012 and 2019.
The EPYC “Trento” 7A53 processor on the Crusher and Frontier is a custom chip, AMD did not disclose too many details, only that it is a derivative of the Zen 3 architecture code-named Milan. It is rumored that its I/O chip uses Infinity Fabric 3.0 to achieve a memory interface consistent with the GPU. Each EPYC “Trento” 7A53 processor is divided into four NUMA regions, and each NUMA region is connected to an Instinct MI250X compute card.
The CPU and GPU are connected through the Infinity Fabric with an interface bandwidth of 36+36GB/s, and the total bandwidth of 288GB/s between the CPU and the GPU is distributed among the eight GCDs in the node. Each Crusher node is connected via four HPE Slingshot 200GBps Ethernet NICs (25GB/s), providing 800Gbps (100 GB/s) of node bandwidth.
The Frontier supercomputing system has been delivered in 2021 but is still undergoing integration and testing, and the specific operation timetable has not yet been determined. The Frontier supercomputing system is the first ExaFLOP-class supercomputer in the U.S. The U.S. Department of Energy expects to open it to researchers in January 2023.
