| Latest revision |
Your text |
| Line 1: |
Line 1: |
| − | {{sc title|Astra}} | + | {{sc title|Astra}}[[File:astra supercomputer illustration.png|thumb|right|class=wikichip_ogimage|Astra Illustration]] |
| − | {{supercomputer
| + | '''Astra''' is a [[petascale]] [[ARM]] supercomputer designed for [[Sandia National Laboratories]] expeced to be deployed in mid-[[2018]]. This is the first ARM-based supercomputer to exceed 1 [[petaFLOPS]]. |
| − | |name=Astra
| |
| − | |image=astra supercomputer illustration.png
| |
| − | |sponsor=U.S. Department of Energy
| |
| − | |designer=Cavium | |
| − | |operator=Sandia National Laboratories | |
| − | |introduction=2018 | |
| − | |peak dpflops=2.322 petaFLOPS | |
| − | }}
| |
| − | '''Astra''' is a [[petascale]] [[ARM]] [[supercomputer]] operating by the [[DoE]] [[Sandia National Laboratories]]. Officially launching in late 2018, Astra became the first ARM-based supercomputer to exceed 1 [[petaFLOPS]] and the first [[ARM]]-based system to enter the [[TOP500]] list. | |
| | | | |
| | == History == | | == History == |
| | Astra is an [[ARM]]-based supercomputer expected to be deployed at [[Sandia National Laboratories]]. The computer is one of a series of prototypes commissioned by the [[U.S. Department of Energy]] as part of a program that evaluates the feasibility of emerging high-performance computing architectures as production platforms to support NNSA's mission. Specifically, Astra is designed to demonstrate the viability of ARM for [[DOE]] NNSA Supercomputing. Astra is the fourth prototype as part of the Vanguard project initiative and is by far the largest ARM-based supercomputer designed to that point. | | Astra is an [[ARM]]-based supercomputer expected to be deployed at [[Sandia National Laboratories]]. The computer is one of a series of prototypes commissioned by the [[U.S. Department of Energy]] as part of a program that evaluates the feasibility of emerging high-performance computing architectures as production platforms to support NNSA's mission. Specifically, Astra is designed to demonstrate the viability of ARM for [[DOE]] NNSA Supercomputing. Astra is the fourth prototype as part of the Vanguard project initiative and is by far the largest ARM-based supercomputer designed to that point. |
| − |
| |
| − | :[[File:sandia arm roadmap (2018).png|800px]]
| |
| | | | |
| | == Overview == | | == Overview == |
| Line 23: |
Line 12: |
| | <tr><th>Processors</th><td>5,184<br>2 x 72 x 36</td><td> </td><th>Type</th><td>[[DDR4]]</td><td>[[NVMe]]</td></tr> | | <tr><th>Processors</th><td>5,184<br>2 x 72 x 36</td><td> </td><th>Type</th><td>[[DDR4]]</td><td>[[NVMe]]</td></tr> |
| | <tr><th>Racks</th><td>36</td><td> </td><th>Node</th><td>128 GiB</td><td>?</td></tr> | | <tr><th>Racks</th><td>36</td><td> </td><th>Node</th><td>128 GiB</td><td>?</td></tr> |
| − | <tr><th>Peak FLOPS</th><td>2.322 petaFLOPS (DP)<br>4.644 petaFLOPS (SP)</td><td> </td><th>Astra</th><td>324 TiB</td><td>403 TB</td></tr> | + | <tr><th>Peak FLOPS</th><td>2.322 petaFLOPS</td><td> </td><th>Astra</th><td>324 TiB</td><td>403 TB</td></tr> |
| | </table> | | </table> |
| | | | |
| | == Architecture == | | == Architecture == |
| | + | {{empty section}} |
| | === System === | | === System === |
| − | Astra consists of 36 compute racks, 12 cooling racks, 3 networking racks, 2 storage racks, and a single utility rack.
| + | {{empty section}} |
| − | | |
| − | :[[File:astra floorplan.svg|300px]]
| |
| − | | |
| − | :[[File:astra_supercomputer_illustration.png|500px]]
| |
| − | | |
| − | :[[File:astra 540-port switch.svg|thumb|right|540-port Switch]]
| |
| − | | |
| − | The system has a peak Wall power of 1.6 MW.
| |
| − | | |
| − | {| class="wikitable" | |
| − | |-
| |
| − | ! colspan="4" | Total Power (kW)
| |
| − | |-
| |
| − | ! Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle
| |
| − | |-
| |
| − | | 1,631.5 || 1,440.3 || 1,357.3 || 274.9
| |
| − | |}
| |
| − | | |
| − | Servers are linked via Mellanox IB EDR interconnect in a three-level fat tree topology with a 2:1 tapered fat-tree at L1. Astra uses three 540-port switches. Those are formed from 30 level 2 switches that provide 18 ports each (540 in total) with the remaining 18 links going for each of the 18 level 3 switches.
| |
| − | | |
| − | | |
| − | :[[File:astra rack connection l1 to l2.svg|650px]]
| |
| − | | |
| | === Compute Rack === | | === Compute Rack === |
| − | Each of the 36 compute racks consist of 18 of HPE's Apollo 70 Chassis along with 3 36-port InfiniBand EDR switches. There is a single switch for every 6 chassis taking 24 ports.
| + | {{empty section}} |
| − | | |
| − | : [[File:astra racks and cooling (annotated).png|500px]]
| |
| − | [[File:astra rack connection.svg|thumb|right|Compute Node L1 links]]
| |
| − | Every IB link from each of the 24 nodes go directly to the 36-port switch. There are 12 remaining ports that go to the level 2 switches (discussed later). A full rack with 72 nodes has 4,032 cores yielding a peak performance of 64.51 [[teraFLOPS]] with over 24.57 TB/s of peak bandwidth.
| |
| − | | |
| − | <table class="wikitable">
| |
| − | <tr><th colspan="5">Full Rack Capabilities</th></tr>
| |
| − | <tr><th>Processors</th><td>144<br>72 × 2 × CPU</td></tr>
| |
| − | <tr><th>Core</th><td>4,032 (16,128 threads)<br>72 × 56 (224 threads)</td></tr>
| |
| − | <tr><th>FLOPS (SP)</th><td>129 TFLOPS<br>72 × 2 × 28 × 32 GFLOPS</td></tr>
| |
| − | <tr><th>FLOPS (DP)</th><td>64.51 TFLOPS<br>72 × 2 × 28 × 16 GFLOPS</td></tr>
| |
| − | <tr><th>Memory</th><td>9 TiB (DRR4)<br>72 × 2 × 8 × 8 GiB</td></tr>
| |
| − | <tr><th>Memory BW</th><td>24.57 TB/s<br>72 × 16 × 21.33 GB/s</td></tr>
| |
| − | </table>
| |
| − | | |
| − | Each compute rack has a projected peak power of 1295.8 kW (1436.0 kW Wall) with a nominal 1217.0 kW of power under [[linpack]].
| |
| − | | |
| | === Compute Node === | | === Compute Node === |
| − | [[File:apollo 70 system.png|right|thumb|Apollo 70]]
| + | {{empty section}} |
| − | The basic compute server is the HPE Apollo 70. Those use a highly dense chassis system architecture that fit in just 2U and consist of four dual-socket compute nodes.
| + | ==== Socket ==== |
| − | | + | {{empty section}} |
| − | :[[File:astra apollo 70 node.jpg|500px]]
| + | ==== Full-node ==== |
| − | | + | {{empty section}} |
| − | Each node has two 1,600 W power supplies, 1 Gbps Ethernet management port, and a Mellanox ConnectX-5 EDR link. Each node has a dual-socket [[Cavium]] {{cavium|ThunderX2}} [[ThunderX2 CN9975]] ({{cavium|Vulcan|l=arch}}) processor with 28 cores operating at 2 GHz. For Astra, Sandia is using 28-core parts operating at 2 GHz, likely due to a better performance/power efficiency design point. Each chip supports up to eight channels of DDR4 DIMMs with rates up to 2666 MT/s as well as 56 PCIe 3 lanes.
| |
| − | | |
| − | :[[File:astra node diagram.svg|600px]]
| |
| − | | |
| − | The [[ThunderX2 CN9975]] supports two-way [[multiprocessing]]. Communication is done over second-generation [[Cavium Coherent Processor Interconnect]] (CCPI2) which provides 600 Gbps of aggregated bandwidth. For the Astra supercomputer, each node uses 8 GiB DDR4-2666 dual-rank DIMM per controller for a total of 64 GiB and 170.7 GB/s of aggregated memory bandwidth per socket. For each node, there is a single Mellanox EDR InfiniBand ConnectX-5 VPI card designed for the Open Compute Project (OCP) providing the 100 Gb/s link.
| |
| − | | |
| − | With eight DIMMs per controller, each node has 128 GiB of memory feeding 56 cores with a total bandwidth of 341.33 GB/s per node. Those cores operate at up to 2 GHz, each with 2 NEON 128-bit units providing a peak theoretical performance of 8 single-precision FLOPS/cycle. This works out to 16 GFLOPS per core.
| |
| − | | |
| − | <table class="wikitable">
| |
| − | <tr><th colspan="5">Full Node Capabilities</th></tr>
| |
| − | <tr><th> </th><th>Socket</th><th>Node</th></tr>
| |
| − | <tr><th>Processors</th><td>1 × CPU</td><td>2 × CPU</td></tr>
| |
| − | <tr><th>Core</th><td>28 (112 threads)</td><td>56 (224 threads)</td></tr>
| |
| − | <tr><th>FLOPS (SP)</th><td>896 GFLOPS<br>28 × 32 GFLOPS</td><td>1,792 GFLOPS<br>2 × 28 × 32 GFLOPS</td></tr>
| |
| − | <tr><th>FLOPS (DP)</th><td>448 GFLOPS<br>28 × 16 GFLOPS</td><td>896 GFLOPS<br>2 × 28 × 16 GFLOPS</td></tr>
| |
| − | <tr><th>Memory</th><td>64 GiB (DRR4)<br>8 × 8 GiB</td><td>128 GiB (DRR4)<br>2 × 8 × 8 GiB</td></tr>
| |
| − | <tr><th>Bandwidth</th><td>170.7 GB/s<br>8 × 21.33 GB/s</td><td>341.33 GB/s<br>16 × 21.33 GB/s</td></tr>
| |
| − | </table>
| |
| − | | |
| − | == Power & Cooling == | |
| − | Astra has a nominal power consumption of 1.35 MW under [[LINPACK]]. Cooling the entire system are just 12 MCS-300 fan coil racks.
| |
| − | {| class="wikitable"
| |
| − | ! colspan="11" | Projected power of the system by component
| |
| − | |-
| |
| − | ! colspan="5" | Per constituent rack type (W) !! !! colspan="5" | Total (kW)
| |
| − | |-
| |
| − | ! Rack || Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle !! !! Racks !! style="background-color: yellow" | '''Wall''' !! Peak !! style="background-color: yellow" | '''Nominal (LINPACK)''' !! idle
| |
| − | |-
| |
| − | | Compute || 39,888 || 35,993 || 33,805 || 6,761 || || 36 || 1436.0 || 1295.8 || 1217.0 || 243.4
| |
| − | |-
| |
| − | | MCS-300 || 10,500 || 7,400 || 7,400 || 170 || || 12 || 126.0 || 88.8 || 88.8 || 2.0
| |
| − | |-
| |
| − | | Network || 12,624 || 10,023 || 9,021 || 9,021 || || 3 || 37.9 || 30.1 || 27.1 || 27.1
| |
| − | |-
| |
| − | | Storage || 11,520 || 10,000 || 10,000 || 1,000 || || 2 || 23.0 || 20.0 || 20.0 || 2.0
| |
| − | |-
| |
| − | | Utility || 8,640 || 5,625 || 4,500 || 450 || || 1 || 8.6 || 5.6 || 4.5 || 0.5
| |
| − | |-
| |
| − | | || || || || || || || style="background-color: yellow" | '''1,631.5''' || 1,440.3 || style="background-color: yellow" | '''1,357.3''' || 274.9
| |
| − | |}
| |
| | | | |
| | == Bibliography == | | == Bibliography == |
| | * SNL (personal communication, August 2018). | | * SNL (personal communication, August 2018). |
| | * DOE. (June 18, 2018). "''[https://share-ng.sandia.gov/news/resources/news_releases/arm_supercomputer/ Arm-based supercomputer prototype to be deployed at Sandia National Laboratories]''" [Press Release] | | * DOE. (June 18, 2018). "''[https://share-ng.sandia.gov/news/resources/news_releases/arm_supercomputer/ Arm-based supercomputer prototype to be deployed at Sandia National Laboratories]''" [Press Release] |
| − | * Kevin Pedretti, Jim H. Laros III, Si Hammond. (June 28, 2018). ''"ISC 2018"''. "''Vanguard Astra: Maturing the ARM Software Ecosystem for U.S. DOE/ASC Supercomputing''" | + | * Kevin Pedretti, Jim H. Laros III, Si Hammond. (June 28, 2018). "''Vanguard Astra: Maturing the ARM Software Ecosystem for U.S. DOE/ASC Supercomputing''" |
| − | * Schor, David. (August, 2018). "''[https://fuse.wikichip.org/news/1583/cavium-takes-arm-to-petascale-with-astra/ Cavium Takes ARM to Petascale with Astra]''"
| + | |
| | | | |
| | [[category:supercomputers]] | | [[category:supercomputers]] |
+