Editing supercomputers/astra

{{sc title|Astra}}
{{supercomputer
|name=Astra
|image=astra supercomputer illustration.png
|sponsor=U.S. Department of Energy
|designer=Cavium
|operator=Sandia National Laboratories
|introduction=2018
|peak dpflops=2.322 petaFLOPS
}}
'''Astra''' is a [[petascale]] [[ARM]] [[supercomputer]] operating by the [[DoE]] [[Sandia National Laboratories]]. Officially launching in late 2018, Astra became the first ARM-based supercomputer to exceed 1 [[petaFLOPS]] and the first [[ARM]]-based system to enter the [[TOP500]] list.

== History ==
Astra is an [[ARM]]-based supercomputer expected to be deployed at [[Sandia National Laboratories]]. The computer is one of a series of prototypes commissioned by the [[U.S. Department of Energy]] as part of a program that evaluates the feasibility of emerging high-performance computing architectures as production platforms to support NNSA's mission. Specifically, Astra is designed to demonstrate the viability of ARM for [[DOE]] NNSA Supercomputing. Astra is the fourth prototype as part of the Vanguard project initiative and is by far the largest ARM-based supercomputer designed to that point.

:[[File:sandia arm roadmap (2018).png|800px]]

== Overview ==
Astra is the first [[ARM]]-based petascale supercomputer. The system consists of 5,184 [[Cavium]] [[ThunderX2 CN9975]] processors with slightly over 1.2 MW power consumption for a peak performance of 2.322 [[petaFLOPS]]. Each [[ThunderX2 CN9975]] has [[28 cores]] operating at 2 GHz. Astra has close around 700 [[terabytes]] of memory and uses a 3-level fat tree [[interconnect architecture|interconnect]].

<table class="wikitable">
<tr><th></th><td>Components</td><td>&nbsp;</td><th></th><td colspan="2">Total Memory</td></tr>
<tr><th>Processors</th><td>5,184<br>2 x 72 x 36</td><td>&nbsp;</td><th>Type</th><td>[[DDR4]]</td><td>[[NVMe]]</td></tr>
<tr><th>Racks</th><td>36</td><td>&nbsp;</td><th>Node</th><td>128 GiB</td><td>?</td></tr>
<tr><th>Peak FLOPS</th><td>2.322 petaFLOPS (DP)<br>4.644 petaFLOPS (SP)</td><td>&nbsp;</td><th>Astra</th><td>324 TiB</td><td>403 TB</td></tr>
</table>

== Architecture ==
=== System ===
Astra consists of 36 compute racks, 12 cooling racks, 3 networking racks, 2 storage racks, and a single utility rack.

:[[File:astra floorplan.svg|300px]]

:[[File:astra_supercomputer_illustration.png|500px]]

:[[File:astra 540-port switch.svg|thumb|right|540-port Switch]]

The system has a peak Wall power of 1.6 MW.

{| class="wikitable"
|-
! colspan="4" | Total Power (kW)
|-
! Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle
|-
| 1,631.5 || 1,440.3 || 1,357.3 || 274.9
|}

Servers are linked via Mellanox IB EDR interconnect in a three-level fat tree topology with a 2:1 tapered fat-tree at L1. Astra uses three 540-port switches. Those are formed from 30 level 2 switches that provide 18 ports each (540 in total) with the remaining 18 links going for each of the 18 level 3 switches.


:[[File:astra rack connection l1 to l2.svg|650px]]

=== Compute Rack ===
Each of the 36 compute racks consist of 18 of HPE's Apollo 70 Chassis along with 3 36-port InfiniBand EDR switches. There is a single switch for every 6 chassis taking 24 ports.

: [[File:astra racks and cooling (annotated).png|500px]]

A full rack with 72 nodes has 4,032 cores yielding a peak performance of 64.51 [[teraFLOPS]] with over 24.57 TB/s of peak bandwidth.

<table class="wikitable">
<tr><th colspan="5">Full Rack Capabilities</th></tr>
<tr><th>Processors</th><td>144<br>72 × 2 × CPU</td></tr>
<tr><th>Core</th><td>4,032 (16,128 threads)<br>72 × 56 (224 threads)</td></tr>
<tr><th>FLOPS (SP)</th><td>129 TFLOPS<br>72 × 2 × 28 × 32 GFLOPS</td></tr>
<tr><th>FLOPS (DP)</th><td>64.51 TFLOPS<br>72 × 2 × 28 × 16 GFLOPS</td></tr>
<tr><th>Memory</th><td>9 TiB (DRR4)<br>72 × 2 × 8 × 8 GiB</td></tr>
<tr><th>Memory BW</th><td>24.57 TB/s<br>72 × 16 × 21.33 GB/s</td></tr>
</table>

Each compute rack has a projected peak power of 1295.8 kW (1436.0 kW Wall) with a nominal 1217.0 kW of power under [[linpack]].
=== Compute Node ===
[[File:apollo 70 system.png|right|thumb|Apollo 70]]
The basic compute server is the HPE Apollo 70. Those use a highly dense chassis system architecture that fit in just 2U and consist of four dual-socket compute nodes.

:[[File:astra apollo 70 node.jpg|500px]]

Each node has two 1,600 W power supplies, 1 Gbps Ethernet management port, and a Mellanox ConnectX-5 EDR link. Each node has a dual-socket [[Cavium]] {{cavium|ThunderX2}} [[ThunderX2 CN9975]] ({{cavium|Vulcan|l=arch}}) processor with 28 cores operating at 2 GHz. For Astra, Sandia is using 28-core parts operating at 2 GHz, likely due to a better performance/power efficiency design point. Each chip supports up to eight channels of DDR4 DIMMs with rates up to 2666 MT/s as well as 56 PCIe 3 lanes.

:[[File:astra node diagram.svg|600px]]

The [[ThunderX2 CN9975]] supports two-way [[multiprocessing]]. Communication is done over second-generation [[Cavium Coherent Processor Interconnect]] (CCPI2) which provides 600 Gbps of aggregated bandwidth. For the Astra supercomputer, each node uses 8 GiB DDR4-2666 dual-rank DIMM per controller for a total of 64 GiB and 170.7 GB/s of aggregated memory bandwidth per socket. For each node, there is a single Mellanox EDR InfiniBand ConnectX-5 VPI card designed for the Open Compute Project (OCP) providing the 100 Gb/s link.

With eight DIMMs per controller, each node has 128 GiB of memory feeding 56 cores with a total bandwidth of 341.33 GB/s per node. Those cores operate at up to 2 GHz, each with 2 NEON 128-bit units providing a peak theoretical performance of 8 single-precision FLOPS/cycle. This works out to 16 GFLOPS per core.

<table class="wikitable">
<tr><th colspan="5">Full Node Capabilities</th></tr>
<tr><th>&nbsp;</th><th>Socket</th><th>Node</th></tr>
<tr><th>Processors</th><td>1 × CPU</td><td>2 × CPU</td></tr>
<tr><th>Core</th><td>28 (112 threads)</td><td>56 (224 threads)</td></tr>
<tr><th>FLOPS (SP)</th><td>896 GFLOPS<br>28 × 32 GFLOPS</td><td>1,792 GFLOPS<br>2 × 28 × 32 GFLOPS</td></tr>
<tr><th>FLOPS (DP)</th><td>448 GFLOPS<br>28 × 16 GFLOPS</td><td>896 GFLOPS<br>2 × 28 × 16 GFLOPS</td></tr>
<tr><th>Memory</th><td>64 GiB (DRR4)<br>8 × 8 GiB</td><td>128 GiB (DRR4)<br>2 × 8 × 8 GiB</td></tr>
<tr><th>Bandwidth</th><td>170.7 GB/s<br>8 × 21.33 GB/s</td><td>341.33 GB/s<br>16 × 21.33 GB/s</td></tr>
</table>

== Power & Cooling ==
Astra has a nominal power consumption of 1.35 MW under [[LINPACK]]. Cooling the entire system are just 12 MCS-300 fan coil racks.
{| class="wikitable"
! colspan="11" | Projected power of the system by component 
|-
! colspan="5" | Per constituent rack type (W) !! !! colspan="5" | Total (kW)
|-
! Rack || Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle !! !! Racks !! style="background-color: yellow" |  '''Wall''' !! Peak !! style="background-color: yellow" |  '''Nominal (LINPACK)''' !! idle 
|-
| Compute || 39,888 || 35,993 || 33,805 || 6,761 || || 36 || 1436.0 || 1295.8 || 1217.0 || 243.4 
|-
| MCS-300 || 10,500 || 7,400 || 7,400 || 170 || || 12 || 126.0 || 88.8 || 88.8 || 2.0 
|-
| Network || 12,624 || 10,023 || 9,021 || 9,021 || || 3 || 37.9 || 30.1 || 27.1 || 27.1
|-
| Storage || 11,520 || 10,000 || 10,000 || 1,000 || || 2 || 23.0 || 20.0 || 20.0 || 2.0 
|-
| Utility || 8,640 || 5,625 || 4,500 || 450 || || 1 || 8.6 || 5.6 || 4.5 || 0.5 
|-
| || || || || || || || style="background-color: yellow" | '''1,631.5''' || 1,440.3 || style="background-color: yellow" | '''1,357.3''' || 274.9 
|}

== Bibliography ==
* SNL (personal communication, August 2018).
* DOE. (June 18, 2018). "''[https://share-ng.sandia.gov/news/resources/news_releases/arm_supercomputer/ Arm-based supercomputer prototype to be deployed at Sandia National Laboratories]''" [Press Release]
* Kevin Pedretti, Jim H. Laros III, Si Hammond. (June 28, 2018). ''"ISC 2018"''. "''Vanguard Astra: Maturing the ARM Software Ecosystem for U.S. DOE/ASC Supercomputing''"
* Schor, David. (August, 2018). "''[https://fuse.wikichip.org/news/1583/cavium-takes-arm-to-petascale-with-astra/ Cavium Takes ARM to Petascale with Astra]''"

[[category:supercomputers]]
designer	Cavium +
introductory date	2018 +
main image	+
name	Astra +
operator	Sandia National Laboratories +
peak flops (double-precision)	2.322e+15 FLOPS (2,322,000,000,000 KFLOPS, 2,322,000,000 MFLOPS, 2,322,000 GFLOPS, 2,322 TFLOPS, 2.322 PFLOPS, 0.00232 EFLOPS, 2.322e-6 ZFLOPS) +
sponsor	United States Department of Energy (DOE) +