Editing supercomputers/astra

{{sc title|Astra}}
{{supercomputer
|name=Astra
|image=astra supercomputer illustration.png
|sponsor=U.S. Department of Energy
|designer=Cavium
|operator=Sandia National Laboratories
|introduction=2018
|peak dpflops=2.322 petaFLOPS
}}
'''Astra''' is a [[petascale]] [[ARM]] [[supercomputer]] operating by the [[DoE]] [[Sandia National Laboratories]]. Officially launching in late 2018, Astra became the first ARM-based supercomputer to exceed 1 [[petaFLOPS]] and the first [[ARM]]-based system to enter the [[TOP500]] list.

== History ==
Astra is an [[ARM]]-based supercomputer expected to be deployed at [[Sandia National Laboratories]]. The computer is one of a series of prototypes commissioned by the [[U.S. Department of Energy]] as part of a program that evaluates the feasibility of emerging high-performance computing architectures as production platforms to support NNSA's mission. Specifically, Astra is designed to demonstrate the viability of ARM for [[DOE]] NNSA Supercomputing. Astra is the fourth prototype as part of the Vanguard project initiative and is by far the largest ARM-based supercomputer designed to that point.

:[[File:sandia arm roadmap (2018).png|800px]]

== Overview ==
Astra is the first [[ARM]]-based petascale supercomputer. The system consists of 5,184 [[Cavium]] [[ThunderX2 CN9975]] processors with slightly over 1.2 MW power consumption for a peak performance of 2.322 [[petaFLOPS]]. Each [[ThunderX2 CN9975]] has [[28 cores]] operating at 2 GHz. Astra has close around 700 [[terabytes]] of memory and uses a 3-level fat tree [[interconnect architecture|interconnect]].

<table class="wikitable">
<tr><th></th><td>Components</td><td>&nbsp;</td><th></th><td colspan="2">Total Memory</td></tr>
<tr><th>Processors</th><td>5,184<br>2 x 72 x 36</td><td>&nbsp;</td><th>Type</th><td>[[DDR4]]</td><td>[[NVMe]]</td></tr>
<tr><th>Racks</th><td>36</td><td>&nbsp;</td><th>Node</th><td>128 GiB</td><td>?</td></tr>
<tr><th>Peak FLOPS</th><td>2.322 petaFLOPS (DP)<br>4.644 petaFLOPS (SP)</td><td>&nbsp;</td><th>Astra</th><td>324 TiB</td><td>403 TB</td></tr>
</table>

== Architecture ==
=== System ===
Astra consists of 36 compute racks, 12 cooling racks, 3 networking racks, 2 storage racks, and a single utility rack.

:[[File:astra floorplan.svg|300px]]

:[[File:astra_supercomputer_illustration.png|500px]]

:[[File:astra 540-port switch.svg|thumb|right|540-port Switch]]

The system has a peak Wall power of 1.6 MW.

{| class="wikitable"
|-
! colspan="4" | Total Power (kW)
|-
! Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle
|-
| 1,631.5 || 1,440.3 || 1,357.3 || 274.9
|}

Servers are linked via Mellanox IB EDR interconnect in a three-level fat tree topology with a 2:1 tapered fat-tree at L1. Astra uses three 540-port switches. Those are formed from 30 level 2 switches that provide 18 ports each (540 in total) with the remaining 18 links going for each of the 18 level 3 switches.


:[[File:astra rack connection l1 to l2.svg|650px]]

=== Compute Rack ===
Each of the 36 compute racks consist of 18 of HPE's Apollo 70 Chassis along with 3 36-port InfiniBand EDR switches. There is a single switch for every 6 chassis taking 24 ports.

: [[File:astra racks and cooling (annotated).png|500px]]
[[File:astra rack connection.svg|thumb|right|Compute Node L1 links]]
Every IB link from each of the 24 nodes go directly to the 36-port switch. There are 12 remaining ports that go to the level 2 switches (discussed later). A full rack with 72 nodes has 4,032 cores yielding a peak performance of 64.51 [[teraFLOPS]] with over 24.57 TB/s of peak bandwidth.

<table class="wikitable">
<tr><th colspan="5">Full Rack Capabilities</th></tr>
<tr><th>Processors</th><td>144<br>72 × 2 × CPU</td></tr>
<tr><th>Core</th><td>4,032 (16,128 threads)<br>72 × 56 (224 threads)</td></tr>
<tr><th>FLOPS (SP)</th><td>129 TFLOPS<br>72 × 2 × 28 × 32 GFLOPS</td></tr>
<tr><th>FLOPS (DP)</th><td>64.51 TFLOPS<br>72 × 2 × 28 × 16 GFLOPS</td></tr>
<tr><th>Memory</th><td>9 TiB (DRR4)<br>72 × 2 × 8 × 8 GiB</td></tr>
<tr><th>Memory BW</th><td>24.57 TB/s<br>72 × 16 × 21.33 GB/s</td></tr>
</table>

Each compute rack has a projected peak power of 1295.8 kW (1436.0 kW Wall) with a nominal 1217.0 kW of power under [[linpack]].

=== Compute Node ===
[[File:apollo 70 system.png|right|thumb|Apollo 70]]
The basic compute server is the HPE Apollo 70. Those use a highly dense chassis system architecture that fit in just 2U and consist of four dual-socket compute nodes.

:[[File:astra apollo 70 node.jpg|500px]]

Each node has two 1,600 W power supplies, 1 Gbps Ethernet management port, and a Mellanox ConnectX-5 EDR link. Each node has a dual-socket [[Cavium]] {{cavium|ThunderX2}} [[ThunderX2 CN9975]] ({{cavium|Vulcan|l=arch}}) processor with 28 cores operating at 2 GHz. For Astra, Sandia is using 28-core parts operating at 2 GHz, likely due to a better performance/power efficiency design point. Each chip supports up to eight channels of DDR4 DIMMs with rates up to 2666 MT/s as well as 56 PCIe 3 lanes.

:[[File:astra node diagram.svg|600px]]

The [[ThunderX2 CN9975]] supports two-way [[multiprocessing]]. Communication is done over second-generation [[Cavium Coherent Processor Interconnect]] (CCPI2) which provides 600 Gbps of aggregated bandwidth. For the Astra supercomputer, each node uses 8 GiB DDR4-2666 dual-rank DIMM per controller for a total of 64 GiB and 170.7 GB/s of aggregated memory bandwidth per socket. For each node, there is a single Mellanox EDR InfiniBand ConnectX-5 VPI card designed for the Open Compute Project (OCP) providing the 100 Gb/s link.

With eight DIMMs per controller, each node has 128 GiB of memory feeding 56 cores with a total bandwidth of 341.33 GB/s per node. Those cores operate at up to 2 GHz, each with 2 NEON 128-bit units providing a peak theoretical performance of 8 single-precision FLOPS/cycle. This works out to 16 GFLOPS per core.

<table class="wikitable">
<tr><th colspan="5">Full Node Capabilities</th></tr>
<tr><th>&nbsp;</th><th>Socket</th><th>Node</th></tr>
<tr><th>Processors</th><td>1 × CPU</td><td>2 × CPU</td></tr>
<tr><th>Core</th><td>28 (112 threads)</td><td>56 (224 threads)</td></tr>
<tr><th>FLOPS (SP)</th><td>896 GFLOPS<br>28 × 32 GFLOPS</td><td>1,792 GFLOPS<br>2 × 28 × 32 GFLOPS</td></tr>
<tr><th>FLOPS (DP)</th><td>448 GFLOPS<br>28 × 16 GFLOPS</td><td>896 GFLOPS<br>2 × 28 × 16 GFLOPS</td></tr>
<tr><th>Memory</th><td>64 GiB (DRR4)<br>8 × 8 GiB</td><td>128 GiB (DRR4)<br>2 × 8 × 8 GiB</td></tr>
<tr><th>Bandwidth</th><td>170.7 GB/s<br>8 × 21.33 GB/s</td><td>341.33 GB/s<br>16 × 21.33 GB/s</td></tr>
</table>

== Power & Cooling ==
Astra has a nominal power consumption of 1.35 MW under [[LINPACK]]. Cooling the entire system are just 12 MCS-300 fan coil racks.
{| class="wikitable"
! colspan="11" | Projected power of the system by component 
|-
! colspan="5" | Per constituent rack type (W) !! !! colspan="5" | Total (kW)
|-
! Rack || Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle !! !! Racks !! style="background-color: yellow" |  '''Wall''' !! Peak !! style="background-color: yellow" |  '''Nominal (LINPACK)''' !! idle 
|-
| Compute || 39,888 || 35,993 || 33,805 || 6,761 || || 36 || 1436.0 || 1295.8 || 1217.0 || 243.4 
|-
| MCS-300 || 10,500 || 7,400 || 7,400 || 170 || || 12 || 126.0 || 88.8 || 88.8 || 2.0 
|-
| Network || 12,624 || 10,023 || 9,021 || 9,021 || || 3 || 37.9 || 30.1 || 27.1 || 27.1
|-
| Storage || 11,520 || 10,000 || 10,000 || 1,000 || || 2 || 23.0 || 20.0 || 20.0 || 2.0 
|-
| Utility || 8,640 || 5,625 || 4,500 || 450 || || 1 || 8.6 || 5.6 || 4.5 || 0.5 
|-
| || || || || || || || style="background-color: yellow" | '''1,631.5''' || 1,440.3 || style="background-color: yellow" | '''1,357.3''' || 274.9 
|}

== Models ==
:;[[supercomputers|Supercomputers]] • [[DoE]]

*'''Astra''' (2018) - HPE, Apollo 70, [[Marvell]] ThunderX2 ARM CN9975-2000 28C @2.0GHz, 4x EDR Infiniband
:• Cores: 143,640 • Rmax: 1,833,000 GFLOPS • Rpeak: 2,298,240 GFLOPS

{| border="0" cellpadding="5" width="90%"
|- 
|width="45%" valign="top" align="left"|
:;SNL/NNSA
*{{sc|Mead}} (CTS-2) • 2025
*{{sc|El Dorado}} (Cray) • 2024 • ({{sc|El Capitan}})
*{{sc|Hops}} (CTS-2) • 2024
*{{sc|Stout}} (CTS-2) • 2023
*{{sc|Amber}} (CTS-2) • 2022
*{{sc|Manzano}} (CTS-1) • 2020
*{{sc|Attaway}} (CTS-1) • 2019
*{{sc|Astra}} (Apollo) • 2018
*{{sc|Eclipse}} (CTS-1) • 2017
|width="45%" valign="top" align="left"|
*{{sc|Cayenne}} (CTS-1) • 2016
*{{sc|Serrano}} (CTS-1) • 2016
*{{sc|Dark Ghost}} (CTS-1) • 2016
*{{sc|Sky Bridge}} (Cray) • 2015
*{{sc|Dark Bridge}} (Xtreme) • 2012
*{{sc|Dark Sand}} (Xtreme) • 2012
*{{sc|Pecos}} (Xtreme) • 2012
*{{sc|Chama}} (Xtreme) • 2011
*{{sc|Red Storm}} (Cray) • 2009
*{{sc|Thunderbird}} (PowerEdge) • 2005
|}

== TOP500 ==
*[https://top500.org/site/48748/ Sandia National Laboratories] (SNL)

<pre>
Sandia National Laboratories
URL:	http://www.sandia.gov/
Segment	Research
City:	Albuquerque
Country/Region:	United States

System	Year	Vendor		• Cores	       • Rmax (GFlop/s) • Rpeak (GFlop/s)

*El Dorado - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8 GHz, AMD Instinct MI300A, Slingshot-11, TOSS
:2024	HPE			• 383,040	• 68,020,000	• 95,290,620
*Hops (SNL CTS-2) - PowerEdge XE8640, Intel Xeon Platinum 8480+ 56C 2 GHz, NVIDIA H100 SXM5 80GB, I/B NDR
:2024	DELL, Linux/TOSS	• 40,960	• 10,030,000	• 17,585,150
*Stout (SNL CTS-2) - PowerEdge C6220, Intel Xeon Platinum 8480+ 56C 2 GHz, Cornelis Networks Omni-Path
:2023	DELL, Linux/TOSS	• 170,128	• 9,533,440	• 10,888,190
*Amber (SNL/NNSA CTS-2) - PowerEdge C6220, Intel Xeon Platinum 8480+ 56C 2 GHz, Cornelis Networks Omni-Path
:2022	DELL	• 169,344	• 9,508,360	• 10,838,020
*Mead (SNL CTS-2) - PowerEdge C6220, Intel Xeon Platinum 8480+ 56C 2 GHz, Cornelis Networks Omni-Path
:2025	DELL, Linux/TOSS	• 123,648	• 7,056,580	• 7,913,470
*Manzano (SNL/NNSA CTS-1) - Tundra Extreme Scale, Intel Xeon Platinum 8268 24C 2.9 GHz, Intel Omni-Path
:2020	Penguin Computing, Inc.	• 71,424	• 4,281,000	• 6,628,150
*Attaway (SNL/NNSA CTS-1)  - Tundra Extreme Scale, Intel Xeon Gold 6140 18C 2.3 GHz, Intel Omni-Path
:2019	Penguin Computing, Inc.	• 52,920	• 2,724,520	• 3,894,912
*Astra - Apollo 70, Marvell ThunderX2 ARM CN9975-2000 28C 2 GHz, 4x EDR Infiniband
:2018	HPE			• 143,640	• 1,833,000	• 2,298,240
*Eclipse (SNL/NNSA CTS-1) - Tundra Extreme Scale, Intel Xeon E5-2695v4 18C 2.1 GHz, Intel Omni-Path
:2017	Penguin Computing, Inc.	• 54,648	• 1,361,770	• 1,836,173
*Cayenne (SNL/NNSA CTS-1) - Tundra Extreme Scale, Intel Xeon E5-2695v4 18C 2.1 GHz, Intel Omni-Path
:2016	Penguin Computing, Inc.	• 40,392	• 1,223,700	• 1,357,171
*Serrano (SNL/NNSA CTS-1) - Tundra Extreme Scale, Intel Xeon E5-2695v4 18C 2.1 GHz, Intel Omni-Path
:2016	Penguin Computing, Inc.	• 40,392	• 1,223,700	• 1,357,171
*Dark Ghost (SNL CTS-1) - Tundra Extreme Scale, Intel Xeon E5-2695v4 18C 2.1 GHz, Intel Omni-Path
:2016	Penguin Computing, Inc.	• 25,920	• 761,787	• 870,912
*Sky Bridge - Cray CS300-LC, Intel Xeon E5-2670 8C 2.6 GHz, Intel Truscale
:2015	Cray/HPE		• 29,584	• 532,900	• 615,347
*Pecos - Xtreme-X, Intel Xeon E5-2670 8C 2.6 GHz, Infiniband QDR
:2012	Cray/HPE		• 19,712	• 336,800	• 410,010
*Chama - Xtreme-X GreenBlade GB512X, Intel Xeon E5-2670 8C 2.6 GHz, Infiniband QDR
:2011	Cray/HPE		• 19,680	• 332,000	• 409,344
*Dark Bridge - Appro Xtreme-X, Intel Xeon E5-2670 8C 2.6 GHz, Infiniband QDR
:2012	Cray/HPE		• 14,720	• 268,100	• 306,176
*Dark Sand - Appro Xtreme-X, Intel Xeon E5-2670 8C 2.6 GHz, Infiniband QDR
:2012	Cray/HPE		• 14,720	• 268,100	• 306,176
*Dark Bridge - Xtreme-X, Intel Xeon E5-2670 8C 2.6 GHz, Infiniband QDR
:2012	Cray/HPE		• 14,720	• 268,100	• 306,176
*Red Storm (Sandia/Cray) - Cray XT3/XT4
:2009	Cray/HPE		• 38,208	• 204,200	• 284,000
*Thunderbird - PowerEdge 1850, 3.6 GHz, Infiniband
:2005	DELL			• 8,000		• 38,270	• 57,600
</pre>

== See also ==
*[[AMD]] • [[EPYC]]
*[[Intel]] • [[Xeon]]
*[[IBM]] • [[NVIDIA]]
*[[DoE]] • [[NERSC]]

== Bibliography ==
* Sandia National Laboratories [SNL] (personal communication, August 2018).
* [[DoE]]. (June 18, 2018). "''[https://share-ng.sandia.gov/news/resources/news_releases/arm_supercomputer/ Arm-based supercomputer prototype to be deployed at Sandia National Laboratories]''" [Press Release]
* Kevin Pedretti, Jim H. Laros III, Si Hammond. (June 28, 2018). ''"ISC 2018"''. "''Vanguard Astra: Maturing the ARM Software Ecosystem for U.S. DOE/ASC Supercomputing''"
* Schor, David. (August, 2018). "''[https://fuse.wikichip.org/news/1583/cavium-takes-arm-to-petascale-with-astra/ Cavium Takes ARM to Petascale with Astra]''"

[[Category:supercomputers]]
[[Category:all supercomputers]]
designer	Cavium +
introductory date	2018 +
main image	+
name	Astra +
operator	Sandia National Laboratories +
peak flops (double-precision)	2.322e+15 FLOPS (2,322,000,000,000 KFLOPS, 2,322,000,000 MFLOPS, 2,322,000 GFLOPS, 2,322 TFLOPS, 2.322 PFLOPS, 0.00232 EFLOPS, 2.322e-6 ZFLOPS) +
sponsor	U.S. Department of Energy (DoE) +