From WikiChip
Difference between revisions of "supercomputers/astra"
< supercomputers

(History)
(Power & Cooling: Fix comma)
 
(21 intermediate revisions by one other user not shown)
Line 1: Line 1:
{{sc title|Astra}}[[File:astra supercomputer illustration.png|thumb|right|class=wikichip_ogimage|Astra Illustration]]
+
{{sc title|Astra}}
'''Astra''' is a [[petascale]] [[ARM]] supercomputer designed for [[Sandia National Laboratories]] expeced to be deployed in mid-[[2018]]. This is the first ARM-based supercomputer to exceed 1 [[petaFLOPS]].
+
{{supercomputer
 +
|name=Astra
 +
|image=astra supercomputer illustration.png
 +
|sponsor=U.S. Department of Energy
 +
|designer=Cavium
 +
|operator=Sandia National Laboratories
 +
|introduction=2018
 +
|peak dpflops=2.322 petaFLOPS
 +
}}
 +
'''Astra''' is a [[petascale]] [[ARM]] [[supercomputer]] operating by the [[DoE]] [[Sandia National Laboratories]]. Officially launching in late 2018, Astra became the first ARM-based supercomputer to exceed 1 [[petaFLOPS]] and the first [[ARM]]-based system to enter the [[TOP500]] list.
  
 
== History ==
 
== History ==
 
Astra is an [[ARM]]-based supercomputer expected to be deployed at [[Sandia National Laboratories]]. The computer is one of a series of prototypes commissioned by the [[U.S. Department of Energy]] as part of a program that evaluates the feasibility of emerging high-performance computing architectures as production platforms to support NNSA's mission. Specifically, Astra is designed to demonstrate the viability of ARM for [[DOE]] NNSA Supercomputing. Astra is the fourth prototype as part of the Vanguard project initiative and is by far the largest ARM-based supercomputer designed to that point.
 
Astra is an [[ARM]]-based supercomputer expected to be deployed at [[Sandia National Laboratories]]. The computer is one of a series of prototypes commissioned by the [[U.S. Department of Energy]] as part of a program that evaluates the feasibility of emerging high-performance computing architectures as production platforms to support NNSA's mission. Specifically, Astra is designed to demonstrate the viability of ARM for [[DOE]] NNSA Supercomputing. Astra is the fourth prototype as part of the Vanguard project initiative and is by far the largest ARM-based supercomputer designed to that point.
 +
 +
:[[File:sandia arm roadmap (2018).png|800px]]
  
 
== Overview ==
 
== Overview ==
Astra is the first [[ARM]]-based petascale supercomputer. The system consists of 5,184 [[Cavium]] [[ThunderX2 CN9975]] processors with a 1.2 MW power consumption for a peak performance of 2.322 [[petaFLOPS]]. Each [[ThunderX2 CN9975]] has [[28 cores]] operating at 2 GHz. There are also 108 36-port switches and 3 540-port spine switches.
+
Astra is the first [[ARM]]-based petascale supercomputer. The system consists of 5,184 [[Cavium]] [[ThunderX2 CN9975]] processors with slightly over 1.2 MW power consumption for a peak performance of 2.322 [[petaFLOPS]]. Each [[ThunderX2 CN9975]] has [[28 cores]] operating at 2 GHz. Astra has close around 700 [[terabytes]] of memory and uses a 3-level fat tree [[interconnect architecture|interconnect]].
  
 
<table class="wikitable">
 
<table class="wikitable">
<tr><th></th><td>Components</td></tr>
+
<tr><th></th><td>Components</td><td>&nbsp;</td><th></th><td colspan="2">Total Memory</td></tr>
<tr><th>Processors</th><td>5,184<br>2 x 72 x 36</td></tr>
+
<tr><th>Processors</th><td>5,184<br>2 x 72 x 36</td><td>&nbsp;</td><th>Type</th><td>[[DDR4]]</td><td>[[NVMe]]</td></tr>
<tr><th>Racks</th><td>36</td></tr>
+
<tr><th>Racks</th><td>36</td><td>&nbsp;</td><th>Node</th><td>128 GiB</td><td>?</td></tr>
<tr><th>Peak FLOPS</th><td>2.322 petaFLOPS</td></tr>
+
<tr><th>Peak FLOPS</th><td>2.322 petaFLOPS (DP)<br>4.644 petaFLOPS (SP)</td><td>&nbsp;</td><th>Astra</th><td>324 TiB</td><td>403 TB</td></tr>
 
</table>
 
</table>
  
Astra has close around 700 [[terabytes]] of memory.
+
== Architecture ==
 +
=== System ===
 +
Astra consists of 36 compute racks, 12 cooling racks, 3 networking racks, 2 storage racks, and a single utility rack.
 +
 
 +
:[[File:astra floorplan.svg|300px]]
 +
 
 +
:[[File:astra_supercomputer_illustration.png|500px]]
 +
 
 +
:[[File:astra 540-port switch.svg|thumb|right|540-port Switch]]
 +
 
 +
The system has a peak Wall power of 1.6 MW.
 +
 
 +
{| class="wikitable"
 +
|-
 +
! colspan="4" | Total Power (kW)
 +
|-
 +
! Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle
 +
|-
 +
| 1,631.5 || 1,440.3 || 1,357.3 || 274.9
 +
|}
 +
 
 +
Servers are linked via Mellanox IB EDR interconnect in a three-level fat tree topology with a 2:1 tapered fat-tree at L1. Astra uses three 540-port switches. Those are formed from 30 level 2 switches that provide 18 ports each (540 in total) with the remaining 18 links going for each of the 18 level 3 switches.
 +
 
 +
 
 +
:[[File:astra rack connection l1 to l2.svg|650px]]
 +
 
 +
=== Compute Rack ===
 +
Each of the 36 compute racks consist of 18 of HPE's Apollo 70 Chassis along with 3 36-port InfiniBand EDR switches. There is a single switch for every 6 chassis taking 24 ports.
 +
 
 +
: [[File:astra racks and cooling (annotated).png|500px]]
 +
[[File:astra rack connection.svg|thumb|right|Compute Node L1 links]]
 +
Every IB link from each of the 24 nodes go directly to the 36-port switch. There are 12 remaining ports that go to the level 2 switches (discussed later). A full rack with 72 nodes has 4,032 cores yielding a peak performance of 64.51 [[teraFLOPS]] with over 24.57 TB/s of peak bandwidth.
  
 
<table class="wikitable">
 
<table class="wikitable">
<tr><th colspan="4">Astra Total Memory</th></tr>
+
<tr><th colspan="5">Full Rack Capabilities</th></tr>
<tr><th>Type</th><td>[[DDR4]]</td><td>[[NVMe]]</td><tr>
+
<tr><th>Processors</th><td>144<br>72 × 2 × CPU</td></tr>
<tr><th>Node</th><td>128 GiB</td><td>?</td></tr>
+
<tr><th>Core</th><td>4,032 (16,128 threads)<br>72 × 56 (224 threads)</td></tr>
<tr><th>Astra</th><td>324 TiB</td><td>403 TB</td></tr>
+
<tr><th>FLOPS (SP)</th><td>129 TFLOPS<br>72 × 2 × 28 × 32 GFLOPS</td></tr>
 +
<tr><th>FLOPS (DP)</th><td>64.51 TFLOPS<br>72 × 2 × 28 × 16 GFLOPS</td></tr>
 +
<tr><th>Memory</th><td>9 TiB (DRR4)<br>72 × 2 × 8 × 8 GiB</td></tr>
 +
<tr><th>Memory BW</th><td>24.57 TB/s<br>72 × 16 × 21.33 GB/s</td></tr>
 
</table>
 
</table>
  
== Architecture ==
+
Each compute rack has a projected peak power of 1295.8 kW (1436.0 kW Wall) with a nominal 1217.0 kW of power under [[linpack]].
{{empty section}}
+
 
=== System ===
 
{{empty section}}
 
=== Compute Rack ===
 
{{empty section}}
 
 
=== Compute Node ===
 
=== Compute Node ===
{{empty section}}
+
[[File:apollo 70 system.png|right|thumb|Apollo 70]]
==== Socket ====
+
The basic compute server is the HPE Apollo 70. Those use a highly dense chassis system architecture that fit in just 2U and consist of four dual-socket compute nodes.
{{empty section}}
+
 
==== Full-node ====
+
:[[File:astra apollo 70 node.jpg|500px]]
{{empty section}}
+
 
 +
Each node has two 1,600 W power supplies, 1 Gbps Ethernet management port, and a Mellanox ConnectX-5 EDR link. Each node has a dual-socket [[Cavium]] {{cavium|ThunderX2}} [[ThunderX2 CN9975]] ({{cavium|Vulcan|l=arch}}) processor with 28 cores operating at 2 GHz. For Astra, Sandia is using 28-core parts operating at 2 GHz, likely due to a better performance/power efficiency design point. Each chip supports up to eight channels of DDR4 DIMMs with rates up to 2666 MT/s as well as 56 PCIe 3 lanes.
 +
 
 +
:[[File:astra node diagram.svg|600px]]
 +
 
 +
The [[ThunderX2 CN9975]] supports two-way [[multiprocessing]]. Communication is done over second-generation [[Cavium Coherent Processor Interconnect]] (CCPI2) which provides 600 Gbps of aggregated bandwidth. For the Astra supercomputer, each node uses 8 GiB DDR4-2666 dual-rank DIMM per controller for a total of 64 GiB and 170.7 GB/s of aggregated memory bandwidth per socket. For each node, there is a single Mellanox EDR InfiniBand ConnectX-5 VPI card designed for the Open Compute Project (OCP) providing the 100 Gb/s link.
 +
 
 +
With eight DIMMs per controller, each node has 128 GiB of memory feeding 56 cores with a total bandwidth of 341.33 GB/s per node. Those cores operate at up to 2 GHz, each with 2 NEON 128-bit units providing a peak theoretical performance of 8 single-precision FLOPS/cycle. This works out to 16 GFLOPS per core.
 +
 
 +
<table class="wikitable">
 +
<tr><th colspan="5">Full Node Capabilities</th></tr>
 +
<tr><th>&nbsp;</th><th>Socket</th><th>Node</th></tr>
 +
<tr><th>Processors</th><td>1 × CPU</td><td>2 × CPU</td></tr>
 +
<tr><th>Core</th><td>28 (112 threads)</td><td>56 (224 threads)</td></tr>
 +
<tr><th>FLOPS (SP)</th><td>896 GFLOPS<br>28 × 32 GFLOPS</td><td>1,792 GFLOPS<br>2 × 28 × 32 GFLOPS</td></tr>
 +
<tr><th>FLOPS (DP)</th><td>448 GFLOPS<br>28 × 16 GFLOPS</td><td>896 GFLOPS<br>2 × 28 × 16 GFLOPS</td></tr>
 +
<tr><th>Memory</th><td>64 GiB (DRR4)<br>8 × 8 GiB</td><td>128 GiB (DRR4)<br>2 × 8 × 8 GiB</td></tr>
 +
<tr><th>Bandwidth</th><td>170.7 GB/s<br>8 × 21.33 GB/s</td><td>341.33 GB/s<br>16 × 21.33 GB/s</td></tr>
 +
</table>
 +
 
 +
== Power & Cooling ==
 +
Astra has a nominal power consumption of 1.35 MW under [[LINPACK]]. Cooling the entire system are just 12 MCS-300 fan coil racks.
 +
{| class="wikitable"
 +
! colspan="11" | Projected power of the system by component
 +
|-
 +
! colspan="5" | Per constituent rack type (W) !! !! colspan="5" | Total (kW)
 +
|-
 +
! Rack || Wall !! Peak !! Nominal ([[LINPACK]]) !! Idle !! !! Racks !! style="background-color: yellow" |  '''Wall''' !! Peak !! style="background-color: yellow" |  '''Nominal (LINPACK)''' !! idle
 +
|-
 +
| Compute || 39,888 || 35,993 || 33,805 || 6,761 || || 36 || 1436.0 || 1295.8 || 1217.0 || 243.4
 +
|-
 +
| MCS-300 || 10,500 || 7,400 || 7,400 || 170 || || 12 || 126.0 || 88.8 || 88.8 || 2.0
 +
|-
 +
| Network || 12,624 || 10,023 || 9,021 || 9,021 || || 3 || 37.9 || 30.1 || 27.1 || 27.1
 +
|-
 +
| Storage || 11,520 || 10,000 || 10,000 || 1,000 || || 2 || 23.0 || 20.0 || 20.0 || 2.0
 +
|-
 +
| Utility || 8,640 || 5,625 || 4,500 || 450 || || 1 || 8.6 || 5.6 || 4.5 || 0.5
 +
|-
 +
| || || || || || || || style="background-color: yellow" | '''1,631.5''' || 1,440.3 || style="background-color: yellow" | '''1,357.3''' || 274.9
 +
|}
  
 
== Bibliography ==
 
== Bibliography ==
 
* SNL (personal communication, August 2018).
 
* SNL (personal communication, August 2018).
 
* DOE. (June 18, 2018). "''[https://share-ng.sandia.gov/news/resources/news_releases/arm_supercomputer/ Arm-based supercomputer prototype to be deployed at Sandia National Laboratories]''" [Press Release]
 
* DOE. (June 18, 2018). "''[https://share-ng.sandia.gov/news/resources/news_releases/arm_supercomputer/ Arm-based supercomputer prototype to be deployed at Sandia National Laboratories]''" [Press Release]
* Kevin Pedretti, Jim H. Laros III, Si Hammond. (June 28, 2018). "''Vanguard Astra: Maturing the ARM Software Ecosystem for U.S. DOE/ASC Supercomputing''"
+
* Kevin Pedretti, Jim H. Laros III, Si Hammond. (June 28, 2018). ''"ISC 2018"''. "''Vanguard Astra: Maturing the ARM Software Ecosystem for U.S. DOE/ASC Supercomputing''"
 
+
* Schor, David. (August, 2018). "''[https://fuse.wikichip.org/news/1583/cavium-takes-arm-to-petascale-with-astra/ Cavium Takes ARM to Petascale with Astra]''"
  
 
[[category:supercomputers]]
 
[[category:supercomputers]]

Latest revision as of 22:36, 2 June 2020

Edit Values
Astra
astra supercomputer illustration.png
General Info
SponsorsU.S. Department of Energy
DesignersCavium
OperatorsSandia National Laboratories
Introduction2018
Peak FLOPS2.322 petaFLOPS

Astra is a petascale ARM supercomputer operating by the DoE Sandia National Laboratories. Officially launching in late 2018, Astra became the first ARM-based supercomputer to exceed 1 petaFLOPS and the first ARM-based system to enter the TOP500 list.

History[edit]

Astra is an ARM-based supercomputer expected to be deployed at Sandia National Laboratories. The computer is one of a series of prototypes commissioned by the U.S. Department of Energy as part of a program that evaluates the feasibility of emerging high-performance computing architectures as production platforms to support NNSA's mission. Specifically, Astra is designed to demonstrate the viability of ARM for DOE NNSA Supercomputing. Astra is the fourth prototype as part of the Vanguard project initiative and is by far the largest ARM-based supercomputer designed to that point.

sandia arm roadmap (2018).png

Overview[edit]

Astra is the first ARM-based petascale supercomputer. The system consists of 5,184 Cavium ThunderX2 CN9975 processors with slightly over 1.2 MW power consumption for a peak performance of 2.322 petaFLOPS. Each ThunderX2 CN9975 has 28 cores operating at 2 GHz. Astra has close around 700 terabytes of memory and uses a 3-level fat tree interconnect.

Components Total Memory
Processors5,184
2 x 72 x 36
 TypeDDR4NVMe
Racks36 Node128 GiB?
Peak FLOPS2.322 petaFLOPS (DP)
4.644 petaFLOPS (SP)
 Astra324 TiB403 TB

Architecture[edit]

System[edit]

Astra consists of 36 compute racks, 12 cooling racks, 3 networking racks, 2 storage racks, and a single utility rack.

astra floorplan.svg
astra supercomputer illustration.png
540-port Switch

The system has a peak Wall power of 1.6 MW.

Total Power (kW)
Wall Peak Nominal (LINPACK) Idle
1,631.5 1,440.3 1,357.3 274.9

Servers are linked via Mellanox IB EDR interconnect in a three-level fat tree topology with a 2:1 tapered fat-tree at L1. Astra uses three 540-port switches. Those are formed from 30 level 2 switches that provide 18 ports each (540 in total) with the remaining 18 links going for each of the 18 level 3 switches.


astra rack connection l1 to l2.svg

Compute Rack[edit]

Each of the 36 compute racks consist of 18 of HPE's Apollo 70 Chassis along with 3 36-port InfiniBand EDR switches. There is a single switch for every 6 chassis taking 24 ports.

astra racks and cooling (annotated).png
Compute Node L1 links

Every IB link from each of the 24 nodes go directly to the 36-port switch. There are 12 remaining ports that go to the level 2 switches (discussed later). A full rack with 72 nodes has 4,032 cores yielding a peak performance of 64.51 teraFLOPS with over 24.57 TB/s of peak bandwidth.

Full Rack Capabilities
Processors144
72 × 2 × CPU
Core4,032 (16,128 threads)
72 × 56 (224 threads)
FLOPS (SP)129 TFLOPS
72 × 2 × 28 × 32 GFLOPS
FLOPS (DP)64.51 TFLOPS
72 × 2 × 28 × 16 GFLOPS
Memory9 TiB (DRR4)
72 × 2 × 8 × 8 GiB
Memory BW24.57 TB/s
72 × 16 × 21.33 GB/s

Each compute rack has a projected peak power of 1295.8 kW (1436.0 kW Wall) with a nominal 1217.0 kW of power under linpack.

Compute Node[edit]

Apollo 70

The basic compute server is the HPE Apollo 70. Those use a highly dense chassis system architecture that fit in just 2U and consist of four dual-socket compute nodes.

astra apollo 70 node.jpg

Each node has two 1,600 W power supplies, 1 Gbps Ethernet management port, and a Mellanox ConnectX-5 EDR link. Each node has a dual-socket Cavium ThunderX2 ThunderX2 CN9975 (Vulcan) processor with 28 cores operating at 2 GHz. For Astra, Sandia is using 28-core parts operating at 2 GHz, likely due to a better performance/power efficiency design point. Each chip supports up to eight channels of DDR4 DIMMs with rates up to 2666 MT/s as well as 56 PCIe 3 lanes.

astra node diagram.svg

The ThunderX2 CN9975 supports two-way multiprocessing. Communication is done over second-generation Cavium Coherent Processor Interconnect (CCPI2) which provides 600 Gbps of aggregated bandwidth. For the Astra supercomputer, each node uses 8 GiB DDR4-2666 dual-rank DIMM per controller for a total of 64 GiB and 170.7 GB/s of aggregated memory bandwidth per socket. For each node, there is a single Mellanox EDR InfiniBand ConnectX-5 VPI card designed for the Open Compute Project (OCP) providing the 100 Gb/s link.

With eight DIMMs per controller, each node has 128 GiB of memory feeding 56 cores with a total bandwidth of 341.33 GB/s per node. Those cores operate at up to 2 GHz, each with 2 NEON 128-bit units providing a peak theoretical performance of 8 single-precision FLOPS/cycle. This works out to 16 GFLOPS per core.

Full Node Capabilities
 SocketNode
Processors1 × CPU2 × CPU
Core28 (112 threads)56 (224 threads)
FLOPS (SP)896 GFLOPS
28 × 32 GFLOPS
1,792 GFLOPS
2 × 28 × 32 GFLOPS
FLOPS (DP)448 GFLOPS
28 × 16 GFLOPS
896 GFLOPS
2 × 28 × 16 GFLOPS
Memory64 GiB (DRR4)
8 × 8 GiB
128 GiB (DRR4)
2 × 8 × 8 GiB
Bandwidth170.7 GB/s
8 × 21.33 GB/s
341.33 GB/s
16 × 21.33 GB/s

Power & Cooling[edit]

Astra has a nominal power consumption of 1.35 MW under LINPACK. Cooling the entire system are just 12 MCS-300 fan coil racks.

Projected power of the system by component
Per constituent rack type (W) Total (kW)
Rack Wall Peak Nominal (LINPACK) Idle Racks Wall Peak Nominal (LINPACK) idle
Compute 39,888 35,993 33,805 6,761 36 1436.0 1295.8 1217.0 243.4
MCS-300 10,500 7,400 7,400 170 12 126.0 88.8 88.8 2.0
Network 12,624 10,023 9,021 9,021 3 37.9 30.1 27.1 27.1
Storage 11,520 10,000 10,000 1,000 2 23.0 20.0 20.0 2.0
Utility 8,640 5,625 4,500 450 1 8.6 5.6 4.5 0.5
1,631.5 1,440.3 1,357.3 274.9

Bibliography[edit]

designerCavium +
introductory date2018 +
main imageFile:astra supercomputer illustration.png +
nameAstra +
operatorSandia National Laboratories +
peak flops (double-precision)2.322e+15 FLOPS (2,322,000,000,000 KFLOPS, 2,322,000,000 MFLOPS, 2,322,000 GFLOPS, 2,322 TFLOPS, 2.322 PFLOPS, 0.00232 EFLOPS, 2.322e-6 ZFLOPS) +
sponsorUnited States Department of Energy (DOE) +