Semiconductor & Computer Engineering
ZettaScaler is a series of Japanese supercomputers using processors designed by PEZY and liquid cooling systems designed by ExaScaler.
Overview
ZettaScaler supercomputers are constructed using dense server aggregates called 'Bricks'. The system is cooled using enclosures containing a number of bricks. The cooling system is designed by ExaScaler and makes use of 3M's Fluorinert Electronic liquid which is an electrically insulating fluorocarbon-based inert liquid.
ZettaScalers use a host CPU (typically a Xeon E5) which is then used to offload work to the PEZY-SCx processors which act as GPDSP/GPGPU accelerators using OpenCL-like programming called PZCL.
ZettaScaler-1.x
ZettaScaler-1.x (ZS-1.x) is based on the PEZY-SC. Each node (brick) was constructed using two Xeon E5 connected using Intel's QPI along with 4 set of dual PEZY-SC chips in PCIe card slots using InfiniBand FDR HCA. The use of the general-purpose Intel chips was to reduce cost and speedup development cycle. The original ZS-1.0 used an off-the-shelf SuperMicro board in order to speed-up the development cycle. Starting with the ZS-1.4, PEZY moved to a custom "brick".
With the introduction of the ZS-1.6, PEZY redesigned the package of the PEZY-SC. The new chip, PEZY-SCnp (NP for New Package), addressed a number of issues relating to signal quality (DRAM, and PCIe frequency failure).
| System | ZettaScaler-1.0 | ZettaScaler-1.4 | ZettaScaler-1.5 | ZettaScaler-1.6 |
|---|---|---|---|---|
| Introduction | October 2014 | April 2015 | October 2015 | April 2016 |
| Board | SuperMicro Board | Brick | ||
| Host | 2x Xeon E5-2660 v2 | 2x Xeon E5-2618L v3 | ||
| Chip | PEZY-SC | PEZY-SCnp | ||
| Memory | DDR3 256 GB | DDR4 64 GB | DDR4 128 GB | |
| TOP500 | Suiren: 187 TFLOPS (Rank 369) 2014/11 206 TFLOPS (Rank 366) 2015/06 |
Shoubu: 412 TFLOPS (Rank 169) 2015/06 SuirenBlue: 194 TFLOPS (Rank 392) 2015/06 |
Shoubu: 1,001 TFLOPS (Rank 94) 2016/06 Satsuki: 290 TFLOPS (Rank 486) 2016/06 | |
| Green500 | Suiren: 4.95 GFLOPS/W (Rank 2) 2014/11 6.22 GFLOPS/W (Rank 3) |
Shoubu: 7.03 GFLOPS/W (Rank 1) 2015/06 SuirenBlue: 6.84 GFLOPS/W (Rank 2) 2015/06 |
Shoubu: 6.67 GFLOPS/W (Rank 1) 2016/06 Satsuki 6.20 GFLOPS/W (Rank 2) 2016/06 | |
ZettaScaler-2.x
With the introduction of the ZettaScaler-2.x, PEZY has redesigned the Brick. PEZY moved to higher integration through advanced 3D packaging technology using ThruChip Interface (TCI) to interconnect the DRAM to the processor which uses wireless near-field inductive coupling instead of the traditional TSV. Powering the ZS-2.x computers is the PEZY-SC2 which incorporates 2,048 cores.
On the ZS-2.x, a single module contains a beefier 48V power supply. The module is measured 160mm (W) x 100mm (D) x 28 mm (H). Each brick contains 32 modules.
16 such bricks go into a single tank.
ZettaScaler-2.2
The ZettaScaler-2.2 (ZS-2.2) was introduced to the Top500 list in November 2017. The system also peaked at number one on the Green500 list with an efficienct performance of 14.69 GFLOPSw/, surpassing the previous record of 14.11 held by the TSUBAME 3.0. The systems features 7,056 PEZY-SC2 chips, however in order to improve yield they each have 1,984 cores enabled instead of the expect 2,048 for a total of 13,999,104 cores. In the ZS-2.2, the SC2 chips are clocked at 700 MHz instead of their announced frequency of 1 GHz. The ZS-2.2 has a Linpack performance of 14.13 PFLOPS with a theoretical peak performance of 19.89 PFLOPS consuming 962.3kW. With 7,056 PEZY-SC2 chips, only 13.8 immersion tanks are filled out of the 26 tanks available, leaving a lot of room to grow.
References
- Some information was obtained directly from PEZY
- 3M Fluorinert Helps PEZY/ExaScaler Rise to the Top of the Green500 List