ZettaScaler is a series of Japanese supercomputers using processors designed by PEZY and liquid cooling systems designed by ExaScaler.

Overview

ZettaScaler supercomputers are constructed using dense server aggregates called 'Bricks'. The system is cooled using enclosures containing a number of bricks. The cooling system is designed by ExaScaler and makes use of 3M's Fluorinert Electronic liquid which is an electrically insulating fluorocarbon-based inert liquid.

ZettaScalers use a host CPU (typically a Xeon E5) which is then used to offload work to the PEZY-SCx processors which act as GPDSP/GPGPU accelerators using OpenCL-like programming called PZCL.

ZettaScaler-1.x

ZettaScaler-1.x (ZS-1.x) is based on the PEZY-SC. Each node (brick) was constructed using two Xeon E5 connected using Intel's QPI along with 4 set of dual PEZY-SC chips in PCIe card slots using InfiniBand FDR HCA. The use of the general-purpose Intel chips was to reduce cost and speedup development cycle. The original ZS-1.0 used an off-the-shelf SuperMicro board in order to speed-up the development cycle. Starting with the ZS-1.4, PEZY moved to a custom "brick".

With the introduction of the ZS-1.6, PEZY redesigned the package of the PEZY-SC. The new chip, PEZY-SCnp (NP for New Package), addressed a number of issues relating to signal quality (DRAM, and PCIe frequency failure).

System	ZettaScaler-1.0	ZettaScaler-1.4	ZettaScaler-1.5	ZettaScaler-1.6
Introduction	October 2014	April 2015	October 2015	April 2016
Board	SuperMicro Board	Brick
Host	2x Xeon E5-2660 v2	2x Xeon E5-2618L v3
Chip	PEZY-SC			PEZY-SCnp
Memory	DDR3 256 GB	DDR4 64 GB	DDR4 128 GB
TOP500	Suiren: 187 TFLOPS (Rank 369) 2014/11 206 TFLOPS (Rank 366) 2015/06	Shoubu: 412 TFLOPS (Rank 169) 2015/06 SuirenBlue: 194 TFLOPS (Rank 392) 2015/06		Shoubu: 1,001 TFLOPS (Rank 94) 2016/06 Satsuki: 290 TFLOPS (Rank 486) 2016/06
Green500	Suiren: 4.95 GFLOPS/W (Rank 2) 2014/11 6.22 GFLOPS/W (Rank 3)	Shoubu: 7.03 GFLOPS/W (Rank 1) 2015/06 SuirenBlue: 6.84 GFLOPS/W (Rank 2) 2015/06		Shoubu: 6.67 GFLOPS/W (Rank 1) 2016/06 Satsuki 6.20 GFLOPS/W (Rank 2) 2016/06

ZettaScaler-2.x

With the introduction of the ZettaScaler-2.x, PEZY has redesigned the Brick. PEZY moved to higher integration through advanced 3D packaging technology using ThruChip Interface (TCI) to interconnect the DRAM to the processor which uses wireless near-field inductive coupling instead of the traditional TSV. Powering the ZS-2.x computers is the PEZY-SC2 which incorporates 2,048 cores.

Module:

Brick:

On the ZS-2.x, a single module contains a beefier 48V power supply. The module is measured 160mm (W) x 100mm (D) x 28 mm (H). Each brick contains 32 modules.

16 such bricks go into a single tank.

ZettaScaler-2.2

ZS-2.2 Enclosure

The ZettaScaler-2.2 (ZS-2.2) was introduced to the Top500 list in November 2017. The system also peaked at number one on the Green500 list with an efficienct performance of 14.69 GFLOPS/W, surpassing the previous record of 14.11 held by the TSUBAME 3.0. The systems features 7,056 PEZY-SC2 chips, however in order to improve yield they each have 1,984 cores enabled instead of the expect 2,048 for a total of 13,999,104 cores. In the ZS-2.2, the SC2 chips are clocked at 700 MHz instead of their announced frequency of 1 GHz For every eight SC2 chips, there is a single 16-core Xeon D processor for a combined total of 882 Xeon processors and 14,112 cores. The ZS-2.2 has a Linpack performance of 14.13 PFLOPS with a theoretical peak performance of 19.89 PFLOPS consuming 962.3kW. With 7,056 PEZY-SC2 chips, only 13.8 immersion tanks are filled out of the 26 tanks available, leaving a lot of room to grow. Additionally, with 14,013,216 cores (13,999,104 PEZY + 14,112 x86), the ZettaScaler-2.2 surpassed the Chinese supercomputer Sunway TaihuLight as the highest core-count supercomputer by over 3 million cores.

References

• Some information was obtained directly from PEZY
• 3M Fluorinert Helps PEZY/ExaScaler Rise to the Top of the Green500 List