ZettaScaler is a series of Japanese supercomputers using processors designed by PEZY and liquid cooling systems designed by ExaScaler.

Overview

ZettaScaler supercomputers are constructed using dense server aggregates called 'Bricks'. The system is cooled using enclosures containing a number of bricks. The cooling system is designed by ExaScaler and makes use of 3M's Fluorinert Electronic liquid which is an electrically insulating fluorocarbon-based inert liquid.

ZettaScalers use a host CPU (typically a Xeon E5) which is then used to offload work to the PEZY-SCx processors which act as GPDSP/GPGPU accelerators using OpenCL-like programming called PZCL.

Collaboration

The systems are a joint collaboration between three related start-ups:

• PEZY - Microprocessor design firm that designs the PEZY-SCx series.
• Exascaler - The firm that designs the water cooling solutions
• UltraMemory - A firm that designs TCI-based DRAM

ZettaScaler-1.x

ZettaScaler-1.x (ZS-1.x) is based on the PEZY-SC. Each node (brick) was constructed using two Xeon E5 connected using Intel's QPI along with 4 set of dual PEZY-SC chips in PCIe card slots using InfiniBand FDR HCA. The use of the general-purpose Intel chips was to reduce cost and speedup development cycle. The original ZS-1.0 used an off-the-shelf SuperMicro board in order to speed-up the development cycle. Starting with the ZS-1.4, PEZY moved to a custom "brick".

With the introduction of the ZS-1.6, PEZY redesigned the package of the PEZY-SC. The new chip, PEZY-SCnp (NP for New Package), addressed a number of issues relating to signal quality (DRAM, and PCIe frequency failure).

System	ZettaScaler-1.0	ZettaScaler-1.4	ZettaScaler-1.5	ZettaScaler-1.6
Introduction	October 2014	April 2015	October 2015	April 2016
Board	SuperMicro Board	Brick
Host	2x Xeon E5-2660 v2	2x Xeon E5-2618L v3
Chip	PEZY-SC			PEZY-SCnp
Memory	DDR3 256 GB	DDR4 64 GB	DDR4 128 GB
TOP500	Suiren: 187 TFLOPS (Rank 369) 2014/11 206 TFLOPS (Rank 366) 2015/06	Shoubu: 412 TFLOPS (Rank 169) 2015/06 SuirenBlue: 194 TFLOPS (Rank 392) 2015/06		Shoubu: 1,001 TFLOPS (Rank 94) 2016/06 Satsuki: 290 TFLOPS (Rank 486) 2016/06
Green500	Suiren: 4.95 GFLOPS/W (Rank 2) 2014/11 6.22 GFLOPS/W (Rank 3)	Shoubu: 7.03 GFLOPS/W (Rank 1) 2015/06 SuirenBlue: 6.84 GFLOPS/W (Rank 2) 2015/06		Shoubu: 6.67 GFLOPS/W (Rank 1) 2016/06 Satsuki 6.20 GFLOPS/W (Rank 2) 2016/06

ZettaScaler-2.x

With the introduction of the ZettaScaler-2.x, PEZY has redesigned the Brick. PEZY moved to higher integration through advanced 3D packaging technology using ThruChip Interface (TCI) to interconnect the DRAM to the processor which uses wireless near-field inductive coupling instead of the traditional TSV. Powering the ZS-2.x computers is the PEZY-SC2 which incorporates 2,048 cores.

Module:

Brick:

On the ZS-2.x, a single module contains a beefier 48V power supply. The module is measured 160mm (W) x 100mm (D) x 28 mm (H). Each brick contains 32 modules.

16 such bricks go into a single tank.

ZettaScaler-2.2

ZS-2.2 Enclosure

The ZettaScaler-2.2 (ZS-2.2) architecture was introduced to the Top500 list in November 2017 with multiple systems being upgraded.

The highest-performing system was the Gyoukou supercomputer which officially peaked at an efficiency of 14.173 GFLOPS/W, surpassing the previous record of 14.11 held by the TSUBAME 3.0. The systems features 10,000 PEZY-SC2 chips, however in order to improve yield they each have 1,984 cores enabled instead of the expected 2,048 for a total of 19,840,000 cores. In the ZS-2.2, the SC2 chips are clocked at 700 MHz instead of their announced frequency of 1 GHz. For every eight SC2 chips, there is a single 16-core Xeon D processor for a combined total of 1250 Xeon processors and 20,000 cores. The Gyoukou ZS-2.2 has a Linpack performance of 19,135.8 TFLOPS with a theoretical peak performance of 28.192.0 PFLOPS consuming 1,350 kW. With 7,056 PEZY-SC2 chips, only 19.53 immersion tanks are filled out of the 26 tanks available, leaving a lot of room to grow. Additionally, with 19,860,000 cores (19,840,000 PEZY + 20,000 x86), the ZettaScaler-2.2 surpassed the Chinese supercomputer Sunway TaihuLight as the highest core-count supercomputer by over 3 million cores.

In addition to the Gyoukou supercomputer, three other smaller Japanese supercomputers based on the ZettaScaler-2.2 were introduced - all three beating the Gyoukou in efficiency, reaching the top three spot on the Green500 list:

Top500/Green500 November 2017 Ranking
Green500 Rank	Top500 Rank	System	Cores	Rmax	Power	Efficiency
1	259	Shoubu system B	794,400	842.0 TFLOPS	50 kW	17.009 GFLOPS/W
			50x (8x PEZY-SC2 + 1x Xeon D-1571)
2	307	Suiren2	762,624	788.2 TFLOPS	47 kW	16.759 GFLOPS/W
			48x (8x PEZY-SC2 + 1x Xeon D-1571)
3	276	Sakura	794,400	824.7 TFLOPS	50 kW	16.657 GFLOPS/W
			50x (8x PEZY-SC2 + 2x Xeon E5-2618L v3)
5	4	Gyoukou	19,860,000	19,135.8 TFLOPS	1,350 kW	14.173 GFLOPS/W
			1250x (8x PEZY-SC2 + 1x Xeon D-1571)

References

• Some information was obtained directly from PEZY
• 3M Fluorinert Helps PEZY/ExaScaler Rise to the Top of the Green500 List