From WikiChip
ZettaScaler

ZettaScaler is a series of Japanese supercomputers using processors designed by PEZY and liquid cooling systems designed by ExaScaler.

Overview[edit]

ZettaScaler supercomputers are constructed using dense server aggregates called 'Bricks'. The system is cooled using enclosures containing a number of bricks. The cooling system is designed by ExaScaler and makes use of 3M's Fluorinert Electronic liquid which is an electrically insulating fluorocarbon-based inert liquid.

ZettaScalers use a host CPU (typically a Xeon E5) which is then used to offload work to the PEZY-SCx processors which act as GPDSP/GPGPU accelerators using OpenCL-like programming called PZCL.

Collaboration[edit]

The systems are a joint collaboration between three related start-ups:

  • PEZY - Microprocessor design firm that designs the PEZY-SCx series.
  • Exascaler - The firm that designs the water cooling solutions
  • UltraMemory - A firm that designs TCI-based DRAM

Systems[edit]

ZettaScaler-1.x[edit]

ZettaScaler-1.x (ZS-1.x) is based on the PEZY-SC. Each node (brick) was constructed using two Xeon E5 connected using Intel's QPI along with 4 set of dual PEZY-SC chips in PCIe card slots using InfiniBand FDR HCA. The use of the general-purpose Intel chips was to reduce cost and speedup development cycle. The original ZS-1.0 used an off-the-shelf SuperMicro board in order to speed-up the development cycle. Starting with the ZS-1.4, PEZY moved to a custom "brick".

With the introduction of the ZS-1.6, PEZY redesigned the package of the PEZY-SC. The new chip, PEZY-SCnp (NP for New Package), addressed a number of issues relating to signal quality (DRAM, and PCIe frequency failure).

System ZettaScaler-1.0 ZettaScaler-1.4 ZettaScaler-1.5 ZettaScaler-1.6
Introduction October 2014 April 2015 October 2015 April 2016
Board SuperMicro Board Brick
Host 2x Xeon E5-2660 v2 2x Xeon E5-2618L v3
Chip PEZY-SC PEZY-SCnp
Memory DDR3 256 GB DDR4 64 GB DDR4 128 GB
TOP500 Suiren:
187 TFLOPS (Rank 369) 2014/11
206 TFLOPS (Rank 366) 2015/06
Shoubu:
412 TFLOPS (Rank 169) 2015/06
SuirenBlue:
194 TFLOPS (Rank 392) 2015/06
Shoubu:
1,001 TFLOPS (Rank 94) 2016/06
Satsuki: 290 TFLOPS (Rank 486) 2016/06
Green500 Suiren:
4.95 GFLOPS/W (Rank 2) 2014/11
6.22 GFLOPS/W (Rank 3)
Shoubu:
7.03 GFLOPS/W (Rank 1) 2015/06
SuirenBlue: 6.84 GFLOPS/W (Rank 2) 2015/06
Shoubu:
6.67 GFLOPS/W (Rank 1) 2016/06
Satsuki 6.20 GFLOPS/W (Rank 2) 2016/06

ZettaScaler-2.x[edit]

With the introduction of the ZettaScaler-2.x, PEZY has redesigned the Brick. PEZY moved to higher integration through advanced 3D packaging technology using ThruChip Interface (TCI) to interconnect the DRAM to the processor which uses wireless near-field inductive coupling instead of the traditional TSV. Powering the ZS-2.x computers is the PEZY-SC2 which incorporates 2,048 cores.

Module:
pezy-sc2 module.png
Brick:
pezy-sc2 brick.png

On the ZS-2.x, a single module contains a beefier 48V power supply. The module is measured 160mm (W) x 100mm (D) x 28 mm (H). Each brick contains 32 modules.

zettascaler tanks.png

16 such bricks go into a single tank.

ZettaScaler-2.2[edit]

ZS-2.2 Enclosure

The ZettaScaler-2.2 (ZS-2.2) architecture was introduced to the Top500 list in November 2017 with multiple systems being upgraded.

The highest-performing system was the Gyoukou supercomputer which officially peaked at an efficiency of 14.173 GFLOPS/W, surpassing the previous record of 14.11 held by the TSUBAME 3.0. The systems features 10,000 PEZY-SC2 chips, however in order to improve yield they each have 1,984 cores enabled instead of the expected 2,048 for a total of 19,840,000 cores. In the ZS-2.2, the SC2 chips are clocked at 700 MHz instead of their announced frequency of 1 GHz. For every eight SC2 chips, there is a single 16-core Xeon D processor for a combined total of 1250 Xeon processors and 20,000 cores. The Gyoukou ZS-2.2 has a Linpack performance of 19,135.8 TFLOPS with a theoretical peak performance of 28.192.0 PFLOPS consuming 1,350 kW. With 7,056 PEZY-SC2 chips, only 19.53 immersion tanks are filled out of the 26 tanks available, leaving a lot of room to grow. Additionally, with 19,860,000 cores (19,840,000 PEZY + 20,000 x86), the ZettaScaler-2.2 surpassed the Chinese supercomputer Sunway TaihuLight as the highest core-count supercomputer by over 3 million cores.

In addition to the Gyoukou supercomputer, three other smaller Japanese supercomputers based on the ZettaScaler-2.2 were introduced - all three beating the Gyoukou in efficiency, reaching the top three spot on the Green500 list:

Top500/Green500 November 2017 Ranking
Green500
Rank
Top500
Rank
System Cores Rmax Power Efficiency
1 259 Shoubu system B 794,400 842.0 TFLOPS 50 kW 17.009 GFLOPS/W
50x (8x PEZY-SC2 + 1x Xeon D-1571)
2 307 Suiren2 762,624 788.2 TFLOPS 47 kW 16.759 GFLOPS/W
48x (8x PEZY-SC2 + 1x Xeon D-1571)
3 276 Sakura 794,400 824.7 TFLOPS 50 kW 16.657 GFLOPS/W
50x (8x PEZY-SC2 + 2x Xeon E5-2618L v3)
5 4 Gyoukou 19,860,000 19,135.8 TFLOPS 1,350 kW 14.173 GFLOPS/W
1250x (8x PEZY-SC2 + 1x Xeon D-1571)

References[edit]