From WikiChip
Editing brain floating-point format

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

This page supports semantic in-text annotations (e.g. "[[Is specified as::World Heritage Site]]") to build structured and queryable content provided by Semantic MediaWiki. For a comprehensive description on how to use annotations or the #ask parser function, please have a look at the getting started, in-text annotation, or inline queries help pages.

Latest revision Your text
Line 1: Line 1:
 
{{title|Brain floating-point format (bfloat16)}}
 
{{title|Brain floating-point format (bfloat16)}}
'''Brain floating-point format''' ('''bfloat16''' or '''BF16''') is a [[number encoding format]] occupying 16 bits representing a floating-point number. It is equivalent to a standard [[single-precision floating-point]] value with a truncated [[mantissa field]]. Bfloat16 is designed to be used in hardware [[accelerating]] machine learning algorithms. Bfloat was first proposed and implemented by [[Google]] with [[Intel]] supporting it in their FPGAs, Nervana [[neural processors]], and CPUs.
+
'''Brain floating-point format''' ('''bfloat16''') is an [[number encoding format]] occupying 16 bits representing a floating-point number. It is equvilent to a standard [[single-precision floating-point]] value with a truncated [[mantissa field]]. Bfloat16 is designed to be used in hardware [[accelerating]] machine learning algorithms. Bfloat was first proposed and implemented by [[Google]] with [[Intel]] supporting it in their FPGAs, Nervana [[neural processors]], and CPUs.
  
 
== Overview ==
 
== Overview ==
Bfloat16 follows the same format as a standard IEEE 754 [[single-precision floating-point]] but truncates the [[mantissa field]] from 23 bits to just 7 bits. Preserving the exponent bits keeps the format to the same range as the 32-bit single precision FP (~1e<sup>-38</sup> to ~3e<sup>38</sup>). This allows for relatively simpler conversion between the two data types. In other words, while some resolution is lost, numbers can still be represented. [[Microsoft]] developed a similar format for an {{microsoft|msfp8|8-bit floating point}} based on the float16 range.
+
Bfloat16 follows the same format as a standard IEEE 754 [[single-precision floating-point]] but truncates the [[mantissa field]] from 23 bits to just 7 bits. Preserving the exponent bits keeps the format to the same range as the 32-bit single precision FP (~1e<sup>-38</sup> to ~3e<sup>38</sup>). This allows for relatively simpler conversion between the two data types. In other words, while some resolution is lost, numbers can still be represented.
 
 
 
 
:'''float32''': (range ~1e<sup>-38</sup> to ~3e<sup>38</sup>)
 
 
 
::[[File:float32 encoding format.svg|700px]]
 
 
 
 
 
:'''float16''': (range ~5.96e<sup>-8</sup> to 65,504)
 
 
 
::[[File:float16 encoding format.svg|350px]]
 
 
 
 
 
:'''bfloat16''': (range ~1e<sup>-38</sup> to ~3e<sup>38</sup>)
 
 
 
::[[File:bfloat16 encoding format.svg|350px]]
 
  
 
== Motivation ==
 
== Motivation ==
 
The motivation behind the reduced mantissa is derived from Google's experiments that showed that it is fine to reduce the mantissa so long it's still possible to represent tiny values closer to zero as part of the summation of small differences during training. Smaller mantissa brings a number of other advantages such as reducing the multiplier power and physical silicon area.
 
The motivation behind the reduced mantissa is derived from Google's experiments that showed that it is fine to reduce the mantissa so long it's still possible to represent tiny values closer to zero as part of the summation of small differences during training. Smaller mantissa brings a number of other advantages such as reducing the multiplier power and physical silicon area.
  
* float32: 24<sup>2</sup>=576 (100%)
+
* float32: 23<sup>2</sup>=529
* float16: 11<sup>2</sup>=121 (21%)
+
* float16: 10<sup>2</sup>=100
* bfloat16: 8<sup>2</sup>=64 (11%)
+
* bfloat16: 7<sup>2</sup>=49
  
 
From the above, it can be seen that there is a factor of two and a factor of ten in terms of the number of bits that will flip (or components). From an efficiency standpoint, the gain was worth it for Google and their {{google|TPU}} implementation. At the system level, the benefits in memory capacity saving and bandwidth can also be realized if desired.
 
From the above, it can be seen that there is a factor of two and a factor of ten in terms of the number of bits that will flip (or components). From an efficiency standpoint, the gain was worth it for Google and their {{google|TPU}} implementation. At the system level, the benefits in memory capacity saving and bandwidth can also be realized if desired.
Line 32: Line 17:
  
 
== Hardware support ==
 
== Hardware support ==
=== CPUs ===
+
* Google's {{google|TPU}}
* [[Intel]] {{intel|Cooper Lake|l=arch}}, {{intel|Ice Lake (Server)|l=arch}}
+
* Intel's {{nervana|NPU}}, {{intel|Cooper Lake|l=arch}}
* [[IBM]] {{ibm|POWER10|l=arch}}
 
* [[Arm]] {{armh|Neoverse V1|l=arch}}, {{armh|Neoverse N2|l=arch}}
 
 
 
=== NPUs ===
 
{{see also|neural processors}}
 
* [[Arm]] {{arm|Project Trillium|l=arch}}
 
* [[Centaur Technology]] {{centtech|CHA|CHA (NCORE)|l=arch}}
 
* [[Flex Logix]] [[InferX]]
 
* [[Google]] {{google|TPU}}
 
* [[Habana]] {{habana|HL}}
 
* [[Intel]] {{nervana|NNP}}
 
* [[Wave Computing]] {{wave|DPU}}
 
 
 
== ISAs ==
 
* Power: [[Power ISA v3.1]]
 
* x86-64: [[AVX512_BF16]] (part of ''{{intel|DL Boost}} technology'')
 
* AArch64: [[ARMv8.6-BF16]]
 
  
 
== See also ==
 
== See also ==
* {{microsoft|MSFP8|MSFP8-11}}
 
 
* {{intel|Flexpoint}}
 
* {{intel|Flexpoint}}
  
 
== Bibliography ==
 
== Bibliography ==
* Cliff Young, Google AI. (October 2018). "''Codesign in Google TPUs: Inference, Training, Performance, and Scalability''". Keynote speech, Processor Conference 2018.
+
* Cliff Young, Google AI. (October 2018). "''Codesign in Google TPUs: Inference, Training, Performance, and Scalability''". Keynote speech, Processor Conference 2018.
* Carey Kloss, Intel VP Hardware and AI Products Group. (April 2019). "Deep Learning By Design; Building silicon for AI." Processor Conference 2019.
 

Please note that all contributions to WikiChip may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see WikiChip:Copyrights for details). Do not submit copyrighted work without permission!

Cancel | Editing help (opens in new window)