FP16

The FP32 article covered the traditional, full-precision baseline. This article covers the first, most direct step away from it — halving the memory footprint, but introducing a genuine, real risk: FP16.

The simple definition

FP16, or 16-bit floating point (also called half precision), uses half the bits of FP32 to represent a number — 16 bits, or exactly 2 bytes, instead of FP32’s 4 bytes — cutting memory requirements in half at the cost of both precision and range. Recall from the FP32 article’s structure: 1 sign bit, 8 exponent bits, 23 mantissa bits. FP16 restructures this into 1 sign bit, 5 exponent bits, and 10 mantissa bits — a genuinely more compact format, but one that sacrifices real capability to get there.

Why halving both the exponent and mantissa creates a genuine, real problem

Recall from the FP32 article that the exponent bits control a number’s dynamic range — how large or small a value can be represented at all. FP16’s exponent shrinks from 8 bits down to just 5, dramatically narrowing that range to roughly 6×10⁻⁵ up to 6×10⁴. Recall from the Gradient article’s core mechanism — during training, gradients can genuinely become extremely small or, in less stable moments, extremely large.

If a value produced during training falls outside FP16’s narrow representable range, it doesn’t just lose precision — it can overflow to infinity or underflow to zero entirely, a real, catastrophic failure that can silently corrupt or outright crash a training run.

flowchart LR
    A["FP16: 1 sign + 5 exponent + 10 mantissa bits"] --> B[Half the memory of FP32]
    B --> C["Real risk: narrow range (~6e-5 to 6e4) can overflow or underflow during training"]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a bathroom scale designed to measure weights only between 1 and 500 pounds, with excellent precision within that specific range — genuinely useful for its intended purpose, but if something weighing 800 pounds or 0.1 pounds needs measuring, the scale simply can’t represent that value at all, rather than just being slightly less accurate about it.

Where this breaks down: A bathroom scale’s limited range is an obvious, visible design constraint anyone using it understands upfront. FP16’s narrow range operates silently, deep inside a training process’s numerical calculations — a value overflowing or underflowing can corrupt training in ways that aren’t immediately obvious, which is precisely why this became a real, documented, and frustrating problem for early researchers who adopted FP16 without accounting for it.

Why this specific, real limitation directly led to BF16’s creation

This connects directly to the next article’s origin story. Recall from the TPU article’s design philosophy — Google engineers recognized that neural networks genuinely need FP32’s wide dynamic range far more than they need FP32’s fine-grained precision. FP16 got this trade-off backwards for AI’s specific needs, preserving reasonable precision while sacrificing the wide range that actually mattered more for numerical stability during training — a real, structural mismatch that directly motivated Google Brain’s alternative format, covered in full in the very next article.

A concrete example, layered

For a simple beginner example: a model trained naively in pure FP16 might work correctly for many steps, then suddenly produce NaN (Not a Number) errors partway through training, when a gradient calculation happens to produce a value too small or too large for FP16’s narrow range to represent — a real, frustrating, and hard-to-diagnose failure mode.

For a production example: a 7-billion-parameter model that requires roughly 28 gigabytes to store its weights in FP32 needs only about 14 gigabytes in FP16 — real, exact savings that directly determine whether a model fits on a given GPU’s VRAM, covered in its own article earlier in this phase — and Nvidia’s mixed-precision training techniques, real and widely documented, use FP16 for the bulk of this memory-efficient computation, while selectively keeping certain sensitive calculations — like accumulating gradients — in full FP32 precision specifically to avoid this exact overflow and underflow problem.

Why FP16 still remains genuinely useful despite this real risk

It’s worth being fair to FP16 here, not presenting it as simply broken. For inference — running an already-trained model, rather than training one from scratch — FP16’s narrower range is a much smaller practical risk, since a trained model’s activations tend to fall within more predictable, stable bounds than the wild swings possible during training. This is exactly why FP16 remains a genuinely common, practical choice for inference specifically, even though BF16, covered next, has become the more common default for training.

Common misconception

A named-model application

Hugging Face Diffusers documents loading Stable Diffusion pipelines with torch.float16 on supported GPUs. FP16 reduces model memory and can accelerate supported operations, making image generation practical on smaller GPUs than an FP32-only setup.

pipeline = DiffusionPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
)

The short example shows the developer-facing choice, but the library and GPU decide which individual operations can safely and efficiently use FP16.

For training, mixed precision and loss scaling are more important because tiny gradients can disappear or large values can overflow. Inference does not maintain training gradients, so its numerical risks and memory needs are different.

Read the 16-bit layout

FP16 = 1 sign bit + 5 exponent bits + 10 fraction bits

Each FP16 value occupies 2 bytes, half the storage of FP32. The smaller exponent gives FP16 a much narrower numerical range than BF16.

Why loss scaling may be needed

During training, some gradients are too tiny for FP16 and can round to zero. Loss scaling temporarily multiplies the loss and gradients by a scale factor before backpropagation, then removes that scale before the optimizer update.

tiny gradient -> multiply before FP16 work -> preserve useful value
              -> divide before weight update

Dynamic loss scaling adjusts the factor when overflow is detected. Frameworks such as PyTorch automate much of this mixed-precision process.

Real model usage

FP16 is common on NVIDIA GPUs for training and inference where hardware support is strong. A 7-billion-weight model needs approximately 14 GB for FP16 weights alone instead of 28 GB at FP32. The real footprint remains higher because inference also needs the KV cache and working memory.

Verified sources

A frequent beginner assumption: that FP16 and BF16, covered in the next article, are simply interchangeable “half-precision” formats, differing only in name. As this article’s exponent-versus-mantissa breakdown explained, they’re structurally quite different — FP16 keeps more mantissa precision but sacrifices range; BF16 makes the opposite trade, and that structural difference has real, measurable consequences for training stability specifically.

Where this fits in what comes next

You now understand FP16’s genuine memory savings and its real, documented range limitation. The next article, BF16, covers Google’s own direct, purpose-built solution to exactly this problem — a format designed specifically to keep FP32’s safe range while still cutting memory in half.

In one sentence

FP16 halves FP32’s memory footprint by shrinking both its exponent and mantissa, but its narrowed dynamic range creates a real, documented risk of overflow and underflow during training, a genuine limitation that directly motivated the alternative format covered in the next article.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026