The BF16 article covered the second major floating-point format in this phase. This article covers a genuinely different category entirely — abandoning floating-point decimals altogether in favor of simple, whole numbers: INT8.
The simple definition
INT8 represents a number using 8 bits — exactly 1 byte, a quarter of FP32’s 4 bytes and half of FP16/BF16’s 2 bytes — as a plain integer, with no decimal point or fractional component at all, rather than the floating-point formats covered throughout the rest of this phase. Recall from the FP32, FP16, and BF16 articles’ shared structure — sign bit, exponent, mantissa, all built to represent numbers with a decimal point. INT8 abandons that entire structure, representing only whole numbers, typically ranging from -128 to 127, using just 8 bits total.
Why moving to whole numbers requires a genuinely different technique
Recall from the Weights article that a model’s parameters are typically small, decimal values — numbers like 0.0347 or -0.128, not whole numbers at all. Simply rounding these values to the nearest integer would destroy nearly all of the meaningful information they carry. INT8 quantization instead uses a technique called scaling: mapping the actual, real range of a layer’s weights (say, -0.5 to 0.5) onto the full integer range INT8 can represent (-128 to 127), then storing a single “scale factor” alongside the integers so the original, approximate decimal values can be reconstructed when needed for computation.
flowchart LR
A[Original weights: small decimal values, e.g., -0.5 to 0.5] --> B[Scaling: map onto integer range -128 to 127]
B --> C[Store as INT8 integers + one scale factor]
C --> D[Reconstruct approximate decimal values during computation]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a tailor’s measuring tape marked only in whole centimeters, with a separate note specifying “multiply every reading by 0.1” to actually get the real measurement in meters — the tape itself only ever shows simple whole numbers, but the conversion note lets you recover the actual, more precise real-world value it represents.
Where this breaks down: A tailor’s conversion note is a single, fixed, simple multiplier. Real INT8 quantization often uses more sophisticated calibration — different scale factors for different layers or even different groups of weights within a layer, determined by actually analyzing a real, representative sample of data through the model beforehand, a genuine, careful engineering process rather than one simple, universal conversion rule.
Why this specific format requires real, careful calibration to work well
This is worth being direct about, since it’s a genuine, documented technical challenge distinct from the floating-point formats covered earlier in this phase. Recall from the Quantization article’s honest limitation — aggressive precision reduction can measurably degrade output quality. INT8’s integer-only range is considerably coarser than even FP16’s reduced precision, meaning naive, uncalibrated INT8 quantization can noticeably hurt a model’s accuracy. Real, production INT8 quantization typically uses a calibration process — running representative data through the model and observing the actual range of values each layer produces — to choose scale factors that minimize this real, measurable quality loss.
A concrete example, layered
For a simple beginner example: a small image-classification model quantized to INT8 might run twice as fast as its FP16 equivalent on hardware with dedicated INT8 support, since integer arithmetic is genuinely simpler and cheaper for a chip to compute than floating-point arithmetic, while losing only a small, often acceptable amount of classification accuracy.
For a production example: Nvidia’s TensorRT, a real, published inference optimization toolkit, and Google’s own TPUs, referenced throughout this phase, both offer dedicated, hardware-accelerated INT8 support specifically for production inference, where the calibration effort is a worthwhile, one-time cost in exchange for genuinely faster, cheaper serving at scale.
Why INT8 has become the standard choice specifically for inference, not training
It’s worth being direct about this real, practical distinction, tying together the pattern established throughout this phase. Recall from the FP16 article’s real training instability risk. INT8’s even coarser precision makes it considerably riskier for training, where a model’s weights are actively, delicately being adjusted through gradient descent — but for inference, running an already-trained, stable model, INT8’s coarser precision is a much more manageable, well-understood trade-off, which is exactly why INT8 is common for serving deployed models but rare for training them from scratch.
Common misconception
Symmetric and asymmetric INT8
Symmetric quantization places zero at the center and commonly uses one scale:
negative values <---- 0 ----> positive valuesAsymmetric quantization also uses a zero point, allowing the stored integer range to shift toward where the real values occur:
real value ≈ scale × (integer - zero point)Symmetric mapping is simple and often convenient for weights. Asymmetric mapping can represent one-sided activation ranges more efficiently. Hardware kernels and model behavior determine which choice is useful.
A named-model example
Hugging Face demonstrates bitsandbytes INT8 loading with real Transformer checkpoints such as BLOOM-1.7B and OPT models. The method keeps important outlier calculations in a wider format instead of forcing every value into INT8.
This is why production engineers name the exact method—such as LLM.int8()—rather than saying only “we converted the model to integers.”
Turn decimals into INT8 values
Signed INT8 stores whole numbers from -128 to 127. A quantizer uses a scale, and sometimes a zero point, to map floating-point values into that limited grid.
approximate real value = scale × (stored integer - zero point)If the scale is 0.1, stored integer 23 approximately represents 2.3. Many original decimal values must share one integer step, so calibration chooses ranges carefully.
Weights, activations, and outliers
Some LLM values are unusual outliers. Hugging Face’s bitsandbytes documentation explains that LLM.int8() handles many values with INT8 while processing outlier features with FP16. This is a mixed strategy, not a demand that every number fit INT8 equally well.
Loading supported Transformer models in 8-bit with bitsandbytes can roughly halve weight memory compared with 16-bit loading. NVIDIA TensorRT also provides hardware-accelerated INT8 paths for production inference.
When not to use it
Avoid assuming INT8 is an automatic upgrade. Measure task quality, latency, throughput, and actual hardware support. A model can become smaller without becoming faster if the chosen device or serving engine cannot execute the quantized operations efficiently.
Verified sources
A frequent beginner assumption: that INT8 is simply “FP8,” an even smaller floating-point format continuing the same pattern as FP32, FP16, and BF16. As this article has explained, INT8 is a genuinely different kind of number representation entirely — integers with no fractional component, requiring the scaling and calibration process described above — not simply a smaller version of the same floating-point structure covered throughout the rest of this phase.
Where this fits in what comes next
You now understand the first integer-based format in this phase, and the genuine calibration work it requires. The final article in this phase, INT4, pushes this same integer approach even further — the exact format llama.cpp’s real, published goal, covered in the Quantization article, was specifically built around.
In one sentence
INT8 represents weights as plain, whole-number integers rather than floating-point decimals, using a scaling technique to approximate the original values, and while it requires genuine, careful calibration to avoid meaningfully hurting accuracy, its dedicated hardware support across real, published tools like Nvidia’s TensorRT has made it a standard, practical choice specifically for fast, efficient production inference.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed