INT4

The INT8 article covered the first integer-only format. This final article in the phase pushes precision reduction to its practical extreme, closing the loop this entire phase opened with: INT4.

The simple definition

INT4 represents a number using just 4 bits — 16 possible whole-number values total — an eighth the size of FP32’s original 32 bits, and the most aggressive, common precision reduction used in practice. Recall from the INT8 article’s scaling technique, mapping a layer’s real weight range onto the integers a format can represent. INT4 uses this exact same core approach, but with a dramatically coarser set of possible values — only 16 distinct integers available to represent an entire layer’s worth of weights, compressed from INT8’s 256.

Why this specific, extreme format connects directly back to where this phase began

Recall directly from the Quantization article’s real, documented story — Georgi Gerganov’s llama.cpp, released in March 2023 with the explicit, published goal of running a large language model using 4-bit integer quantization on an ordinary MacBook. INT4 is precisely that format. This isn’t a coincidence of naming — it’s the exact, specific precision level llama.cpp targeted because it represented the genuine sweet spot between memory savings and usable model quality for running large models on consumer hardware with no dedicated GPU at all.

flowchart LR
    A["INT4: only 16 possible values per number"] --> B[1/8th the memory footprint of FP32]
    B --> C["Real example: 7B model shrinks from ~28GB to ~4GB"]
    C --> D[Fits on an ordinary consumer laptop]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of describing every color in a photograph using only 16 total crayons from a box, rather than millions of possible paint mixtures — genuinely coarse, and some fine gradations and subtle shades will inevitably be lost, but a skilled, careful selection of which 16 colors to use can still produce a recognizable, often surprisingly effective rendition of the original image.

Where this breaks down: Choosing 16 crayons is a one-time, holistic artistic decision. Real INT4 quantization techniques, like the k-quant methods used in llama.cpp’s GGUF format, apply genuinely sophisticated, statistically-informed calibration — grouping weights into small blocks and calculating separate scale factors for each block, achieving an effective precision of somewhere between 2 and 6 bits per weight depending on the specific method, rather than one single, uniform 4-bit rule applied identically everywhere.

Why INT4 represents a genuine, real trade-off, not a free lunch

It’s worth being honest about this directly, closing out the trade-off theme this entire phase has traced. Recall from the Quantization article’s honest limitation — published benchmarks do show measurable accuracy degradation as bit-width drops this low, and INT4 sits close to the practical edge of what most models can tolerate before genuinely noticeable quality loss sets in.

Real, careful INT4 implementations, using the block-wise calibration described above, manage to keep this degradation surprisingly small for many practical purposes — but it’s a genuine, real cost, not an illusion, and some tasks requiring especially precise or nuanced output may notice the difference more than others.

A concrete example, layered

For a simple beginner example: a 7-billion-parameter model requiring roughly 28 gigabytes of VRAM at FP32 precision can shrink to around 4 gigabytes at INT4 — small enough to run comfortably on a consumer laptop’s GPU or even, with llama.cpp’s CPU-first design, without a dedicated GPU at all. For a production example: Ollama and LM Studio, real, widely used applications for running open-weight models locally, both build directly on llama.cpp and commonly default to or prominently offer INT4 quantized models specifically because this precision level represents the genuine, practical balance the entire open-source local-AI community converged on.

Common misconception

Why group size matters

Suppose 128 weights share one scale. That adds little scale metadata, but one unusual weight can stretch the range used by all 128 values.

If groups contain only 32 weights, each scale can fit its local values more closely:

group size 128 -> fewer scales -> smaller metadata -> rougher local fit
group size 32  -> more scales  -> larger metadata  -> closer local fit

Smaller groups often preserve quality better, but consume more metadata and may run differently depending on available kernels. Therefore, “INT4” alone is not enough information to reproduce a result. The report should also name the method, group size, compute format, hardware, and model.

A second real model connection

Hugging Face bitsandbytes documents four-bit NF4 loading for Transformer models and BF16 computation for supported hardware. QLoRA uses this kind of four-bit base-model storage together with small trainable LoRA adapters, connecting INT4-family techniques to practical fine-tuning as well as inference.

What four bits can store

Four bits provide only 16 possible bit patterns. A signed scheme might use values around -8 through 7, although exact representations vary by format.

FP16 weight:  0.13782
INT4-like code: one of only 16 available levels
scale metadata: helps turn that code back into an approximation

Because sixteen levels are coarse, practical methods often use separate scales for small groups of weights instead of forcing a whole model to share one scale.

INT4, FP4, and NF4 are not identical

INT4 uses four-bit integer-style levels.
FP4 uses a tiny floating-point representation.
NF4 is a four-bit format designed for normally distributed neural-network weights and is widely associated with QLoRA.

People sometimes say “4-bit model” without naming which method they mean. The format, grouping, calibration, and hardware kernel all affect quality and speed.

Real examples

Hugging Face Transformers and bitsandbytes can load supported models with FP4 or NF4 and can use BF16 as the computation type. NVIDIA reported that INT4 AWQ with TensorRT-LLM allowed the Falcon-180B model to run on one H200 GPU with 141 GB of memory, whereas the unoptimized setup previously required at least eight A100 GPUs in NVIDIA’s comparison.

This is a specific optimized configuration, not a promise that every 180-billion-parameter model fits or performs the same way.

Verified sources

A frequent beginner assumption: that going from INT8 down to INT4 simply doubles the precision loss, in some straightforward, linear way — twice as coarse, so roughly twice as much quality lost. As the block-wise calibration discussion above explained, real INT4 quantization’s actual impact depends heavily on the specific technique used, not a simple linear relationship with bit count — careful, statistically-informed calibration methods can keep INT4’s real-world quality surprisingly close to INT8’s, while a naive, uncalibrated approach at either precision level can perform far worse than the raw bit count alone would suggest.

Closing out this phase

This article completes the Infrastructure & Serving phase, and it’s worth tracing the full arc it built: GPU and TPU covered the hardware, and CUDA the software layer that makes GPUs usable, while VRAM introduced the hard, physical memory wall underlying everything that followed.

Inference Server and Model Serving covered running a model reliably at real scale, with Batch Inference and Streaming as the two opposing strategies for handling requests, measured by Latency, Throughput, and the single concrete number, Tokens Per Second (TPS), that ties them together.

Quantization named the technique connecting all of it back to VRAM’s original constraint, and FP32, FP16, BF16, INT8, and INT4 traced that technique through every real, specific number format — each one a deliberate, documented trade-off, from FP16’s genuine crash risk to BF16’s targeted fix to INT4’s extreme, practical compression, closing the exact gap the VRAM article opened between what a model needs and what ordinary hardware can actually provide.

In one sentence

INT4 compresses each weight down to just 16 possible values, an eighth of FP32’s original size, and it’s the exact, specific precision level llama.cpp’s real, published 2023 goal was built around — turning models that once required a data center into something that genuinely runs on an ordinary consumer laptop, completing this entire phase’s arc from raw hardware down to the smallest practical unit of precision.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026