Inference Optimization | TechByteByByte

Before you continue: three tools for this module

Inference: using fixed learned parameters to produce output.
Logit: a raw score for a possible next token.
Decoding: the rule that chooses tokens from model scores.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Inference Optimization solve inside a real language-model system?

Keep that central question about Inference Optimization in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

request → efficient prefill/decode/cache/batching → faster cheaper tokens

1. What You Will Learn

Learning outcomes

Locate the main compute and memory costs during prefill and decoding.
Explain batching, KV caching, quantization, and efficient attention.
Distinguish latency, throughput, time to first token, and tokens per second.
Choose optimizations while checking quality and hardware tradeoffs.

In one sentence

💡 Big picture

Inference optimization makes model responses faster or cheaper by reducing repeated work, memory use, or numerical precision carefully.

2. Why This Module Exists

The problem this module solves

Large models can be slow and expensive to serve.
Caching, batching, quantization, and efficient kernels help, but every optimization must be checked for quality and hardware tradeoffs.

3. Intuition

every optimization in this module targets a specific, identifiable cost from Module 14’s analysis — quantization shrinks the memory footprint of the weights and KV cache; batching better utilizes GPU compute across concurrent requests; speculative decoding and distillation attack generation speed and model size directly.

4. Core Concept

Quantization:      reducing the numerical PRECISION used to store
                   model weights (and sometimes activations) --
                   e.g., 32-bit floats down to 8-bit or 4-bit
                   integers -- trading a small amount of accuracy
                   for substantial memory/speed gains

KV cache               (Module 14) -- storing computed Key/Value
(recap):            vectors to avoid redundant recomputation

Batching:                processing MULTIPLE requests together to
                       better utilize GPU compute

Continuous               a more efficient batching strategy:
batching:              add/remove individual requests from an
                       active batch as they finish, rather than
                       waiting for an entire fixed batch to
                       complete together

Speculative                use a SMALLER, faster model to
decoding:                propose multiple candidate tokens, then
                       verify them with the FULL model in one
                       pass -- can produce multiple tokens per
                       full-model forward pass

Model distillation:        training a SMALLER model to mimic a
                          LARGER model's behavior -- trading some
                          capability for substantially reduced
                          size/cost

5. How It Works — Step by Step (Quantization)

1. Model weights are normally stored as 32-bit or 16-bit floating
   point numbers
2. QUANTIZATION converts these to lower-precision representations
   (8-bit or 4-bit integers), using a scaling scheme to preserve
   as much of the original value's meaning as possible
3. This directly reduces MEMORY footprint (fewer bytes per
   parameter) and can improve inference SPEED (less data to move,
   sometimes faster low-precision compute)
4. Some ACCURACY is typically lost -- the trade-off's severity
   depends on the quantization scheme and how aggressively
   precision is reduced

6. Mathematical Intuition

Read the mathematics as a story

request → efficient prefill/decode/cache/batching → faster cheaper tokens

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

Memory footprint is directly proportional to num_parameters × bytes_per_parameter. Reducing bytes-per-parameter (via quantization) produces a direct, linear reduction in total memory required — verified precisely below.

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

A 7-billion-parameter model stored in standard 32-bit floating point requires roughly 4 bytes per parameter; quantizing to 4-bit integers requires only 0.5 bytes per parameter — an 8x reduction in raw memory footprint for storing the weights.

Analogy: Vacuum Seal Packing & The Busy Lawyer’s Speculative Intern Think of inference optimization in terms of packaging and workflow delegation:

Quantization (Vacuum Sealing): You are moving houses. Instead of packing thick, heavy winter jackets in giant boxes (32-bit floating point precision), you put them in plastic bags and suck the air out (quantizing to 4-bit integers).

The jackets take up 8x less volume in the moving truck (memory footprint). They might get slightly wrinkled (minor accuracy drop), but they still keep you warm.

You can fit the entire move on a single small van instead of hiring a heavy cargo truck.

Continuous Batching (The Adaptive Bus Driver): A public bus driver who doesn’t wait at the terminal for the bus to fill up before driving. Instead, they pick up passengers at street corners and let others off individually as soon as they reach their stop, maximizing seat utilization.

Speculative Decoding (The Lawyer & The Intern): A busy lawyer (the large, expensive model) hires a cheap intern (the small draft model).

The intern writes draft sentences for a contract (proposes candidate tokens).

The lawyer reads the draft page and approves 5 sentences at a time in a single glance (verifies candidates in one parallel pass).

If the intern makes a typo, the lawyer crosses it out and writes the correction. You save hours of typing.

📊 Visual Chart: Static vs. Continuous Batching GPU Efficiency

Here is how continuous batching fills idle calculation slots compared to static batching:

graph TD
    subgraph StaticBatching ["1. Naive Static Batching (Fixed synchronization)"]
        direction TB
        B1["Request 1 (10 tokens) -> finished"] --> Wait1["IDLE PADDING (Wait 90 cycles)"]
        B2["Request 2 (50 tokens) -> finished"] --> Wait2["IDLE PADDING (Wait 50 cycles)"]
        B3["Request 3 (100 tokens) -> finished"] --> Done["Batch completes together after 100 cycles"]
    end

    subgraph ContinuousBatching ["2. Continuous Batching (Dynamic sliding window)"]
        direction TB
        C1["Request 1 finishes at cycle 10"] --> Slot1["Slot instantly refilled with Request 4"]
        C2["Request 2 finishes at cycle 50"] --> Slot2["Slot instantly refilled with Request 5"]
        C3["Request 3 finishes at cycle 100"] --> Slot3["Slot instantly refilled with Request 6"]
    end

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Inference Optimization.
# Follow the inputs, transformations, and output in order.
num_params = 7_000_000_000  # 7B model

precisions = {
    "FP32 (32-bit)": 4,
    "FP16/BF16 (16-bit)": 2,
    "INT8 (8-bit)": 1,
    "INT4 (4-bit)": 0.5,
}

print(f"Memory footprint for a {num_params/1e9:.0f}B parameter model:\n")
for name, bytes_per_param in precisions.items():
    total_gb = (num_params * bytes_per_param) / (1024**3)
    print(f"  {name:20s}: {total_gb:.2f} GB")

fp32_gb = num_params * 4 / (1024**3)
int4_gb = num_params * 0.5 / (1024**3)
print(f"\nMemory reduction FP32 -> INT4: {fp32_gb/int4_gb:.1f}x smaller")

# --- Static batching's GPU utilization waste ---
def static_batch_gpu_utilization(request_lengths, batch_size):
    total_gpu_time = 0
    total_useful_time = 0
    for i in range(0, len(request_lengths), batch_size):
        batch = request_lengths[i:i+batch_size]
        max_len = max(batch)
        total_gpu_time += max_len * len(batch)
        total_useful_time += sum(batch)
    return total_useful_time / total_gpu_time

request_lengths = [10, 50, 15, 100, 20, 12, 80, 18]
static_util = static_batch_gpu_utilization(request_lengths, batch_size=4)
print(f"\nRequest lengths: {request_lengths}")
print(f"STATIC batching GPU utilization: {static_util*100:.1f}%")

Expected Output:

Memory footprint for a 7B parameter model:

  FP32 (32-bit)       : 26.08 GB
  FP16/BF16 (16-bit)  : 13.04 GB
  INT8 (8-bit)        : 6.52 GB
  INT4 (4-bit)        : 3.26 GB

Memory reduction FP32 -> INT4: 8.0x smaller

Request lengths: [10, 50, 15, 100, 20, 12, 80, 18]
STATIC batching GPU utilization: 42.4%

9. How It Works

Quantizing from FP32 to INT4 reduces a 7B model’s memory footprint from 26.08 GB to 3.26 GB — a real, exactly 8.0x reduction (matching the precision’s byte-size ratio precisely) — the direct difference between needing multiple high-end GPUs and fitting comfortably on a single one.
Static batching, grouping requests into fixed batches and waiting for every request in a batch to finish before starting the next batch, achieved only 42.4% GPU utilization on this realistic mix of request lengths — because shorter requests in a batch sit idle, effectively “wasted,” waiting for the longest request in that same batch to complete. Continuous batching avoids this specific waste by managing requests individually rather than as fixed groups.

10. Speculative Decoding — Conceptually

1. A SMALL, fast "draft" model proposes several candidate NEXT
   tokens in a row (cheap, since it's a small model)
2. The FULL, large model verifies these candidates in ONE forward
   pass (checking multiple positions at once, rather than one at
   a time)
3. If the draft model's guesses were correct, MULTIPLE tokens are
   accepted per full-model forward pass -- effectively amortizing
   the large model's expensive computation across more than one
   generated token
4. If a guess was wrong, generation falls back to the large
   model's own prediction at that point

This directly reduces the number of full, expensive forward passes (Module 14) needed per generated token, when the draft model’s guesses are frequently correct.

11. Model Distillation — Conceptually

A smaller “student” model is trained to mimic a larger “teacher” model’s outputs/behavior — trading some capability for substantially reduced size, memory footprint, and inference cost, useful when the smaller model’s reduced capability is still sufficient for the target task.

12. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Every production LLM serving system uses some combination of these techniques — quantization is nearly universal for cost-effective serving, continuous batching is standard in modern serving frameworks, and speculative decoding/distillation are increasingly common for latency-sensitive or cost-sensitive deployments.

13. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: High, practically. Agent systems with high request volume or latency-sensitive interactive use cases benefit directly from these optimizations — quantized or distilled smaller models for fast, cheap sub-tasks (like routing or classification), continuous batching for efficiently serving many concurrent agent sessions.

14. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming quantization always significantly hurts accuracy.

Why it is incorrect: Modern quantization techniques (especially at 8-bit) often preserve accuracy remarkably well; the trade-off’s severity depends heavily on the specific technique and how aggressive the precision reduction is.

⚠️ Mistake

Incorrect idea: assuming static batching is “good enough.”

Why it is incorrect: As verified directly, it can waste over half of available GPU compute in realistic scenarios — a genuine, significant inefficiency continuous batching directly addresses.

⚠️ Mistake

Incorrect idea: confusing speculative decoding with simply using a smaller model.

Why it is incorrect: Speculative decoding still uses the FULL model for final verification — it’s about generating more tokens per expensive forward pass, not replacing the large model entirely.

15. Important Distinctions

Static Batching	Continuous Batching
Fixed groups, wait for the LONGEST request	Individual requests added/removed as they finish
Verified directly: 42.4% GPU utilization in a realistic example	Substantially higher utilization by design

Quantization	Distillation
Reduces PRECISION of existing weights	Trains a genuinely SMALLER model from scratch (or fine-tuned) to mimic a larger one

16. When to Use

Use quantization for nearly any production deployment where memory/cost matters — the trade-off is often favorable. Use continuous batching for any multi-request serving system. Use speculative decoding and distillation when latency or cost constraints are especially tight and the trade-offs are acceptable for the specific application.

17. When Not to Use

Extremely aggressive quantization (very low bit-widths) may not be appropriate for tasks requiring maximum precision/capability — a real, evaluatable trade-off rather than a universal default.

18. Production Considerations

Quantization’s accuracy impact should be evaluated per application — not assumed negligible or assumed severe without actual testing.
Continuous batching is standard in modern serving frameworks — a foundational infrastructure choice, not an advanced/optional technique.
GPU memory constraints directly interact with quantization, batching, and KV cache size (Module 14) — these optimizations are genuinely interconnected, not independent levers.

19. What You Should Remember

Quantization directly, linearly reduces memory footprint — verified directly: 8.0x smaller going from FP32 to INT4.
Static batching wastes real GPU compute waiting for the slowest request in each batch — verified directly (42.4% utilization); continuous batching addresses this directly.
Speculative decoding and distillation attack generation speed and model size respectively, complementing quantization and batching.

20. Interview Questions

Beginner

Q: What is quantization, and why is it used for LLM serving?

Ans: Quantization reduces the numerical precision used to store a model’s weights — for example, from 32-bit floating point down to 8-bit or 4-bit integers — directly reducing the model’s memory footprint and often improving inference speed, at the cost of some accuracy loss.

Verified directly: quantizing a 7B parameter model from FP32 to INT4 reduced its memory footprint by exactly 8x.

Intermediate

Q: Why does static batching waste GPU compute, and how does continuous batching address this?

Ans: Static batching groups a fixed set of requests together and waits for every request in that batch to finish before processing the next batch — meaning shorter requests sit idle, effectively wasting compute, while waiting for the longest request in their batch to complete.

Verified directly: this produced only 42.4% GPU utilization on a realistic mix of request lengths. Continuous batching instead manages requests individually, adding new requests and removing completed ones from the active batch dynamically, avoiding this “wait for the slowest” inefficiency.

Advanced

Q: Explain how speculative decoding reduces the number of expensive, full-model forward passes needed to generate a sequence.

Ans: A smaller, faster “draft” model proposes several candidate next tokens in sequence — a cheap operation given the draft model’s small size.

The full, large model then verifies all these candidates in a SINGLE forward pass (checking multiple positions simultaneously, rather than generating them one at a time as in standard decoding, Module 7).

When the draft model’s guesses are correct (which happens frequently for predictable continuations), multiple tokens get accepted per full-model forward pass — directly reducing the total number of expensive full-model computations needed relative to standard, one-token-per-pass decoding.

Scenario

**Q: A team serving an LLM notices their GPU utilization metrics are surprisingly low despite having many concurrent user requests.

What would you investigate, based on this module?** A: I’d investigate their batching strategy first — as demonstrated directly, static batching (waiting for an entire fixed-size batch to finish, including its longest request, before processing the next batch) can produce significantly underutilized GPU compute when request lengths vary substantially, exactly matching the described symptom.

Switching to continuous batching, which manages requests individually rather than in fixed groups, would likely substantially improve utilization by eliminating the “wait for the slowest request in this batch” waste inherent to the static approach.

AI Engineering

Q: Why might a team choose to combine quantization, continuous batching, AND a smaller/distilled model for a cost-sensitive production deployment, rather than relying on just one optimization?

Ans: These techniques target genuinely different, complementary cost dimensions: quantization reduces per-parameter memory footprint (verified directly, an 8x reduction going to INT4), continuous batching improves GPU utilization across concurrent requests (verified directly, avoiding significant waste from static batching), and using a smaller/ distilled model directly reduces the underlying compute required per request.

Combining all three compounds their individual benefits — each addresses a distinct inefficiency, and none of them substitutes for what the others provide, making a combined approach substantially more cost-effective than relying on any single optimization alone.

21. Next Step

Next: Module 25 — Open-Source vs Closed-Source LLMs — self-hosting vs. API-based inference, and the genuine trade-offs across privacy, cost, and customization.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026