Diffusion Model

The Latent Space article closed with a promise: the currently dominant technique for generating genuinely new content. This article covers exactly that: the diffusion model.

The simple definition

A diffusion model generates new content by starting from pure random noise and gradually removing that noise, step by step, until a coherent image (or other content) emerges. Recall from the GAN article’s competition-based approach to generation. A diffusion model works completely differently — no competing networks, no adversarial training. Instead, it learns a single, repeatable skill: given a noisy image, predict what the noise looks like, and subtract it — repeated dozens of times, gradually sharpening static into a genuine, coherent picture.

Why “learning to remove noise” turns out to be a genuinely powerful training approach

This is worth explaining precisely, since it directly solves the GAN article’s honest limitation. Recall that GAN training can be unstable — mode collapse, an imbalanced generator-discriminator race. A diffusion model sidesteps this by training on a much simpler, more stable task: take a real image, deliberately add a specific, known amount of random noise to it, and train the model to predict and remove exactly that noise.

+This is a genuine supervised learning problem, as covered throughout the Data Handling phase — there’s a precise, known correct answer (the exact noise that was added) — none of the adversarial instability a GAN’s competing-networks setup introduces.

flowchart LR
    A[Pure random noise] --> B[Model predicts and removes a bit of noise]
    B --> C[Slightly less noisy image]
    C --> D[Repeat 50-100 times]
    D --> E[Coherent, final image]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a restorer gradually cleaning a badly damaged, static-covered old photograph — not all at once, but through many careful, incremental passes, each one removing a bit more grain and revealing a bit more of the underlying picture, until a clear, recognizable image emerges from what looked like pure noise at the start.

Where this breaks down: A photo restorer works from a genuine, specific damaged original they’re trying to recover. A diffusion model, when generating something entirely new (rather than restoring a real photo), starts from pure, random noise with no real underlying image hidden inside it at all — the “recovery” is really construction, guided by everything the model learned during training about what real images tend to look like, not literal restoration of a specific hidden picture.

Why this process happens in latent space, not raw pixels

Recall directly from the Latent Space article’s mention of Stable Diffusion. Running the noise-removal process on full-resolution pixel data, for every one of dozens of steps, would be enormously expensive computationally. Real production diffusion models instead perform most of this denoising process inside a compressed latent space, as covered in the previous article, and only decode the final result back into a full-resolution image at the very end — a deliberate, published architectural choice that makes the whole process dramatically cheaper to run.

The real story: an industry given away for free

This is worth telling in full, because it’s a genuinely consequential moment in the field’s history. In August 2022, Stability AI, working with researchers from CompVis at Ludwig Maximilian University of Munich and the company Runway, released Stable Diffusion publicly — and, critically, released its actual code and model weights openly, under a permissive license, for anyone to download and run.

+This was a genuine departure from how OpenAI’s DALL-E 2 (released just one month earlier, in July 2022) and Midjourney worked — both accessible only through paid, closed, cloud-hosted services. Stable Diffusion could run on a consumer laptop’s own GPU, with some optimized versions needing as little as roughly 2.4 to 8 gigabytes of video memory — hardware plenty of enthusiasts already owned.

+This single decision — open-sourcing a genuinely capable diffusion model rather than keeping it locked behind a paid API — triggered an explosion of community experimentation, fine-tuned variants, and derivative tools that a closed model could never have produced at the same pace.

A concrete example, layered

For a simple beginner example: generating “a cat riding a skateboard on Mars” starts with a canvas of pure random static, and across roughly 50 to 100 refinement steps, the model gradually shapes that noise into an increasingly clear, coherent image matching the text description, guided at every step by the prompt.

+For a production example: Stable Diffusion’s own real, documented technical process, as described in its published research, works by translating a text prompt into a numeric representation, then performing the noise-removal process within a compressed latent image space across those 50-100 steps, before a final decoder step converts the result into a full, high-resolution pixel image — precisely the pipeline this article and the Latent Space article together have described.

Why diffusion models mostly displaced GANs for image generation

It’s worth stating this directly, connecting to the training-stability limitation raised in the GAN article. Diffusion models’ more stable, straightforward supervised training process, combined with genuinely higher output quality and diversity in practice, is a major reason most leading text-to-image and text-to-video systems covered later in this phase — DALL-E 2 and 3, Stable Diffusion, and the video models covered in the Text-to-Video article — are built on diffusion rather than the GAN architecture that opened this phase.

Why diffusion models have a real, honest weakness of their own

It’s worth being fair to the GAN article’s approach here, not presenting diffusion as a strictly superior replacement. Recall from the GAN article that a trained generator produces a result in a single, fast forward pass.

+A diffusion model, by contrast, needs those same 50 to 100 sequential denoising steps at generation time, every single time, not just during training — meaning diffusion models are typically much slower to actually generate a single output than a comparably-sized GAN.

+This is a genuine, well-documented trade-off, and it’s exactly why a substantial area of diffusion research since Stable Diffusion’s release has focused specifically on reducing the number of steps needed, through techniques that distill a slow, many-step diffusion process into a faster one, echoing the distillation technique covered earlier in the Advanced Architectures phase.

Watch denoising happen in five teaching steps

The following is simplified to make the direction visible:

Step 5: random noise
Step 4: a rough bright shape appears
Step 3: the shape begins to resemble a cat
Step 2: eyes, ears, and skateboard become clearer
Step 1: textures and edges become detailed
Step 0: final image

At each step, the model predicts which part of the current representation behaves like noise and the scheduler decides how to move toward a cleaner result. Real pipelines may use a different number of steps and different scheduling mathematics.

What changes the result?

• The prompt describes the requested content.
• The seed helps create the starting noise, so reusing a seed can help reproduce or compare a result.
• The guidance setting controls how strongly generation follows the text condition.
• The number of inference steps affects the denoising journey, latency, and sometimes quality.

These controls influence generation together; a seed does not store an already-finished picture inside it.

A real model you can run: Stable Diffusion XL

SDXL is a diffusion-based text-to-image model available through Hugging Face. Its documented pipeline can use a base model to create noisy latents and an optional refiner specialized for the final denoising steps.

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0"
)
image = pipe("An astronaut in a jungle, muted colors").images[0]

Internally, this one call hides repeated denoising steps. The prompt is encoded into a numerical representation, the model gradually predicts and removes noise from a latent, and the VAE decodes the result into an image. The SDXL page identifies it as a roughly 3-billion-parameter model and documents both standalone-base and base-plus-refiner usage. Source: SDXL on Hugging Face.

Common misconception

A frequent beginner assumption: that a diffusion model works by literally searching through a huge database of existing images and blending or retrieving the closest matches to a prompt. As this article has explained, this isn’t accurate — a diffusion model never stores or looks up specific training images at generation time; it applies a learned, general noise-removal skill, guided by the prompt, to random noise, genuinely constructing a new image rather than recombining stored ones.

Where this fits in what comes next

You now understand the noise-removal, step-by-step generation process behind most of today’s leading image and video tools. The next article, VAE, covers a related, earlier technique for organizing latent space specifically to make it well-suited for generation — a genuine piece of the puzzle diffusion models themselves often still rely on for their final decoding step.

In one sentence

A diffusion model generates new content by learning to gradually remove noise from a canvas of pure static, guided by a prompt, across dozens of stable, well-understood training steps — a technique that, combined with Stability AI’s decision to release Stable Diffusion openly in August 2022, reshaped the entire generative AI industry within months.

Related Terms

• Latent Space
• GAN (Generative Adversarial Network)
• VAE
• Text-to-Image
• Generative Model