Image-to-Image

The Text-to-Video article covered generating video purely from text. This article covers a related but genuinely different generative task, starting from an existing image rather than starting from nothing: image-to-image.

The simple definition

Image-to-image generation takes an existing image as a starting point, along with some guidance (often a text prompt), and produces a new, transformed image — rather than generating an image entirely from scratch. Recall from the Diffusion Model article’s core process: starting from pure random noise and gradually removing it. Image-to-image works with almost the same underlying machinery, but starts from a real, existing image with only a controlled amount of noise added to it, rather than starting from pure static.

Why starting from a real image instead of pure noise changes everything

This is worth being precise about, since it’s a small technical difference with a large practical effect. Recall from the Diffusion Model article’s step-by-step denoising process.

+If you add only a little noise to a real photo before running the reverse, denoising process, the model has much less freedom to change things — the output stays close to the original image’s structure and composition, but gets reshaped according to whatever new guidance (a text prompt, a different artistic style) is provided.

+Add more noise before denoising, and the model has more freedom to deviate, producing a result more loosely inspired by the original rather than tightly constrained by it.

flowchart LR
    A[Starting image: a real photo] --> B[Add a controlled amount of noise]
    B --> C[Run the reverse diffusion process, guided by new instructions]
    C --> D[Transformed output: same underlying structure, new style or content]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of tracing over an existing photograph with translucent paper, using the original’s outlines and composition as a loose guide while painting in an entirely different style on top — a watercolor version of the same scene, or the same composition with the season changed from summer to winter. The underlying structure stays recognizable; the surface treatment changes dramatically.

Where this breaks down: A human artist tracing a photo applies genuine, deliberate artistic choices about what to keep and change. Image-to-image’s “tracing” is the precise, controllable noise-level parameter described above — a single, tunable number directly controlling how much of the original structure survives the transformation, not an artist’s evolving, in-the-moment creative judgment.

What this actually gets used for, concretely

This is worth being specific about, since image-to-image covers several genuinely distinct, real applications.

Style transfer takes a real photo and re-renders it in a different artistic style — turning a photograph into something resembling a Van Gogh painting, while keeping the original composition recognizable.

Sketch-to-image takes a rough, hand-drawn sketch and refines it into a polished, detailed final image, using the sketch’s basic shapes as loose structural guidance.

Inpainting, a related, widely used technique, lets a user select a specific region of an image and have the model regenerate just that region — removing an unwanted object from a photo, or adding something new into a specific spot, while leaving the rest of the image untouched.

Outpainting works in the opposite direction, extending an image beyond its original borders, generating new, consistent content in previously empty space — useful for changing an image’s aspect ratio or revealing “more” of a scene that was never actually photographed.

The real, precise fix for a genuine control problem: ControlNet

It’s worth naming a specific, widely used, published technique here, since it solves a real limitation the noise-level parameter alone can’t fully address.

+Simply adding noise to an image and letting a diffusion model regenerate it gives only loose, approximate control — the exact pose of a person, or the precise layout of a room, can still drift during generation. ControlNet, introduced in a 2023 paper by Lvmin Zhang and Maneesh Agrawala, solves this directly by adding a second, small neural network alongside a frozen, pretrained diffusion model, conditioned on a specific structural input — a Canny edge map, a depth map, or a detected human pose skeleton.

+This lets a user specify, with real precision, “generate a completely new image, but force it to match this exact pose” or “follow these exact edges,” rather than only hoping the noise level happens to preserve what they wanted. Critically, the paper’s own published results demonstrated the technique training reliably even on small datasets of under 50,000 images, and it’s since become a standard, widely available extension across major Stable Diffusion tools.

flowchart LR
    A[Reference input: e.g., a pose skeleton or edge map] --> B[ControlNet: small, trained conditioning network]
    C[Frozen, pretrained Diffusion Model] --> D[Combined generation]
    B --> D
    D --> E[New image, precisely matching the reference structure]

A concrete example, layered

For a simple beginner example: a user might upload a photo of their living room and prompt “make this look like a cozy cabin interior,” and image-to-image generation preserves the room’s actual layout — where the windows and furniture are — while transforming the materials, colors, and decor to match the new description.

+For a production example: Stable Diffusion, referenced throughout this phase, explicitly supports image-to-image generation as a core, documented capability alongside its text-to-image function, and ControlNet extensions are widely used in real production pipelines for tasks like virtual product photography (generating consistent product shots in many settings from one reference pose) and pre-visualization in game and film concept art, where an artist’s rough pose sketch gets turned into a fully rendered character illustration while keeping the exact pose intact.

Why this technique connects so directly to real creative workflows

It’s worth being direct about why this specific capability matters so much in practice, beyond the headline-grabbing pure text-to-image case covered in the previous articles. Most real creative work isn’t starting from nothing — a designer usually has some existing material (a rough sketch, a reference photo, an earlier draft) they want to refine or transform, not generate entirely from scratch. Image-to-image directly supports this far more common, iterative creative process, which is part of why it’s become such a standard, heavily used feature in real, professional creative tools rather than a niche capability.

Why getting the noise level right is a genuine, honest struggle

It’s worth being direct about a real, practical limitation, not presenting this technique as a simple, reliable dial. The noise-level parameter described earlier in this article is notoriously finicky in practice — set it too low, and the transformation barely changes anything meaningful; set it too high, and the model drifts so far from the original that it stops preserving the exact structure, object identity, or fine detail a user actually wanted kept.

+There’s no single, universally correct setting — finding the right balance for a specific image and a specific desired change is a real, trial-and-error process, and preserving a very specific detail (an exact face, a specific product’s logo) across a transformation remains a genuinely imperfect, actively researched problem.

Do not mix up four related applications

Application	What the user supplies	What changes?
Image-to-image	Starting image and usually a prompt	The full image can be reinterpreted
Inpainting	Image, mask, and instruction	Only the masked region should be replaced
Upscaling	Low-resolution image	Resolution and fine detail are increased
ControlNet-style guidance	Image-derived structure and prompt	Pose, edges, depth, or layout guide generation

Visualize the strength setting

strength 0.2 → preserve most of the original; make a gentle change
strength 0.5 → balance original structure with the new instruction
strength 0.9 → allow a major redesign; original details may disappear

These values are teaching examples, not universal boundaries. The exact meaning and valid range depend on the model, scheduler, and application.

A real image-to-image control: strength

Hugging Face’s Stable Diffusion image-to-image pipeline accepts both an initial image and a prompt. It encodes the image into latent space, adds noise, denoises under prompt guidance, and decodes the changed latent.

image = pipe(
    prompt="Turn this room into a cozy reading room",
    image=original_room,
    strength=0.45
).images[0]

The strength value controls how much of the starting image may be replaced. A lower setting tends to preserve more composition; a higher setting permits a larger transformation. Real applications use this for style transfer, color restoration, missing-area repair, product mockups, and iterative design.

OpenAI’s GPT Image 2 also supports image input and image output through its image-edit endpoint. That is the same application category even though its internal architecture and controls need not match Stable Diffusion’s. Sources: Hugging Face image-to-image documentation and official OpenAI documentation for GPT Image 2.

Common misconception

A frequent beginner assumption: that image-to-image generation is simply an advanced filter, similar to the Instagram-style filters that apply a fixed, predetermined visual effect to a photo.

+As this article has explained, this understates what’s actually happening — the underlying diffusion process genuinely regenerates much of the image’s content, guided by both the original structure and new instructions, producing results a fixed filter’s simple color and contrast adjustments could never achieve, including entirely new objects, textures, and structural details that weren’t present in the original at all.

+It’s also worth distinguishing plain image-to-image (loose, noise-level-based guidance) from ControlNet-style precise control (exact structural conditioning) — the first offers rough inspiration from a starting image, the second offers genuine, reliable precision.

Where this fits in what comes next

You now understand transforming existing images rather than generating from scratch. The next article, Multimodal Generation, covers the broader capability this phase has been building toward — a single model handling multiple different types of content (text, image, audio) together, rather than each generative task requiring its own separate, specialized system.

In one sentence

Image-to-image generation starts the diffusion process from a real image with a controlled amount of noise added, rather than pure random static, letting a model transform an existing photo or sketch while preserving its underlying structure — a genuinely practical, widely used capability that supports the iterative, refine-rather-than-start-from-scratch nature of most real creative work.

Related Terms

• Diffusion Model
• Text-to-Image
• Latent Space
• Text-to-Video
• Multimodal Generation