Topic · Chaos

Denoising Diffusion Models

Static, walked backwards into a picture.

Take a photograph, sprinkle in a little Gaussian noise, repeat a thousand times — the photograph turns into pure static. A diffusion model is a neural network trained to walk that process in reverse: given static, predict the slightly less-static version, then again, then again, until a fresh image appears. Stable Diffusion, Midjourney, DALL·E and Sora are all this same idea, scaled up with text-conditioning and a learned latent space.

Discover the story → Open the Explorer

First encounter

Static in. Image out.

The big idea

Diffusion treats image generation as denoising. The forward process is a Markov chain that gradually corrupts data into Gaussian noise; the reverse process is a learned chain that takes pure noise back to data. Train the reverse on enough images and it generalises — sample fresh noise, walk it back, get a brand-new picture that has never existed.

A concrete example

Show a U-Net thousands of (image, noise-level, partially-noised-image) tuples and ask: which exact noise was added? Once it can guess the noise reliably at any level, you reverse the question — start from pure noise and subtract the prediction over a thousand small steps. What used to be static converges, frame by frame, into something coherent.

Why it matters

Diffusion replaced GANs in 2022 as the dominant generative-image method because it trains stably, scales cleanly and accepts arbitrary conditioning. The same math now drives drug-design backbones, audio synthesis, video models and even climate downscaling — anything where you can imagine a process slowly destroying structure, you can imagine learning to run it the other way.

Walk a tiny image into static and back. Train a toy denoiser on a 2D blob dataset and watch noise reorganise into structure.

Section 01 · The forward process

How a picture turns into static

Pick a noise schedule — a small sequence β₁ < β₂ < … < β_T of variances. At step t, replace each pixel by √(1−β_t)·x_{t−1} + √β_t·ε, where ε is fresh Gaussian noise. Iterated T = 1000 times, this drives every image to a sample of N(0, I): the original content is irrecoverably gone. A clean piece of algebra rolls the whole chain into a single closed form, q(x_t | x_0) = N(√ᾱ_t · x_0, (1−ᾱ_t) I), so you can jump straight to any noise level without simulating every step.

Forward process · q(x_t | x_0)

q(x_t | x_0) = N(√ᾱ_t · x_0, (1−ᾱ_t) I)

signal 100% · noise 0%

t = 0 / 8clean · x₀

Scrub through noise levelst = 0 / 8

At t=0 the image is the original. Each step multiplies signal by √(1−β) and folds in fresh Gaussian noise weighted by √β. By t=T the signal is gone and only noise remains. The reverse process is a neural network trained to walk this ladder upward from t=T back to t=0.

Section 02 · The reverse process

Learning to undo noise, one step at a time

The reverse chain p_θ(x_{t−1} | x_t) is what the network learns. Sohl-Dickstein (2015) showed that if the forward steps are small, the reverse steps are also Gaussian, so a neural net only has to learn their means and variances. Ho, Jain & Abbeel (2020) — the DDPM paper — simplified the loss to ‖ε − ε_θ(x_t, t)‖², i.e. predict the noise added to x_0. That reframing is what made diffusion practical: a tiny, clean regression target replaces a finicky variational bound.

Section 03 · Why a U-Net predicts noise

Same shape in, same shape out

The denoiser ε_θ has to read an image and output another image of the same dimensions, so the natural architecture is a U-Net: convolutional down-then-up, with skip connections that carry fine detail across the bottleneck. Time t and the text prompt enter as conditioning — sinusoidal time embeddings added to every block, cross-attention to a frozen text encoder for the prompt. The model never sees the clean image directly; it only learns to recognise and remove noise, and that turns out to be enough.

Section 04 · Sampling

Climbing back from pure noise

To generate, draw x_T ~ N(0, I) and iterate x_{t−1} = (1/√α_t) · (x_t − ((1−α_t)/√(1−ᾱ_t)) · ε_θ(x_t, t)) + σ_t · z. DDPM does this for all T steps; DDIM (Song 2020) collapses the chain into a deterministic ODE you can solve in 20–50 steps. Faster solvers — DPM-Solver, Heun, Euler — push generation under a second on a GPU. Each step is a small denoising correction; pile up enough of them and a coherent image emerges from the noise.

Section 05 · Classifier-free guidance & text-conditioning

Steering the walk with a prompt

Pure unconditional samples are blurry and uncontrolled. To follow a prompt, train the U-Net to take a text embedding c (from CLIP or T5) via cross-attention, randomly dropping the conditioning 10–20% of the time. At sample time use classifier-free guidance: ε̃ = ε_θ(x_t, ∅) + w · (ε_θ(x_t, c) − ε_θ(x_t, ∅)). The guidance scale w sharpens prompt alignment at the cost of diversity — w = 7.5 is the canonical Stable-Diffusion default. The same trick conditions on edges, depth maps, poses or other images for ControlNet.

Section 06 · From thermodynamics to Stable Diffusion

A short, fast lineage

Sohl-Dickstein and colleagues introduced the forward/reverse framework in 2015, framing it as non-equilibrium thermodynamics. Song & Ermon (2019) reached it from a different angle — score matching with Langevin dynamics — and proved the two views were equivalent up to a reparametrisation. Ho, Jain & Abbeel (2020) closed the loop with the noise-prediction loss and SOTA samples on CIFAR. Rombach et al. (2022) moved diffusion into the latent space of a VAE, slashing compute by 50× and producing Stable Diffusion. Seven years from thermodynamic curiosity to the image engine of the internet.

2015

Sohl-Dickstein

2019

Song & Ermon

2020

Ho · Jain · Abbeel

2022

Rombach et al.

See the math in motion

Train a tiny denoiser. Sample fresh data.

The Explorer trains a small score network on a 2D dataset, then samples by running the reverse chain from pure Gaussian noise. You can watch random points reorganise themselves onto the data manifold in real time — the same trick that turns static into images, on a scale you can read.

→ Open the Explorer

Where you meet it

How and where this technique lives in the world today.

Image generation

Stable Diffusion, Midjourney, DALL·E 3 and Imagen are all latent-diffusion models. Type a prompt, the model walks Gaussian noise back to an image consistent with the text.

Video generation

Sora, Veo and Runway extend the same diffusion math to three dimensions (height × width × time) so the denoiser learns space-time consistency along with appearance.

Drug & protein design

RFdiffusion (Baker lab) and Chroma generate novel protein backbones by denoising 3D coordinates instead of pixels — published candidates have been synthesised and shown to fold.

Audio & speech

AudioLDM, Riffusion and the speech-synthesis line ElevenLabs/Vall-E use 1D diffusion over waveforms or spectrograms to generate music and natural-sounding voices from text.

Physics & thermodynamics

The forward process is literally Langevin dynamics — the noise schedule mirrors a system relaxing toward thermal equilibrium. Sohl-Dickstein's original 2015 paper was framed as non-equilibrium thermodynamics.