Improved Denoising Diffusion Probabilistic Models (iDDPM)
this paper tries to patch things DDPM is not so hot at: the log-likelihood is bad, and sampling takes thousands of steps. the fixes are small and mostly orthogonal to the original framework. nothing here changes the core idea. it’s variance, schedule, objective, and sampling, and together they’re enough to make diffusion competitive with likelihood models and cut sampling by an order of magnitude (~100 steps instead of 4000).
if you haven’t read the DDPM note yet, start there, and the diffusion hub has the mental model. this note is just the iDDPM delta.
summary
- problem: DDPM samples look great but the log-likelihood is bad compared to autoregressive models and VAEs, which raises the obvious worry that the model isn’t actually covering the distribution, it’s just painting nice pictures.
- claim: with a few small changes, DDPM can be competitive on likelihood and keep its sample quality. learning the reverse variance also unlocks sampling with an order of magnitude fewer forward passes for basically free.
- key insight: increase $T$, swap the linear noise schedule for a cosine one, learn $\Sigma_\theta$, and train on a hybrid $L_{\text{simple}} + \lambda L_{\text{vlb}}$ objective.
- evidence: learning curves and gradient-noise scales (Figs. 6–7), schedule and objective ablations on ImageNet 64×64 and CIFAR-10 (Tables 1–2), NLL against prior likelihood models (Table 3), precision/recall against BigGAN-deep (Table 4).
- result: 3.53 bits/dim on ImageNet 64×64 and 2.94 on CIFAR-10, FID 2.92 on class-conditional ImageNet 64×64, and higher recall than a comparable GAN.
what’s actually new (versus Ho et al.)
- learned reverse variance via log-space interpolation between $\beta_t$ and $\tilde{\beta}_t$ (Eq. 15).
- hybrid objective $L_{\text{simple}} + \lambda L_{\text{vlb}}$, with a stop-gradient on $\mu_\theta$ (Eq. 16).
- cosine noise schedule (Eq. 17).
- importance sampling for $L_{\text{vlb}}$ to kill gradient noise (Eq. 18).
- fast sampling by striding through a subsequence of timesteps, with the learned variance rescaling itself (Eq. 19).
- precision/recall against GANs, plus FID/NLL scaling with model size and compute.
objective (Eqs. 4–7)
\[\begin{aligned} L_{\text{vlb}} &:= L_0 + L_1 + \cdots + L_{T-1} + L_T && \text{(4)} \\ L_0 &:= -\log p_\theta(x_0 \mid x_1) && \text{(5)} \\ L_{t-1} &:= D_{\text{KL}}\!\big(q(x_{t-1} \mid x_t, x_0) \,\|\, p_\theta(x_{t-1} \mid x_t)\big) && \text{(6)} \\ L_T &:= D_{\text{KL}}\!\big(q(x_T \mid x_0) \,\|\, p(x_T)\big) && \text{(7)} \end{aligned}\]term by term:
- $L_T$ (Eq. 7): how far the final noisy state is from the prior. it’s constant during training (the forward process has no parameters and $p(x_T) = \mathcal{N}(0, I)$), so it gets no gradient and is usually ignored.
- $L_{t-1}$ (Eq. 6): the workhorse. the KL between the true forward posterior (what the noising process actually did, given $x_0$) and the learned reverse step. this is the term that trains the denoiser. both sides are Gaussians, so it has a closed form. Hence, trainable.
- $L_0$ (Eq. 5): the final reconstruction: how much likelihood the model gives the clean image given $x_1$. I think of it as a distortion term.
- the sum (Eq. 4) is the bound on $-\log p_\theta(x_0)$, i.e. the actual likelihood objective. it’s the “correct” thing to optimize, but the per-step terms differ in magnitude by orders of magnitude, so its gradient is noisy and it trains poorly on its own. that tension is the rest of the paper: $L_{\text{simple}}$ gives good samples, $L_{\text{vlb}}$ gives likelihood, and iDDPM ends up training both.
learning the reverse variance
Ho et al. fixed $\Sigma_\theta(x_t, t) = \sigma_t^2 I$, either $\sigma_t^2 = \beta_t$ or $\sigma_t^2 = \tilde{\beta}_t$. the authors instead let the network predict a vector $v$ and interpolate in log space:
\[\Sigma_\theta(x_t, t) = \exp\big(v \log \beta_t + (1 - v)\log\tilde{\beta}_t\big)\]log-space keeps the variance between the two known bounds while still letting the model pick where it lands, per dimension.
But $L_{\text{simple}}$ doesn’t depend on $\Sigma_\theta$ at all, so it can’t train the variance. that’s what the hybrid loss is for:
\[L_{\text{hybrid}} = L_{\text{simple}} + \lambda L_{\text{vlb}}\]with $\lambda = 0.001$ and a stop-gradient on $\mu_\theta$ inside the $L_{\text{vlb}}$ term. the division of labor ends up clean: $L_{\text{vlb}}$ mostly shapes $\Sigma_\theta$, and $L_{\text{simple}}$ mostly shapes $\mu_\theta$.
the first few diffusion steps contribute almost all of the VLB (Fig. 2), so fixing the variance there is leaving likelihood on the table. learning it improves NLL without wrecking sample quality.
side note the $v$ here is iDDPM’s variance-interpolation knob. it is not the $v$-target from the noise schedules paper. same letter, unrelated job.
Fig. 4
Fig. 4 plots FID as a function of how much of the reverse process you skip:
- with the linear schedule, you can skip up to ~20% of the reverse process and FID barely moves. that means the tail of the forward process is already almost pure noise, so those reverse steps are wasted.
- with the cosine schedule, skipping hurts FID much sooner, which means the steps are actually doing something all the way through.
so the linear schedule is burning diffusion steps near $t = T$ for nothing. Such an awesome diagnoses by the authors.
the cosine schedule
they replace the linear schedule with:
\[\bar{\alpha}_t = \frac{f(t)}{f(0)}, \qquad f(t) = \cos\!\left(\frac{t/T + s}{1 + s}\cdot\frac{\pi}{2}\right)^2\]with $s = 0.008$, then $\beta_t = 1 - \bar{\alpha}_t / \bar{\alpha}_{t-1}$, clipped at 0.999.
importance sampling the VLB
$L_{\text{vlb}}$ is hard to optimize directly because its gradient is noisy (Fig. 7). the reason is visible in Fig. 2: the per-timestep terms $L_t$ differ in magnitude by orders of magnitude, so sampling $t$ uniformly spends most of its effort on terms that barely matter, and the variance is awful.
the fix is textbook importance sampling:
\[L_{\text{vlb}} = \mathbb{E}_{t \sim p_t}\!\left[\frac{L_t}{p_t}\right], \qquad p_t \propto \sqrt{\mathbb{E}[L_t^2]}, \qquad \sum_t p_t = 1\]we don’t know $\mathbb{E}[L_t^2]$, so they keep a running history of the last 10 values per timestep and update $p_t$ from that. early on, every $t$ is sampled uniformly until it has 10 samples in the history. I have a feeling that high variance data would bite in this scenario, but let’s just roll with it as this gave quite good results.
the effect shows up as the Lvlb (resampled) curve in Fig. 6: much less noisy, better NLL. it’s the usual variance-reduction move: Sample more where the loss is bigger.
fast sampling: just skip timesteps
this section of the paper is still unclear to me. Will update once I have some clarity, or maybe after I read DDIM
pseudocode
training:
for each batch x0:
t ~ Uniform(1..T) or t ~ pt (importance sampling)
eps ~ N(0, I)
xt = sqrt(abar_t) * x0 + sqrt(1 - abar_t) * eps
eps_theta = UNet_eps(xt, t)
v = UNet_v(xt, t)
mu_theta = (1/sqrt(alpha_t)) * (xt - beta_t/sqrt(1-abar_t) * eps_theta)
Sigma = exp(v * log beta_t + (1-v) * log beta_tilde_t)
Lsimple = ||eps - eps_theta||^2
Lvlb = VLB(mu_theta.detach(), Sigma, xt, x0, t)
Lhybrid = Lsimple + lambda * Lvlb
update theta
fast sampling with a subsequence $S$:
S = round(linspace(1, T, K))
compute abar^S, beta^S, beta_tilde^S
x = N(0, I)
for t in reverse(S):
eps_theta = UNet_eps(x, t)
v = UNet_v(x, t)
mu_theta = (1/sqrt(alpha^S_t)) * (x - beta^S_t/sqrt(1-abar^S_t) * eps_theta)
Sigma = exp(v * log beta^S_t + (1-v) * log beta_tilde^S_t)
x = mu_theta + sqrt(Sigma) * N(0, I)
return x
limitations
- $L_{\text{vlb}}$ wins NLL and loses FID; $L_{\text{hybrid}}$ is the compromise. sampling is faster but still not GAN-fast.
- the interpolation range is expressive enough, the 10-sample importance-sampling history is stable, and the cosine schedule transfers across datasets are all assumptions.
- cost: up to 1.5M iterations, 120M params, ~39 GFLOPs forward, $T=4000$; sampling still takes 50–250 steps.
breadcrumbs
- ddpm eq-by-equation (2006.11239) - this note assumes ho et al; that one carries the base objective
- noise schedules, cfg & samplers (2305.08891) - the cosine schedule and sampler corrections, continued
- diffusion-notes - where this fits in the family map
todos
- verify the cosine-schedule claim numerically once: $\bar\alpha_t = \cos^2(\frac{t/T + 0.008}{1.008}\frac{\pi}{2})$ re-derived from the paper’s Fig. 4 rather than trusted.
- how do modern trainers implement the 10-sample importance-sampling history for $L_{vlb}$ - check against a lucidrains-style repo and note the deltas.
- the hybrid loss weighting: pin down what weight actually gets used in practice (the FID-vs-NLL compromise) - currently only qualitative in this note.