Common Diffusion Noise Schedules and Sample Steps are Flawed
stable diffusion kept giving me dead mids and killing off dynamic range, and i wanted to know why. turns out the noise schedule and the sample steps themselves are the problem. these are my notes on Common Diffusion Noise Schedules and Sample Steps are Flawed (arXiv:2305.08891).
why am i reading this?
note to self: still need to compare this scheduler against SNR and the pre-$t$ weighting.
Problem: train-inference mismatch due to inconsistent states and non-zero SNR. Also $\epsilon$ training makes worse training primitive. When considering $t=T$, better to work with $v$ training.
Claim: rescale noise schedule to zero terminal SNR, train with $v$ prediction, start sampling from the last timestep, rescale CFG.
Key insight: simple changes above give better diffusion training and allow full brightness range generation.
Evidence: from the paper: Table 3: lower FID, higher IS; Figure 3: qualitative brightness range; Figure 6: CFG rescale ablation.
Main limitation: requires finetuning the model; not tested on other modalities; assumes variance-preserving formulation.
They solve training-inference mismatch by rescaling existing noise schedules under the variance-preserving formulation to enforce zero terminal SNR, which works because inference run starts at pure Gaussian noise at its last timestep, of which the mean is always centered around zero. So forcing the forward pass end state to be equal to backward pass initial state gives better stability.
Equation-by-equation breakdown
Equations 1–4: Forward diffusion
What does it compute? Standard diffusion forward pass: image to noise with Gaussian noise added over the image, step by step.
- Eq. (1): $q(x_{1:T}\mid x_0) := \prod_{t=1}^T q(x_t \mid x_{t-1})$ - the full Markov chain factorization.
- Eq. (2): $q(x_t \mid x_{t-1}) := \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I)$ - one step of noise addition.
- Eq. (3): $q(x_t \mid x_0) := \mathcal{N}(x_t; \sqrt{\bar\alpha_t}x_0, (1-\bar\alpha_t)I)$ - closed-form jump to any $t$.
- Eq. (4): $x_t := \sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\epsilon$ - reparameterized sample.
What are the variables and shapes?
- $x_0 \in \mathbb{R}^{H \times W \times C}$ - clean image.
- $x_t \in \mathbb{R}^{H \times W \times C}$ - noisy image at timestep $t$.
- $\epsilon \in \mathbb{R}^{H \times W \times C}$ - standard Gaussian noise, $\epsilon \sim \mathcal{N}(0,I)$.
- $\beta_t \in \mathbb{R}$ - variance schedule at step $t$, typically $10^{-4}$ to $0.02$.
- $\alpha_t = 1-\beta_t$, $\bar\alpha_t = \prod_{s=1}^t \alpha_s$ - scalars.
- $a_t = \sqrt{\bar\alpha_t}$, $\sigma_t = \sqrt{1-\bar\alpha_t}$ - signal and noise coefficients, scalars with $a_t^2 + \sigma_t^2 = 1$.
Why is it needed? To add Gaussian noise stepwise over an image so that the model can learn to reverse the process. The closed form Eq. (3)/(4) lets you train on any random $t$ without simulating the full chain.
What would happen without it? Random noise would not give useful probability distributions over which we could train models. You would have no tractable way to define $q(x_t \mid x_0)$, and the reverse process would have no well-defined target.
Equation 5: SNR
\[\mathrm{SNR}(t) := \frac{\bar\alpha_t}{1-\bar\alpha_t} = \frac{a_t^2}{\sigma_t^2}\]What does it compute? Ratio of signal power to noise power at timestep $t$.
Why is it needed? It quantifies how much information about $x_0$ survives. It is the natural axis along which diffusion schedules should be compared.
What would happen without it? You could still train, but you would have no principled way to see that the terminal timestep has non-zero SNR, which is the paper’s core diagnosis.
Equations 6–9: Reverse process
- Eq. (6): $p_\theta(x_{0:T}) := p(x_T)\prod_{t=1}^T p_\theta(x_{t-1} \mid x_t)$ - reverse chain.
- Eq. (7): $p_\theta(x_{t-1} \mid x_t) := \mathcal{N}(x_{t-1}; \tilde\mu_t, \tilde\beta_t I)$ - Gaussian reverse step.
- Eq. (8): $\tilde\mu_t := \frac{1}{\sqrt{\alpha_t}}(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon)$ - posterior mean in terms of $\epsilon$.
- Eq. (9): $\tilde\beta_t := \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t$ - posterior variance.
What does it compute? The reverse denoising step, parameterized by the network’s prediction.
Why is it needed? This is the sampling process. The model learns $\tilde\mu_t$ (or equivalently $\epsilon$, $x_0$, or $v$) and iteratively denoises.
What would happen without it? No generative process. You would have a forward diffuser but no way to sample.
Equation 10: Stable Diffusion’s terminal state
\[x_T = 0.068265 \cdot x_0 + 0.997667 \cdot \epsilon\]What does it compute? The actual state at $t=T=1000$ for Stable Diffusion’s schedule.
Why is it needed? It shows the flaw concretely: $\sqrt{\bar\alpha_T} = 0.068265 \neq 0$, so signal leaks through. The model can read the channel mean from this leaked signal.
What would happen without it? You would not see the magnitude of the problem. A non-zero $a_T$ means the model learns to respect a mean that is absent at inference (where pure Gaussian noise has zero mean).
Equations 13–16: Classifier-free guidance rescaling
- Eq. (13): $x_{cfg} = x_{neg} + w(x_{pos} - x_{neg})$ - standard CFG.
- Eq. (14): $\sigma_{pos} = \mathrm{std}(x_{pos})$, $\sigma_{cfg} = \mathrm{std}(x_{cfg})$ - standard deviations.
- Eq. (15): $x_{rescaled} = x_{cfg} \cdot \frac{\sigma_{pos}}{\sigma_{cfg}}$ - rescale back to positive prompt’s std.
- Eq. (16): $x_{final} = \phi \cdot x_{rescaled} + (1-\phi) \cdot x_{cfg}$ - blend with hyperparameter $\phi$.
What does it compute? A rescaled CFG output that prevents over-exposure when terminal SNR is near zero.
Why is it needed? As $\mathrm{SNR}(T) \to 0$, CFG becomes very sensitive and produces over-exposed images. Rescaling controls the scale.
What would happen without it? Over-exposed images, especially with high guidance weights.
Equations 17–22: Sampler corrections
- Eq. (17): $\epsilon = \sqrt{\alpha_t}v + \sqrt{1-\alpha_t}x_t$ - conversion from $v$ to $\epsilon$ (problematic at $a_t=0$).
- Eq. (18): $\tilde\mu_t := \frac{1}{\sqrt{\alpha_t}}(x_t - \frac{\beta_t}{\sqrt{1-\alpha_t}}\epsilon)$ - DDPM sampling in $\epsilon$-form (zero division at $a_t=0$).
- Eq. (19): $x_0 = \sqrt{\alpha_t}x_t - \sqrt{1-\alpha_t}v$ - conversion from $v$ to $x_0$ (safe).
- Eq. (20): $\tilde\mu_t := \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t$ - DDPM sampling in $x_0$-form (safe).
- Eq. (21): $x_{t-1} = \sqrt{\bar\alpha_{t-1}}x_0 + \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\epsilon + \sigma_t z$ - DDIM update.
- Eq. (22): $\sigma_t(\eta) = \eta \sqrt{\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}}\sqrt{1-\frac{\bar\alpha_t}{\bar\alpha_{t-1}}}$ - DDIM stochasticity.
What does it compute? Sampling updates that avoid the singularity at $a_t=0$.
Why is it needed? At zero terminal SNR, $a_t=0$, so any $\epsilon$-formulation divides by zero. The $x_0$-formulation is safe.
What would happen without it? Numerical errors, NaNs, or meaningless samples at the terminal step.
v-target deep dive
Definition:
\[v = a_t \epsilon - \sigma_t x_0\]Matrix form (combining forward and v):
\[\begin{bmatrix} x_t \\ v \end{bmatrix} = \begin{bmatrix} a_t & \sigma_t \\ -\sigma_t & a_t \end{bmatrix} \begin{bmatrix} x_0 \\ \epsilon \end{bmatrix}\]This is a rotation matrix because $a_t^2 + \sigma_t^2 = 1$. It preserves norms.
Inverse relations:
\[x_0 = a_t x_t - \sigma_t v, \qquad \epsilon = \sigma_t x_t + a_t v\]No division by $a_t$ or $\sigma_t$. Safe at both endpoints.
Derivative / alternative form: For cosine schedule, $a_t = \cos\theta$, $\sigma_t = \sin\theta$, and
\[\frac{dx_t}{d\theta} = -\sin\theta\, x_0 + \cos\theta\, \epsilon = a_t \epsilon - \sigma_t x_0 = v\]So $v$ is literally the velocity of the probability-flow ODE. Popularized by Progressive Distillation (2202.00512).
Equivalence note: $a_t$ and $\sigma_t$ vs invariants of paper - they mean the same thing. $a_t$ signal coefficient. $\sigma_t$ noise coefficients
- $a_t = 1, \sigma_t = 0 \rightarrow$ low noise
- $a_t = 0, \sigma_t = 1 \rightarrow$ high noise (zero terminal SNR)
Option A (old): Target $\epsilon$
\[\mathcal{L} = \lVert \epsilon - \epsilon_\theta(x_t,t) \rVert^2\]- Low noise: $x_t \approx x_0 \rightarrow$ small loss & weak gradient
- High noise: $x_t \approx \epsilon \rightarrow$ trivial copy task, no learning
Option B (old): Target $x_0$
\[\mathcal{L} = \lVert x_0 - \hat x_0(x_t,t) \rVert^2\]- Low noise: easy
- High noise: very hard, unstable, huge gradients
New: Target $v$
\[\mathcal{L} = \lVert v_t - v_\theta(x_t,t) \rVert^2\]- Low noise: $v_t \approx \epsilon$ (network focuses on noise - good!)
- High noise: $v_t \approx -x_0$ (network focuses on signal - great!)
- Medium noise: balanced mix, unit-scale target throughout
bottom line: this gives better regression and more expressive gradients. The target scale stays roughly constant across all $t$, so the loss is well-conditioned. At $t=T$ with zero terminal SNR, $v_T = -x_0$, so the model is explicitly trained to output the conditional data mean from pure noise - exactly what inference needs.
Key equations/ideas
- $x_t = a_t x_0 + \sigma_t \epsilon$ - forward diffusion
- $v = a_t \epsilon - \sigma_t x_0$ - better target for diffusion training
- Zero terminal SNR: $a_T = 0$ - align training and inference
- Start sampling from $t=T$ - match training distribution
- CFG rescale: $x_{final} = \phi \cdot x_{rescaled} + (1-\phi) \cdot x_{cfg}$ - prevent over-exposure
Metrics and baselines
What metric? FID (lower better), IS (higher better).
What baselines? SD v2.1-base official - FID 23.76, IS 32.84.
What is the headline number? FID 21.66 (lower is better), IS 36.16 (higher is better).
Do ablations show which components matter?
- Sample step selection: When $S$ is small (e.g. $S=5$), Trailing noticeably outperforms Linspace. For common choices such as $S=25$, the difference is not easily noticeable.
- CFG rescale: When $\phi=0$ (regular CFG), images overexpose. $\phi \in [0.5, 0.75]$ produces the most appealing results. Paper uses $\phi=0.7$.
- $v$-prediction: Paper finetunes with $v$-loss and finds visual quality similar to $\epsilon$-loss, but it is necessary for zero terminal SNR to avoid singularity.
Does the evidence actually support the claim? Yes. Empirically (Table 3, FID/IS) and visually (Figure 3, brightness range; Figure 6, CFG ablation) the ablations show the claim true. The qualitative comparison in Figure 3 is particularly strong: the flawed model cannot generate “Solid black background” or “white background” while the fixed model can.
weaknesses
-
Assumes variance-preserving formulation. The paper explicitly states that variance-exploding formulation cannot truly reach zero terminal SNR. This limits the applicability to VP diffusion models. Score-based models using VE SDEs are not covered. … is this really a weakness or just a design constraint.. i’m not sure.
-
No test on other modalities or architectures. The experiments are all on Stable Diffusion (latent image diffusion). It is unclear whether the same fixes transfer to video diffusion, audio diffusion, or 3D generation. The brightness problem may manifest differently in other domains. Do these principles generalize? My intuition is that anything trained with diffusion should generalize to this.
-
Trailing vs Linspace difference is small for common $S$. The paper admits that for $S=25$, the difference between trailing and linspace is not easily noticeable. So one of the four fixes may be less critical in practice.
some thoughts
I’m sure there are papers that have tried to solve for these since this paper, but I haven’t done a thorough review yet, so just for future ref, these are some things I was thinking right now.
-
Is there a way to rescale the schedule at inference time only, or to train a lightweight adapter that corrects the terminal SNR mismatch without full finetuning?
-
The paper uses a constant $\phi=0.7$. Could $\phi$ be annealed over timesteps or made prompt-dependent, so we have lesser hyper-params to tune
-
Why does the model learn to read the leaked mean at all? The paper says the leaked signal contains low-frequency information like channel mean, and the model learns to respect it. But is this a bug or a feature? Could the leaked signal be used for improved conditional generation if handled correctly?
breadcrumbs
- diffusion-notes - overview map; the guidance and sampling rows point back here
- ddpm eq-by-equation (2006.11239) - the base objective this schedule work builds on
- improved ddpm (2102.09672) - the previous stop on the schedule line
todos
- the three self-posed questions in some thoughts (inference-only schedule rescaling, annealing $\phi$, leaked-mean bug-or-feature) are the real backlog - turn each into a mini write-up when chased.
- i admitted no thorough follow-up review was done - spend one evening on papers since 2023 citing this, then update the weaknesses section.