(lit review as of 07/2024)

Given a pretrained diffusion model, we seek to generate conditional samples based on an observed signal $y$. For example, we many be given a noisy image and tasked with denoising it, or a black and white image and tasked with recoloring it.

One approach seeks to augment the dynamics of the pretrained diffusion model. We call these guided diffusion models, after ‘guided diffusion’ (Ho & Salimans, 2022).

Early approaches were rather heuristic, for example; a mask-based approach (Lugmayr et al., 2022), SVD inspired (Kawar et al., 2022), null space projection (Wang et al., 2022).

Next came diffusion posterior sampling (DPS), a more principled approach. It starts by rewriting the diffusion SDE to use the unknown posterior score, $\nabla_x \log p(x \mid y)$, rather than the prior score, $\nabla_x \log p(x)$.

\[\begin{align*} dx &= \left[ f(x, t) - g(t)^2 \nabla_x \log p_t(x) \right] dt + g(t) dw \tag{unconditional SDE} \\ dx &= \left[ f(x, t) - g(t)^2 \nabla_x \log p_t(x | y) \right] dt + g(t) dw \tag{conditional SDE} \\ \end{align*}\]

This allows us to generate samples from the posterior by solving the conditional SDE.

But, we don’t know the score of the posterior, so we use Bayes’ rule to rewrite the posterior score in terms of the likelihood and prior scores.

\[\begin{align*} p(x | y) &= \frac{p(y | x) p(x)}{p(y)} \\ \log p(x | y) &= \log p(y | x) + \log p(x) - \log p(y) \\ \nabla_x \log p(x | y) &= \nabla_x \log p(y | x) + \nabla_x \log p(x) \\ \end{align*}\]

DPS (Chung et al., 2023), $\Pi$GDM (Song et al., 2023) and others have shown that it is possible to construct / approximate $\nabla_x \log p_t(x \mid y)$. Note that $\Pi$GDM has also been applied to flows (Pokle et al., 2024).

\[\begin{align*} \nabla_x \log p(y | x_t) &\approx \nabla_x \parallel y - C(x) \parallel^2_2 \tag{DPS} \\ \nabla_x \log p(y | x_t) &\approx (y - H\hat x)^T (r_t^2 H H^T + \sigma^2I)^{-1} H \frac{\partial \hat x_t}{\partial x_t} \tag{$\Pi$GDM} \end{align*}\]

In parallel, a variational approach frames the conditional generation problem as an optimization problem (Ben-Hamu et al., 2024; Mardani et al., 2023; Mardani et al., 2023).

\[\begin{align*} x^* &= \arg \min_z \nabla_z \parallel y - f(D(z)) \parallel^2_2 \tag{DPS} \\ \end{align*}\]

Where $D$ is the diffusion model, $f$ is the forward model, $z$ is the latent variable and $y$ is the observed signal. These variational approaches proidve high quality samples, but are computationally expensive (approx 5-15 minutes for ImageNet-128 with an NVidia V100 GPU).

And finally (Dou & Song, 2024) present a Bayesian filtering perspective which leads to an algorithms that converges to the true posterior.

Bibliography

  1. Ho, J., & Salimans, T. (2022). Classifier-Free Diffusion Guidance. https://arxiv.org/abs/2207.12598
  2. Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022). RePaint: Inpainting using Denoising Diffusion Probabilistic Models. arXiv. http://arxiv.org/abs/2201.09865
  3. Kawar, B., Ermon, S., Elad, M., & Song, J. (2022). Denoising Diffusion Restoration Models. Neural Information Processing Systems.
  4. Wang, Y., Yu, J., & Zhang, J. (2022). Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. arXiv. http://arxiv.org/abs/2212.00490
  5. Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., & Ye, J. C. (2023). Diffusion Posterior Sampling for General Noisy Inverse Problems. arXiv. http://arxiv.org/abs/2209.14687
  6. Song, J., Vahdat, A., Mardani, M., & Kautz, J. (2023). Pseudoinverse-Guided Diffusion Models for Inverse Problems. ICML. https://openreview.net/forum?id=9_gsMA8MRKQ
  7. Pokle, A., Muckley, M. J., Chen, R. T. Q., & Karrer, B. (2024). Training-free Linear Image Inverses via Flows. arXiv. http://arxiv.org/abs/2310.04432
  8. Ben-Hamu, H., Puny, O., Gat, I., Karrer, B., Singer, U., & Lipman, Y. (2024). D-Flow: Differentiating through Flows for Controlled Generation. https://arxiv.org/abs/2402.14017
  9. Mardani, M., Song, J., Kautz, J., & Vahdat, A. (2023). A Variational Perspective on Solving Inverse Problems with Diffusion Models. arXiv. http://arxiv.org/abs/2305.04391
  10. Mardani, M., Song, J., Kautz, J., & Vahdat, A. (2023). A Variational Perspective on Solving Inverse Problems with Diffusion Models. arXiv. http://arxiv.org/abs/2305.04391
  11. Dou, Z., & Song, Y. (2024). DIFFUSION POSTERIOR SAMPLING FOR LINEAR INVERSE PROBLEM SOLVING — A FILTERING PERSPECTIVE.