Coupled Inference in Diffusion Models for Semantic Decomposition

TL;DR

This paper introduces a novel coupled inference framework using diffusion models for semantic decomposition, improving over resonator networks in recognizing and reconstructing compositional visual scene factors.

Contribution

It presents a new method that couples diffusion processes for semantic decomposition, including a reconstruction-guided guidance term and an iterative sampling scheme, unifying resonator networks as a special case.

Findings

Outperforms resonator networks on synthetic tasks

Introduces a reconstruction-driven guidance term

Develops an iterative sampling scheme for better performance

Abstract

Many visual scenes can be described as compositions of latent factors. Effective recognition, reasoning, and editing often require not only forming such compositional representations, but also solving the decomposition problem. One popular choice for constructing these representations is through the binding operation. Resonator networks, which can be understood as coupled Hopfield networks, were proposed as a way to perform decomposition on such bound representations. Recent works have shown notable similarities between Hopfield networks and diffusion models. Motivated by these observations, we introduce a framework for semantic decomposition using coupled inference in diffusion models. Our method frames semantic decomposition as an inverse problem and couples the diffusion processes using a reconstruction-driven guidance term that encourages the composition of factor estimates to match the bound vector. We also introduce a novel iterative sampling scheme that improves the performance of our model. Finally, we show that attention-based resonator networks are a special case of our framework. Empirically, we demonstrate that our coupled inference framework outperforms resonator networks across a range of synthetic semantic decomposition tasks.