Factored Classifier-Free Guidance

TL;DR

This paper introduces Factored Classifier-Free Guidance (FCFG), a novel method for counterfactual generation with diffusion models that allows attribute-wise control to reduce spurious changes and improve causal fidelity.

Contribution

The paper proposes FCFG, a flexible guidance technique that enables attribute-wise control in diffusion-based counterfactuals, addressing CFG's limitations and extending to advanced guidance schemes.

Findings

FCFG improves the causal soundness of counterfactuals in image datasets.

It reduces spurious attribute changes in counterfactual generation.

Enhances counterfactual reversibility and fidelity in natural and medical images.

Abstract

Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions. Diffusion models have emerged as a powerful tool for this task, combining DDIM inversion with conditional generation and classifier-free guidance (CFG). In this work, we identify a key limitation of CFG for counterfactual generation: it prescribes a global guidance scale for all attributes, leading to significant spurious changes in inferred counterfactuals. To mitigate this, we propose Factored Classifier-Free Guidance (FCFG), a flexible and model-agnostic guidance technique that enables attribute-wise control following a causal graph. FCFG complements recent advances in classifier-free guidance and can be seamlessly extended to advanced guidance schemes such as CFG++ and APG. Our experiments demonstrate that FCFG significantly improves the axiomatic soundness of inferred counterfactuals across both natural and medical image datasets, mitigating spurious amplification effects, and enhancing counterfactual reversibility.