A Tilted Seesaw: Revisiting Autoencoder Trade-off for Controllable Diffusion

TL;DR

This paper critically examines the autoencoder evaluation metrics in latent diffusion models, revealing that reconstruction fidelity better predicts controllability than generative metrics like gFID, especially when scaling to controllable diffusion tasks.

Contribution

It provides a theoretical and empirical analysis showing the limitations of gFID-focused evaluation and proposes a more reliable multi-dimensional assessment for controllability in diffusion models.

Findings

gFID is weakly predictive of condition preservation

Reconstruction metrics better indicate controllability

Autoencoder evaluation bias affects controllable diffusion performance

Abstract

In latent diffusion models, the autoencoder (AE) is typically expected to balance two capabilities: faithful reconstruction and a generation-friendly latent space (e.g., low gFID). In recent ImageNet-scale AE studies, we observe a systematic bias toward generative metrics in handling this trade-off: reconstruction metrics are increasingly under-reported, and ablation-based AE selection often favors the best-gFID configuration even when reconstruction fidelity degrades. We theoretically analyze why this gFID-dominant preference can appear unproblematic for ImageNet generation, yet becomes risky when scaling to controllable diffusion: AEs can induce condition drift, which limits achievable condition alignment. Meanwhile, we find that reconstruction fidelity, especially instance-level measures, better indicates controllability. We empirically validate the impact of tilted autoencoder evaluation on controllability by studying several recent ImageNet AEs. Using a multi-dimensional condition-drift evaluation protocol reflecting controllable generation tasks, we find that gFID is only weakly predictive of condition preservation, whereas reconstruction-oriented metrics are substantially more aligned. ControlNet experiments further confirm that controllability tracks condition preservation rather than gFID. Overall, our results expose a gap between ImageNet-centric AE evaluation and the requirements of scalable controllable diffusion, offering practical guidance for more reliable benchmarking and model selection.