Improving Reconstruction of Representation Autoencoder

TL;DR

This paper introduces LV-RAE, a representation autoencoder that enhances image reconstruction fidelity by combining semantic features with low-level details, and improves generative quality through decoder fine-tuning and noise smoothing.

Contribution

LV-RAE is a novel autoencoder that augments semantic features with low-level information and employs decoder fine-tuning and noise injection for improved reconstruction and generation.

Findings

LV-RAE achieves higher reconstruction fidelity.

It maintains semantic consistency in generated images.

Decoder robustness improvements reduce artifacts.

Abstract

Recent work leverages Vision Foundation Models as image encoders to boost the generative performance of latent diffusion models (LDMs), as their semantic feature distributions are easy to learn. However, such semantic features often lack low-level information (\eg, color and texture), leading to degraded reconstruction fidelity, which has emerged as a primary bottleneck in further scaling LDMs. To address this limitation, we propose LV-RAE, a representation autoencoder that augments semantic features with missing low-level information, enabling high-fidelity reconstruction while remaining highly aligned with the semantic distribution. We further observe that the resulting high-dimensional, information-rich latent make decoders sensitive to latent perturbations, causing severe artifacts when decoding generated latent and consequently degrading generation quality. Our analysis suggests that this sensitivity primarily stems from excessive decoder responses along directions off the data manifold. Building on these insights, we propose fine-tuning the decoder to increase its robustness and smoothing the generated latent via controlled noise injection, thereby enhancing generation quality. Experiments demonstrate that LV-RAE significantly improves reconstruction fidelity while preserving the semantic abstraction and achieving strong generative quality. Our code is available at https://github.com/modyu-liu/LVRAE.