Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders

TL;DR

This paper reveals that VAE encoders distilled at low resolutions can generalize to higher resolutions, improving high-resolution image reconstruction without training on high-res data, by learning resolution-consistent latent manifolds.

Contribution

It uncovers a counter-intuitive phenomenon where low-resolution distilled VAEs perform better at higher resolutions and analyzes the underlying latent space behavior.

Findings

Distilled VAEs at low resolution generalize to higher resolutions.

Resolution remapping improves reconstruction metrics significantly.

Latent distributions align more closely with the teacher at higher resolutions.

Abstract

Variational Autoencoder (VAE) encoders play a critical role in modern generative models, yet their computational cost often motivates the use of knowledge distillation or quantification to obtain compact alternatives. Existing studies typically believe that the model work better on the samples closed to their training data distribution than unseen data distribution. In this work, we report a counter-intuitive phenomenon in VAE encoder distillation: a compact encoder distilled only at low resolutions exhibits poor reconstruction performance at its native resolution, but achieves dramatically improved results when evaluated at higher, unseen input resolutions. Despite never being trained beyond $256^2$ resolution, the distilled encoder generalizes effectively to $512^2$ resolution inputs, partially inheriting the teacher model's resolution preference.We further analyze latent distributions across resolutions and find that higher-resolution inputs produce latent representations more closely aligned with the teacher's manifold. Through extensive experiments on ImageNet-256, we show that simple resolution remapping-upsampling inputs before encoding and downsampling reconstructions for evaluation-leads to substantial gains across PSNR, MSE, SSIM, LPIPS, and rFID metrics. These findings suggest that VAE encoder distillation learns resolution-consistent latent manifolds rather than resolution-specific pixel mappings. This also means that the high training cost on memory, time and high-resolution datasets are not necessary conditions for distilling a VAE with high-resolution image reconstruction capabilities. On low resolution datasets, the distillation model still could learn the detailed knowledge of the teacher model in high-resolution image reconstruction.