Instance-Adaptive Parametrization for Amortized Variational Inference

TL;DR

The paper introduces IA-VAE, an adaptive inference framework that uses hypernetworks to generate input-dependent encoder modulations, improving posterior approximation and reducing the amortization gap in VAEs.

Contribution

It proposes a novel instance-adaptive variational autoencoder that enhances inference flexibility with input-specific modulations, outperforming standard VAEs in accuracy and efficiency.

Findings

IA-VAE yields more accurate posterior approximations on synthetic data.

It improves held-out ELBO on image benchmarks compared to baseline VAEs.

The approach achieves similar performance with fewer parameters.

Abstract

Variational autoencoders (VAEs) rely on amortized variational inference to enable efficient posterior approximation, but this efficiency comes at the cost of a shared parametrization, giving rise to the amortization gap. We propose the instance-adaptive variational autoencoder (IA-VAE), an amortized inference framework in which a hypernetwork generates input-dependent modulations of a shared encoder. This enables input-specific adaptation of the inference model while preserving the efficiency of a single forward pass. From a theoretical perspective, we show that the variational family induced by IA-VAE contains that of standard amortized inference, implying that IA-VAE cannot yield a worse optimal ELBO. By leveraging instance-specific parameter modulations, the proposed approach can achieve performance comparable to standard encoders with substantially fewer parameters, indicating a more efficient use of model capacity. Experiments on synthetic data, where the true posterior is known, show that IA-VAE yields more accurate posterior approximations and reduces the amortization gap. Similarly, on standard image benchmarks, IA-VAE consistently improves held-out ELBO over baseline VAEs, with statistically significant gains across multiple runs. These results suggest that increasing the flexibility of the inference parametrization through instance-adaptive modulation is an effective strategy for mitigating amortization-induced suboptimality in deep generative models.