Finetuning Pretrained Transformers into Variational Autoencoders

TL;DR

This paper introduces a simple two-phase finetuning method to convert pretrained Transformers into variational autoencoders, addressing posterior collapse without extensive pretraining, and evaluates its effectiveness.

Contribution

It proposes a novel, resource-efficient finetuning scheme to transform Transformers into VAEs, enabling broader access and application.

Findings

Competitive performance with large-scale pretrained VAEs on some metrics

Effective mitigation of posterior collapse through the proposed method

Comprehensive analysis of existing collapse alleviation techniques

Abstract

Text variational autoencoders (VAEs) are notorious for posterior collapse, a phenomenon where the model's decoder learns to ignore signals from the encoder. Because posterior collapse is known to be exacerbated by expressive decoders, Transformers have seen limited adoption as components of text VAEs. Existing studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et al., 2021) mitigate posterior collapse using massive pretraining, a technique unavailable to most of the research community without extensive computing resources. We present a simple two-phase training scheme to convert a sequence-to-sequence Transformer into a VAE with just finetuning. The resulting language model is competitive with massively pretrained Transformer-based VAEs in some internal metrics while falling short on others. To facilitate training we comprehensively explore the impact of common posterior collapse alleviation techniques in the literature. We release our code for reproducability.