# Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?

## TL;DR

This paper investigates the impact of exposure bias in autoregressive language models, finding that its effects are limited and that models exhibit a self-recovery ability that mitigates potential distortions during open-ended text generation.

## Contribution

The study introduces metrics to quantify exposure bias effects and provides empirical evidence that exposure bias is less severe than previously believed, highlighting the self-recovery ability of language models.

## Key findings

- Exposure bias effects are limited in impact.
- Models demonstrate a strong self-recovery ability.
- Distortions do not significantly accumulate during generation.

## Abstract

Exposure bias has been regarded as a central problem for auto-regressive language models (LM). It claims that teacher forcing would cause the test-time generation to be incrementally distorted due to the training-generation discrepancy. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we focus on the task of open-ended language generation, propose metrics to quantify the impact of exposure bias in the aspects of quality, diversity, and consistency. Our key intuition is that if we feed ground-truth data prefixes (instead of prefixes generated by the model itself) into the model and ask it to continue the generation, the performance should become much better because the training-generation discrepancy in the prefix is removed. Both automatic and human evaluations are conducted in our experiments. On the contrary to the popular belief in exposure bias, we find that the the distortion induced by the prefix discrepancy is limited, and does not seem to be incremental during the generation. Moreover, our analysis reveals an interesting self-recovery ability of the LM, which we hypothesize to be countering the harmful effects from exposure bias.