Improving Constrained Language Generation via Self-Distilled Twisted Sequential Monte Carlo

TL;DR

This paper enhances constrained language generation by combining twisted Sequential Monte Carlo with self-distillation, improving the alignment of the base model with the target distribution and resulting in higher quality outputs.

Contribution

It introduces a novel method that iteratively refines the base language model using self-distillation within the twisted Sequential Monte Carlo framework for better constrained generation.

Findings

Significant improvement in generation quality.

Better alignment with target distribution.

Effective handling of sparse reward signals.

Abstract

Recent work has framed constrained text generation with autoregressive language models as a probabilistic inference problem. Among these, Zhao et al. (2024) introduced a promising approach based on twisted Sequential Monte Carlo, which incorporates learned twist functions and twist-induced proposals to guide the generation process. However, in constrained generation settings where the target distribution concentrates on outputs that are unlikely under the base model, learning becomes challenging due to sparse and uninformative reward signals. We show that iteratively refining the base model through self-distillation alleviates this issue by making the model progressively more aligned with the target, leading to substantial gains in generation quality.