Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo

TL;DR

This paper introduces a novel probabilistic inference method for language models using twisted Sequential Monte Carlo, improving sampling efficiency and evaluation accuracy for various LLM safety and generation tasks.

Contribution

It develops learned twist functions for SMC in language models, proposes a contrastive learning approach for these functions, and introduces bidirectional SMC bounds for inference evaluation.

Findings

Twisted SMC effectively samples undesirable outputs for red-teaming.

The method generates diverse sentiment reviews.

It accurately estimates the KL divergence between distributions.

Abstract

Numerous capability and safety techniques of Large Language Models (LLMs), including RLHF, automated red-teaming, prompt engineering, and infilling, can be cast as sampling from an unnormalized target distribution defined by a given reward or potential function over the full sequence. In this work, we leverage the rich toolkit of Sequential Monte Carlo (SMC) for these probabilistic inference problems. In particular, we use learned twist functions to estimate the expected future value of the potential at each timestep, which enables us to focus inference-time computation on promising partial sequences. We propose a novel contrastive method for learning the twist functions, and establish connections with the rich literature of soft reinforcement learning. As a complementary application of our twisted SMC framework, we present methods for evaluating the accuracy of language model inference techniques using novel bidirectional SMC bounds on the log partition function. These bounds can be used to estimate the KL divergence between the inference and target distributions in both directions. We apply our inference evaluation techniques to show that twisted SMC is effective for sampling undesirable outputs from a pretrained model (a useful component of harmlessness training and automated red-teaming), generating reviews with varied sentiment, and performing infilling tasks.