Length Controlled Generation for Black-box LLMs

TL;DR

This paper introduces an iterative sampling framework combining Metropolis-Hastings and importance sampling to control text length in black-box LLMs, achieving high success rates without fine-tuning.

Contribution

A novel length control method for LLMs that does not require parameter fine-tuning, using an efficient iterative sampling approach.

Findings

Achieves nearly 100% length control success on Llama3.1.

Works effectively for summarization and instruction tasks.

Minimal additional computational overhead.

Abstract

Large language models (LLMs) have demonstrated impressive instruction following capabilities, while still struggling to accurately manage the length of the generated text, which is a fundamental requirement in many real-world applications. Existing length control methods involve fine-tuning the parameters of LLMs, which is inefficient and suboptimal for practical use. In this paper, we propose a novel iterative sampling framework for text length control, integrating the Metropolis-Hastings algorithm with an importance sampling acceleration strategy. This framework efficiently and reliably regulates LLMs to generate length-constrained text without modifying the underlying parameters, thereby preserving the original capabilities of LLMs. Experimental results demonstrate that our framework achieves almost 100\% success rates of length control on Llama3.1 for tasks such as length-controlled abstractive summarization and length-constrained instruction following, with minimal additional computational overhead. This also highlights the significant potential of our method for precise length control across a broader range of applications, without compromising the versatility of LLMs.