CIE: Controlling Language Model Text Generations Using Continuous Signals

TL;DR

This paper introduces CIE, a method for controlling language model text generation using continuous signals, enabling more reliable and scalable control over properties like response length.

Contribution

The paper presents a novel finetuning approach that uses continuous control vectors to steer language model outputs, improving over previous discrete or prompt-based methods.

Findings

More reliable control of response length compared to existing methods

Effective interpolation of control signals in embedding space

Scalable approach for continuous property control in language models

Abstract

Aligning language models (LMs) with user intent is becoming increasingly relevant to enhance user experience. This calls for designing methods that can allow users to control the properties of the language that LMs generate, for example, controlling the length of the generation or the complexity of the language that gets chosen. Most existing work attempts to integrate users' control by conditioning LM generations on natural language prompts or discrete control signals, which are often brittle and hard to scale. In this work, we are interested in continuous control signals, ones that exist along a spectrum that can't easily be captured in a natural language prompt or via existing techniques in conditional generation. Through a case study in controlling the precise response-length of generations, we demonstrate how an LM can be finetuned to expect a control vector that is interpolated between a "low" and a "high" token embedding. Our method more reliably exerts response-length control than in-context learning methods or fine-tuning methods that represent the control signal as a discrete signal.