Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs

TL;DR

This paper introduces metrics and evaluation frameworks for assessing and achieving smooth control over attribute intensity in text generated by large language models, focusing on prompt-based and internal representation methods.

Contribution

It proposes novel metrics and an evaluation framework for attribute control in LLMs, and explores two training-free methods for smooth control of attribute intensity.

Findings

Metrics effectively measure attribute range, calibration, and relevance.

Prompting with semantic shifters and internal modifications enable smooth control.

Evaluation on five attributes across various models demonstrates effectiveness.

Abstract

Controlling the attribute intensity of text generation is crucial across scenarios (e.g., writing conciseness, chatting emotion, and explanation clarity). The remarkable capabilities of large language models (LLMs) have revolutionized text generation, prompting us to explore such \emph{smooth control} of LLM generation. Specifically, we propose metrics to assess the range, calibration, and consistency of the generated text's attribute intensity in response to varying control values, as well as its relevance to the intended context. To quantify the attribute intensity and context relevance, we propose an effective evaluation framework leveraging the Elo rating system and GPT4, both renowned for their robust alignment with human judgment. We look into two viable training-free methods for achieving smooth control of LLMs: (1) Prompting with semantic shifters, and (2) Modifying internal model representations. The evaluations of these two methods are conducted on $5$ different attributes with various models. Our code and dataset can be obtained from \url{https://github.com/ShangDataLab/Smooth-Control}.