ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks

TL;DR

This paper introduces FormatBench, a comprehensive benchmark for evaluating format faithfulness in large language models across diverse tasks, and proposes ReFF, a method to significantly improve LLMs' ability to generate correctly formatted outputs without sacrificing quality.

Contribution

The paper presents FormatBench, a new diverse benchmark for format faithfulness, and proposes ReFF, a novel reinforcement approach that enhances LLMs' formatting accuracy without needing annotated data.

Findings

ReFF improves format faithfulness from 21.6% to 95.0% without quality loss.

ReFF enhances both format accuracy and general quality when combined with labeled data.

State-of-the-art LLMs still struggle with format faithfulness across varied tasks.

Abstract

Following formatting instructions to generate well-structured content is a fundamental yet often unmet capability for large language models (LLMs). To study this capability, which we refer to as format faithfulness, we present FormatBench, a comprehensive format-related benchmark. Compared to previous format-related benchmarks, FormatBench involves a greater variety of tasks in terms of application scenes (traditional NLP tasks, creative works, autonomous agency tasks), human-LLM interaction styles (single-turn instruction, multi-turn chat), and format types (inclusion, wrapping, length, coding). Moreover, each task in FormatBench is attached with a format checker program. Extensive experiments on the benchmark reveal that state-of-the-art open- and closed-source LLMs still suffer from severe deficiency in format faithfulness. By virtue of the decidable nature of formats, we propose to Reinforce Format Faithfulness (ReFF) to help LLMs generate formatted output as instructed without compromising general quality. Without any annotated data, ReFF can substantially improve the format faithfulness rate (e.g., from 21.6% in original LLaMA3 to 95.0% on caption segmentation task), while keep the general quality comparable (e.g., from 47.3 to 46.4 in F1 scores). Combined with labeled training data, ReFF can simultaneously improve both format faithfulness (e.g., from 21.6% in original LLaMA3 to 75.5%) and general quality (e.g., from 47.3 to 61.6 in F1 scores). We further offer an interpretability analysis to explain how ReFF improves both format faithfulness and general quality.