LCTG Bench: LLM Controlled Text Generation Benchmark

TL;DR

This paper introduces LCTG Bench, a comprehensive Japanese benchmark for evaluating and comparing the controllability of large language models, addressing language diversity and unified evaluation challenges.

Contribution

It presents the first Japanese controllability benchmark for LLMs, providing a unified framework for model assessment across diverse use cases.

Findings

Multilingual models lag behind Japanese-specific models in controllability.

LCTG Bench enables effective model selection based on controllability.

Current Japanese LLMs show significant controllability gaps.

Abstract

The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructions. In response, benchmarks focusing on the controllability of LLMs have been developed, but several issues remain: (1) They primarily cover major languages like English and Chinese, neglecting low-resource languages like Japanese; (2) Current benchmarks employ task-specific evaluation metrics, lacking a unified framework for selecting models based on controllability across different use cases. To address these challenges, this research introduces LCTG Bench, the first Japanese benchmark for evaluating the controllability of LLMs. LCTG Bench provides a unified framework for assessing control performance, enabling users to select the most suitable model for their use cases based on controllability. By evaluating nine diverse Japanese-specific and multilingual LLMs like GPT-4, we highlight the current state and challenges of controllability in Japanese LLMs and reveal the significant gap between multilingual models and Japanese-specific models.