LCTG Bench: LLM Controlled Text Generation Benchmark
Kentaro Kurihara, Masato Mita, Peinan Zhang, Shota Sasaki, Ryosuke, Ishigami, Naoaki Okazaki

TL;DR
This paper introduces LCTG Bench, a comprehensive Japanese benchmark for evaluating and comparing the controllability of large language models, addressing language diversity and unified evaluation challenges.
Contribution
It presents the first Japanese controllability benchmark for LLMs, providing a unified framework for model assessment across diverse use cases.
Findings
Multilingual models lag behind Japanese-specific models in controllability.
LCTG Bench enables effective model selection based on controllability.
Current Japanese LLMs show significant controllability gaps.
Abstract
The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructions. In response, benchmarks focusing on the controllability of LLMs have been developed, but several issues remain: (1) They primarily cover major languages like English and Chinese, neglecting low-resource languages like Japanese; (2) Current benchmarks employ task-specific evaluation metrics, lacking a unified framework for selecting models based on controllability across different use cases. To address these challenges, this research introduces LCTG Bench, the first Japanese benchmark for evaluating the controllability of LLMs. LCTG Bench provides a unified framework for assessing control performance, enabling users to select the most suitable model for their use…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Mathematics, Computing, and Information Processing
MethodsAttention Is All You Need · Softmax · Residual Connection · Dropout · Absolute Position Encodings · Byte Pair Encoding · Linear Layer · Multi-Head Attention · Position-Wise Feed-Forward Layer · Label Smoothing
