CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization

TL;DR

This paper introduces CCSBench, a benchmark for evaluating the ability of large language models to perform compositional control over multiple attributes in scientific document summarization, highlighting current limitations in balancing explicit and implicit attributes.

Contribution

The paper presents CCSBench, the first benchmark for assessing compositional controllability in scientific summarization, and provides extensive experiments revealing LLMs' limitations in this task.

Findings

LLMs struggle to balance control over multiple attributes.

Implicit attributes require deeper understanding and are harder to control.

Current methods show significant trade-offs in attribute control.

Abstract

To broaden the dissemination of scientific knowledge to diverse audiences, it is desirable for scientific document summarization systems to simultaneously control multiple attributes such as length and empirical focus. However, existing research typically focuses on controlling single attributes, leaving the compositional control of multiple attributes underexplored. To address this gap, we introduce CCSBench, the first evaluation benchmark for compositional controllable summarization in the scientific domain. Our benchmark enables fine-grained control over both explicit attributes (e.g., length), which are objective and straightforward, and implicit attributes (e.g., conceptual or empirical focus), which are more subjective and abstract. We conduct extensive experiments using various large language models (LLMs) under various settings, including in-context learning, parameter-efficient fine-tuning, and two-stage modular methods for balancing control over different attributes. Our findings reveal significant limitations in LLMs capabilities in balancing trade-offs between control attributes, especially implicit ones that require deeper understanding and abstract reasoning.