Does This Summary Answer My Question? Modeling Query-Focused Summary Readers with Rational Speech Acts

TL;DR

This paper enhances query-focused summarization by modeling user understanding through the RSA framework, improving summary relevance and alignment with user queries by re-ranking candidate summaries based on answer reconstruction.

Contribution

It introduces an answer reconstruction objective within the RSA framework to explicitly model user understanding and improve summary relevance in QFS systems.

Findings

Re-ranking with answer reconstruction improves summary relevance.

Explicit modeling of user understanding enhances QFS performance.

The approach aligns summaries more closely with user queries.

Abstract

Query-focused summarization (QFS) is the task of generating a summary in response to a user-written query. Despite its user-oriented nature, there has been limited work in QFS in explicitly considering a user's understanding of a generated summary, potentially causing QFS systems to underperform at inference time. In this paper, we adapt the Rational Speech Act (RSA) framework, a model of human communication, to explicitly model a reader's understanding of a query-focused summary and integrate it within the generation method of existing QFS systems. In particular, we introduce the answer reconstruction objective which approximates a reader's understanding of a summary by their ability to use it to reconstruct the answer to their initial query. Using this objective, we are able to re-rank candidate summaries generated by existing QFS systems and select summaries that better align with their corresponding query and reference summary. More generally, our study suggests that a simple and effective way of improving a language generation system designed for a user-centered task may be to explicitly incorporate its user requirements into the system's generation procedure.