Fair Summarization: Bridging Quality and Diversity in Extractive Summaries

TL;DR

This paper presents two novel methods, FairExtract and FairGPT, for fair extractive summarization that balances quality and fairness across social groups, evaluated on a diverse dataset with improved fairness metrics.

Contribution

Introduction of two new fair summarization methods, FairExtract and FairGPT, that effectively balance fairness and quality in multi-document extractive summarization.

Findings

FairExtract and FairGPT outperform baselines in fairness metrics.

Both methods maintain competitive summarization quality.

Composite metrics provide nuanced evaluation of fairness and quality trade-offs.

Abstract

Fairness in multi-document summarization of user-generated content remains a critical challenge in natural language processing (NLP). Existing summarization methods often fail to ensure equitable representation across different social groups, leading to biased outputs. In this paper, we introduce two novel methods for fair extractive summarization: FairExtract, a clustering-based approach, and FairGPT, which leverages GPT-3.5-turbo with fairness constraints. We evaluate these methods using Divsumm summarization dataset of White-aligned, Hispanic, and African-American dialect tweets and compare them against relevant baselines. The results obtained using a comprehensive set of summarization quality metrics such as SUPERT, BLANC, SummaQA, BARTScore, and UniEval, as well as a fairness metric F, demonstrate that FairExtract and FairGPT achieve superior fairness while maintaining competitive summarization quality. Additionally, we introduce composite metrics (e.g., SUPERT+F, BLANC+F) that integrate quality and fairness into a single evaluation framework, offering a more nuanced understanding of the trade-offs between these objectives. Our code is available online.