Improving Fairness of Large Language Models in Multi-document Summarization

TL;DR

This paper introduces FairPO, a novel preference tuning method that enhances both summary-level and corpus-level fairness in multi-document summarization, ensuring fairer and more balanced summaries across diverse social attributes.

Contribution

The paper presents FairPO, a new preference tuning approach that simultaneously improves summary-level and corpus-level fairness in multi-document summarization tasks.

Findings

FairPO outperforms strong baselines in fairness metrics.

It maintains summary quality while enhancing fairness.

Dynamic adjustment of preference pair weights improves corpus-level fairness.

Abstract

Fairness in multi-document summarization (MDS) is crucial for providing comprehensive views across documents with diverse social attribute values, which can significantly impact decision-making. For example, a summarization system that tends to overrepresent negative reviews of products can mislead customers into disregarding good products. Previous works measure fairness in MDS at two levels: summary-level and corpus-level. While summary-level fairness focuses on individual summaries, corpus-level fairness focuses on a corpus of summaries. Recent methods primarily focus on summary-level fairness. We propose FairPO, a preference tuning method that focuses on both summary-level and corpus-level fairness in MDS. To improve summary-level fairness, we propose to generate preference pairs by perturbing document sets. To improve corpus-level fairness, we propose fairness-aware preference tuning by dynamically adjusting the weights of preference pairs. Our experiments show that FairPO outperforms strong baselines while maintaining the critical qualities of summaries. The code is available at https://github.com/leehaoyuan/coverage_fairnes.