Understanding Position Bias Effects on Fairness in Social Multi-Document Summarization

TL;DR

This paper investigates how position bias affects fairness in social multi-document summarization, revealing that input order significantly impacts fairness across different linguistic communities, despite consistent summary quality.

Contribution

It provides the first in-depth analysis of position bias effects on fairness in social multi-document summarization involving diverse linguistic groups.

Findings

Summary quality remains stable regardless of input order.

Fairness varies significantly with the order of dialect groups in input.

Position bias impacts social fairness more than textual quality.

Abstract

Text summarization models have typically focused on optimizing aspects of quality such as fluency, relevance, and coherence, particularly in the context of news articles. However, summarization models are increasingly being used to summarize diverse sources of text, such as social media data, that encompass a wide demographic user base. It is thus crucial to assess not only the quality of the generated summaries, but also the extent to which they can fairly represent the opinions of diverse social groups. Position bias, a long-known issue in news summarization, has received limited attention in the context of social multi-document summarization. We deeply investigate this phenomenon by analyzing the effect of group ordering in input documents when summarizing tweets from three distinct linguistic communities: African-American English, Hispanic-aligned Language, and White-aligned Language. Our empirical analysis shows that although the textual quality of the summaries remains consistent regardless of the input document order, in terms of fairness, the results vary significantly depending on how the dialect groups are presented in the input data. Our results suggest that position bias manifests differently in social multi-document summarization, severely impacting the fairness of summarization models.