Detecting Linguistic Bias in Government Documents Using Large language Models

TL;DR

This study introduces a new dataset and fine-tuned BERT models to effectively detect linguistic bias in Dutch government documents, highlighting the importance of specialized datasets for fair governance.

Contribution

The paper presents DGDB, a novel bias-annotated dataset for Dutch government texts, and demonstrates the superior performance of fine-tuned BERT models over generative models in bias detection.

Findings

Fine-tuned BERT models outperform generative models in bias detection.

DGDB dataset enables effective bias identification in Dutch government documents.

Models provide explainable predictions to understand bias detection results.

Abstract

This paper addresses the critical need for detecting bias in government documents, an underexplored area with significant implications for governance. Existing methodologies often overlook the unique context and far-reaching impacts of governmental documents, potentially obscuring embedded biases that shape public policy and citizen-government interactions. To bridge this gap, we introduce the Dutch Government Data for Bias Detection (DGDB), a dataset sourced from the Dutch House of Representatives and annotated for bias by experts. We fine-tune several BERT-based models on this dataset and compare their performance with that of generative language models. Additionally, we conduct a comprehensive error analysis that includes explanations of the models' predictions. Our findings demonstrate that fine-tuned models achieve strong performance and significantly outperform generative language models, indicating the effectiveness of DGDB for bias detection. This work underscores the importance of labeled datasets for bias detection in various languages and contributes to more equitable governance practices.