A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems

TL;DR

This paper introduces a multi-agent framework that reduces dialect biases in privacy policy question-answering systems by translating dialectal queries into standard English and refining predictions, improving accuracy without retraining.

Contribution

The proposed multi-agent approach effectively mitigates dialect biases in privacy policy QA systems without requiring retraining or dialect-specific fine-tuning.

Findings

Significant accuracy improvements on PrivacyQA and PolicyQA datasets.

Outperforms or matches few-shot baselines without additional training data.

Demonstrates the effectiveness of agent collaboration in reducing dialect bias.

Abstract

Privacy policies inform users about data collection and usage, yet their complexity limits accessibility for diverse populations. Existing Privacy Policy Question Answering (QA) systems exhibit performance disparities across English dialects, disadvantaging speakers of non-standard varieties. We propose a novel multi-agent framework inspired by human-centered design principles to mitigate dialectal biases. Our approach integrates a Dialect Agent, which translates queries into Standard American English (SAE) while preserving dialectal intent, and a Privacy Policy Agent, which refines predictions using domain expertise. Unlike prior approaches, our method does not require retraining or dialect-specific fine-tuning, making it broadly applicable across models and domains. Evaluated on PrivacyQA and PolicyQA, our framework improves GPT-4o-mini's zero-shot accuracy from 0.394 to 0.601 on PrivacyQA and from 0.352 to 0.464 on PolicyQA, surpassing or matching few-shot baselines without additional training data. These results highlight the effectiveness of structured agent collaboration in mitigating dialect biases and underscore the importance of designing NLP systems that account for linguistic diversity to ensure equitable access to privacy information.