A Multi-Dimensional Audit of Politically Aligned Large Language Models

TL;DR

This paper introduces a multi-dimensional framework to audit politically aligned large language models across effectiveness, fairness, truthfulness, and persuasiveness, revealing trade-offs and deficiencies in current models.

Contribution

It proposes a novel, quantitative auditing framework inspired by Habermas' theory to evaluate political alignment in LLMs across multiple dimensions.

Findings

Larger models are more effective and truthful but less fair and more biased.

Fine-tuned models show lower bias but reduced reasoning performance and more hallucinations.

All models exhibit deficiencies in at least one of the four evaluated metrics.

Abstract

As the application of Large Language Models (LLMs) spreads across various industries, there are increasing concerns about the potential for their misuse, especially in sensitive areas such as political discourse. Deliberately aligning LLMs with specific political ideologies, through prompt engineering or fine-tuning techniques, can be advantageous in use cases such as political campaigns, but requires careful consideration due to heightened risks of performance degradation, misinformation, or increased biased behavior. In this work, we propose a multi-dimensional framework inspired by Habermas' Theory of Communicative Action to audit politically aligned language models across four dimensions: effectiveness, fairness, truthfulness, and persuasiveness using automated, quantitative metrics. Applying this to nine popular LLMs aligned via fine-tuning or role-playing revealed consistent trade-offs: while larger models tend to be more effective at role-playing political ideologies and truthful in their responses, they were also less fair, exhibiting higher levels of bias in the form of angry and toxic language towards people of different ideologies. Fine-tuned models exhibited lower bias and more effective alignment than the corresponding role-playing models, but also saw a decline in performance reasoning tasks and an increase in hallucinations. Overall, all of the models tested exhibited some deficiency in at least one of the four metrics, highlighting the need for more balanced and robust alignment strategies. Ultimately, this work aims to ensure politically-aligned LLMs generate legitimate, harmless arguments, offering a framework to evaluate the responsible political alignment of these models.