Leveraging In-Context Learning for Political Bias Testing of LLMs

TL;DR

This paper introduces Questionnaire Modeling (QM), a new method using human survey data as in-context examples to improve the stability and reliability of political bias testing in large language models (LLMs).

Contribution

It proposes QM as a novel probing task that enhances bias evaluation stability and enables comparison between instruction-tuned and base LLMs, revealing how tuning and size affect bias.

Findings

QM improves bias evaluation stability

Instruction tuning can alter bias direction

Larger models leverage in-context examples more effectively

Abstract

A growing body of work has been querying LLMs with political questions to evaluate their potential biases. However, this probing method has limited stability, making comparisons between models unreliable. In this paper, we argue that LLMs need more context. We propose a new probing task, Questionnaire Modeling (QM), that uses human survey data as in-context examples. We show that QM improves the stability of question-based bias evaluation, and demonstrate that it may be used to compare instruction-tuned models to their base versions. Experiments with LLMs of various sizes indicate that instruction tuning can indeed change the direction of bias. Furthermore, we observe a trend that larger models are able to leverage in-context examples more effectively, and generally exhibit smaller bias scores in QM. Data and code are publicly available.