Investigating LLMs as Voting Assistants via Contextual Augmentation: A Case Study on the European Parliament Elections 2024

TL;DR

This study evaluates large language models as voting advice tools for the 2024 European Parliament elections, exploring input augmentation techniques to improve accuracy in predicting political stances.

Contribution

It introduces a novel evaluation of LLMs as VAAs in a real-world electoral context and assesses input augmentation methods like RAG and self-reflection to enhance performance.

Findings

MIXTRAL achieves 82% accuracy overall.

Augmenting input with expert-curated info improves accuracy by ~9%.

Performance varies significantly across political groups.

Abstract

In light of the recent 2024 European Parliament elections, we are investigating if LLMs can be used as Voting Advice Applications (VAAs). We audit MISTRAL and MIXTRAL models and evaluate their accuracy in predicting the stance of political parties based on the latest "EU and I" voting assistance questionnaire. Furthermore, we explore alternatives to improve models' performance by augmenting the input context via Retrieval-Augmented Generation (RAG) relying on web search, and Self-Reflection using staged conversations that aim to re-collect relevant content from the model's internal memory. We find that MIXTRAL is highly accurate with an 82% accuracy on average with a significant performance disparity across different political groups (50-95%). Augmenting the input context with expert-curated information can lead to a significant boost of approx. 9%, which remains an open challenge for automated RAG approaches, even considering curated content.