Validating Political Position Predictions of Arguments

TL;DR

This paper introduces a dual-scale validation framework for assessing political stance predictions in argumentative discourse, combining pointwise and pairwise human annotations, and creates a large knowledge base for political arguments.

Contribution

It presents a novel validation methodology for subjective continuous attributes, and constructs a structured knowledge base enabling reasoning and retrieval in political domains.

Findings

Moderate agreement between humans and models on pointwise evaluations (Krippendorff's α=0.578)

High alignment between human and model rankings on pairwise validation (α=0.86)

Demonstrates extraction of ordinal structure from subjective language model predictions

Abstract

Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation. We address this challenge through a dual-scale validation framework applied to political stance prediction in argumentative discourse, combining pointwise and pairwise human annotation. Using 22 language models, we construct a large-scale knowledge base of political position predictions for 23,228 arguments drawn from 30 debates that appeared on the UK politicial television programme \textit{Question Time}. Pointwise evaluation shows moderate human-model agreement (Krippendorff's $\alpha=0.578$), reflecting intrinsic subjectivity, while pairwise validation reveals substantially stronger alignment between human- and model-derived rankings ($\alpha=0.86$ for the best model). This work contributes: (i) a practical validation methodology for subjective continuous knowledge that balances scalability with reliability; (ii) a validated structured argumentation knowledge base enabling graph-based reasoning and retrieval-augmented generation in political domains; and (iii) evidence that ordinal structure can be extracted from pointwise language models predictions from inherently subjective real-world discourse, advancing knowledge representation capabilities for domains where traditional symbolic or categorical approaches are insufficient.