Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas

TL;DR

This paper proposes a hierarchical prompting method and a new taxonomy for propaganda detection in social media, demonstrating the importance of fine-tuning and model choice for improved classification across schemas.

Contribution

It introduces a novel intent-focused taxonomy and hierarchical prompting strategy, showing their effectiveness in propaganda classification across different models and schemas.

Findings

Fine-tuning transforms weak zero-shot baselines into competitive systems.

Qwen models outperform other evaluated models overall.

Hierarchical prompting benefits ambiguous, low-agreement schemas after fine-tuning.

Abstract

Propaganda detection in social media is challenging due to noisy, short texts and low annotation agreements. We introduce a new intent-focused taxonomy of propaganda techniques and compare it against an established, higher-agreement schema. Along three dimensions (model portfolio, schema effects, and prompting strategy) we evaluate the taxonomies as a classification task with the help of four language models (GPT-4.1-nano, Phi-4 14B, Qwen2.5-14B, Qwen3-14B). Our results show that fine-tuning is essential, since it transforms weak zero-shot baselines into competitive systems and reveals methodological differences that are hidden using base models. Across schemas, the Qwen models achieve the strongest overall performance, and Phi-4 14B consistently outperforms GPT-4.1-nano. Our hierarchical prompting method (HiPP), which predicts fine-grained techniques before aggregating them, is especially beneficial after fine-tuning and on the more ambiguous, low-agreement taxonomy, while remaining competitive on the simpler schema. The HQP dataset, annotated with the new intent-based labels, provides a richer lens on propaganda's strategic goals and a challenging benchmark for future work on robust, real-world detection.