Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models

TL;DR

This paper compares the performance and cost of building, borrowing, or fine-tuning NLP models for political science tasks, providing a practical decision framework based on empirical analysis of conflict event classification.

Contribution

It offers an empirical comparison of domain-specific and fine-tuned general NLP models, and develops a decision framework for political scientists to choose appropriate NLP approaches.

Findings

Fine-tuned ModernBERT achieves 75.46% accuracy, slightly below domain-specific ConfliBERT's 79.34%.

Model performance differences are minimal for high-frequency event types.

Rare event categories show larger performance gaps, influencing model choice.

Abstract

Political scientists increasingly face a consequential choice when adopting natural language processing tools: build a domain-specific model from scratch, borrow and adapt an existing one, or simply fine-tune a general-purpose model on task data? Each approach occupies a different point on the spectrum of performance, cost, and required expertise, yet the discipline has offered little empirical guidance on how to navigate this trade-off. This paper provides such guidance. Using conflict event classification as a test case, I fine-tune ModernBERT on the Global Terrorism Database (GTD) to create Confli-mBERT and systematically compare it against ConfliBERT, a domain-specific pretrained model that represents the current gold standard. Confli-mBERT achieves 75.46% accuracy compared to ConfliBERT's 79.34%. Critically, the four-percentage-point gap is not uniform: on high-frequency attack types such as Bombing/Explosion (F1 = 0.95 vs. 0.96) and Kidnapping (F1 = 0.92 vs. 0.91), the models are nearly indistinguishable. Performance differences concentrate in rare event categories comprising fewer than 2% of all incidents. I use these findings to develop a practical decision framework for political scientists considering any NLP-assisted research task: when does the research question demand a specialized model, and when does an accessible fine-tuned alternative suffice? The answer, I argue, depends not on which model is "better" in the abstract, but on the specific intersection of class prevalence, error tolerance, and available resources. The model, training code, and data are publicly available on Hugging Face.