BenCSSmark: Making the Social Sciences Count in LLM Research

TL;DR

This paper advocates for integrating social science datasets into LLM benchmarks to enhance AI evaluation, robustness, and social relevance, introducing the BenCSSmark benchmark composed of social science datasets.

Contribution

It introduces BenCSSmark, a benchmark incorporating social science datasets annotated by social scientists to improve AI evaluation and social scientific inquiry.

Findings

BenCSSmark includes datasets from social sciences.

Integrating social science data can improve AI model robustness.

Benchmark promotes socially relevant AI development.

Abstract

This position paper argues that the under-representation of social science tasks in contemporary LLM benchmarks limits advances in both LLM evaluation and social scientific inquiry. Benchmarks -- standardized tools for assessing computational systems -- are pivotal in the development of artificial intelligence (AI), including large language models (LLMs). Benchmarks do more than measure progress -- they actively structure it, shaping reputations, research agendas, and commercial outcomes. Despite this central role, the social sciences are largely absent from mainstream evaluation frameworks, even though scholars in these fields generate dozens of rigorously annotated, context-sensitive datasets each year. Integrating this work into benchmark design could significantly improve the generalization and robustness of AI models. In turn, models trained on social scientific tasks would likely yield better performance on classic and contemporary tasks in disciplines as diverse as history, sociology, political science or economics. This is all the more pressing as these disciplines are quickly turning to LLMs for assistance. To address this gap, we introduce BenCSSmark, a benchmark composed of datasets annotated by computational social scientists. By integrating social scientific perspectives into benchmarking, BenCSSmark seeks to promote more robust, transparent, and socially relevant AI systems and to foster efficient collaboration.