Assessing Generalization for Subpopulation Representative Modeling via In-Context Learning

TL;DR

This paper evaluates how well Large Language Model-based Subpopulation Representative Models generalize across demographics using in-context learning, revealing uneven benefits and highlighting the need for diverse subpopulation benchmarks.

Contribution

It introduces an evaluation of SRMs' generalization capabilities across demographics with in-context learning, exposing inequities and emphasizing the importance of diverse benchmarks.

Findings

In-context learning improves overall performance but varies across demographics.

Some demographic groups experience performance decline with in-context learning.

Diverse subpopulation benchmarks are needed to better assess generalization.

Abstract

This study evaluates the ability of Large Language Model (LLM)-based Subpopulation Representative Models (SRMs) to generalize from empirical data, utilizing in-context learning with data from the 2016 and 2020 American National Election Studies. We explore generalization across response variables and demographic subgroups. While conditioning with empirical data improves performance on the whole, the benefit of in-context learning varies considerably across demographics, sometimes hurting performance for one demographic while helping performance for others. The inequitable benefits of in-context learning for SRM present a challenge for practitioners implementing SRMs, and for decision-makers who might come to rely on them. Our work highlights a need for fine-grained benchmarks captured from diverse subpopulations that test not only fidelity but generalization.