PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

TL;DR

PopResume introduces a population-based resume dataset for causal fairness auditing of AI screening systems, enabling path-specific effect analysis to distinguish permissible from impermissible disparities.

Contribution

The paper presents PopResume, a novel dataset grounded in population statistics that facilitates causal fairness evaluation of LLM/VLM resume screeners, addressing limitations of existing benchmarks.

Findings

PSE-based evaluation uncovers fairness issues hidden by traditional metrics.

Identified five discrimination patterns across models and occupations.

Causal analysis improves understanding of sources of bias in resume screening.

Abstract

We present PopResume, a population-representative resume dataset for causal fairness auditing of LLM- and VLM-based resume screening systems. Unlike existing benchmarks that rely on manually injected demographic information and outcome-level disparities, PopResume is grounded in population statistics and preserves natural attribute relationships, enabling path-specific effect (PSE)-based fairness evaluation. We decompose the effect of a protected attribute on resume scores into two paths: the business necessity path, mediated by job-relevant qualifications, and the redlining path, mediated by demographic proxies. This distinction allows auditors to separate legally permissible from impermissible sources of disparity. Evaluating four LLMs and four VLMs on PopResume's 60.8K resumes across five occupations, we identify five representative discrimination patterns that aggregate metrics fail to capture. Our results demonstrate that PSE-based evaluation reveals fairness issues masked by outcome-level measures, underscoring the need for causally-grounded auditing frameworks in AI-assisted hiring.