Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift

TL;DR

This paper advocates for human-centered evaluation methods to measure harmful capability uplift in AI safety, emphasizing systematic measurement and practical implementation for stakeholders.

Contribution

It introduces harmful capability uplift as a key safety metric, grounded in social science, with methodological guidance for systematic assessment.

Findings

Proposes a human-centered framework for measuring harmful capability uplift.

Provides concrete methodological guidance for systematic evaluation.

Outlines actionable steps for integrating harmful capability uplift assessment into practice.

Abstract

Current frontier AI safety evaluations emphasize static benchmarks, third-party annotations, and red-teaming. In this position paper, we argue that AI safety research should focus on human-centered evaluations that measure harmful capability uplift: the marginal increase in a user's ability to cause harm with a frontier model beyond what conventional tools already enable. We frame harmful capability uplift as a core AI safety metric, ground it in prior social science research, and provide concrete methodological guidance for systematic measurement. We conclude with actionable steps for developers, researchers, funders, and regulators to make harmful capability uplift evaluation a standard practice.