# Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics

## TL;DR

This paper proposes a new way to evaluate generalist medical AI using psychometric methods to better understand their strengths and limitations.

## Contribution

The paper introduces a construct-oriented evaluation methodology for GMAI using modern psychometric techniques.

## Key findings

- Current benchmarks for GMAI lack explanatory and predictive power.
- Construct-oriented evaluation can better assess professional skills, knowledge, and behaviors.
- Human oversight is crucial for future GMAI adoption.

## Abstract

Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption.