Equitable Evaluation via Elicitation

TL;DR

This paper introduces an interactive AI system for skill elicitation that accurately assesses abilities while respecting individual self-presentation styles, reducing bias in evaluations.

Contribution

It presents a novel skill elicitation method that combines human-like interaction with rigorous fairness constraints, trained using synthetic data from large language models.

Findings

Reduces bias caused by self-presentation differences.

Ensures equitable skill evaluation across diverse individuals.

Demonstrates effective deployment in professional settings.

Abstract

Individuals with similar qualifications and skills may vary in their demeanor, or outward manner: some tend toward self-promotion while others are modest to the point of omitting crucial information. Comparing the self-descriptions of equally qualified job-seekers with different self-presentation styles is therefore problematic. We build an interactive AI for skill elicitation that provides accurate determination of skills while simultaneously allowing individuals to speak in their own voice. Such a system can be deployed, for example, when a new user joins a professional networking platform, or when matching employees to needs during a company reorganization. To obtain sufficient training data, we train an LLM to act as synthetic humans. Elicitation mitigates endogenous bias arising from individuals' own self-reports. To address systematic model bias we enforce a mathematically rigorous notion of equitability ensuring that the covariance between self-presentation manner and skill evaluation error is small.