Behind the Screens: Uncovering Bias in AI-Driven Video Interview Assessments Using Counterfactuals

TL;DR

This paper presents a counterfactual framework using GANs to evaluate and quantify bias in AI-driven personality assessments across multiple modalities, addressing ethical concerns in hiring AI systems.

Contribution

It introduces a novel counterfactual-based method for fairness analysis in multimodal AI assessments, applicable even when models are black boxes.

Findings

Significant demographic disparities in AI assessments.

Effective counterfactual generation validated by protected attribute classifier.

Framework applicable to multimodal, black-box AI systems.

Abstract

AI-enhanced personality assessments are increasingly shaping hiring decisions, using affective computing to predict traits from the Big Five (OCEAN) model. However, integrating AI into these assessments raises ethical concerns, especially around bias amplification rooted in training data. These biases can lead to discriminatory outcomes based on protected attributes like gender, ethnicity, and age. To address this, we introduce a counterfactual-based framework to systematically evaluate and quantify bias in AI-driven personality assessments. Our approach employs generative adversarial networks (GANs) to generate counterfactual representations of job applicants by altering protected attributes, enabling fairness analysis without access to the underlying model. Unlike traditional bias assessments that focus on unimodal or static data, our method supports multimodal evaluation-spanning visual, audio, and textual features. This comprehensive approach is particularly important in high-stakes applications like hiring, where third-party vendors often provide AI systems as black boxes. Applied to a state-of-the-art personality prediction model, our method reveals significant disparities across demographic groups. We also validate our framework using a protected attribute classifier to confirm the effectiveness of our counterfactual generation. This work provides a scalable tool for fairness auditing of commercial AI hiring platforms, especially in black-box settings where training data and model internals are inaccessible. Our results highlight the importance of counterfactual approaches in improving ethical transparency in affective computing.