Grading video interviews with fairness considerations

TL;DR

This paper introduces a methodology for fair and unbiased AI-based assessment of social skills from video interviews across diverse ethnicities and genders, addressing previous issues of bias and inconclusive results.

Contribution

It presents a novel approach including diverse data collection, unbiased rating practices, and two models for predicting social skills, emphasizing fairness and ethical considerations.

Findings

Models show promising accuracy in predicting social skills.

Fairness analysis reveals error disparities across race and gender.

Methodology improves unbiased assessment in video interview scoring.

Abstract

There has been considerable interest in predicting human emotions and traits using facial images and videos. Lately, such work has come under criticism for poor labeling practices, inconclusive prediction results and fairness considerations. We present a careful methodology to automatically derive social skills of candidates based on their video response to interview questions. We, for the first time, include video data from multiple countries encompassing multiple ethnicities. Also, the videos were rated by individuals from multiple racial backgrounds, following several best practices, to achieve a consensus and unbiased measure of social skills. We develop two machine-learning models to predict social skills. The first model employs expert-guidance to use plausibly causal features. The second uses deep learning and depends solely on the empirical correlations present in the data. We compare errors of both these models, study the specificity of the models and make recommendations. We further analyze fairness by studying the errors of models by race and gender. We verify the usefulness of our models by determining how well they predict interview outcomes for candidates. Overall, the study provides strong support for using artificial intelligence for video interview scoring, while taking care of fairness and ethical considerations.