To Trust, or Not to Trust? A Study of Human Bias in Automated Video Interview Assessments

TL;DR

This paper investigates potential human biases in annotating video-based interview assessments and examines how these biases may influence machine learning models trained on such data.

Contribution

It provides empirical evidence of visual biases in human ratings used for training automated interview assessment systems.

Findings

Biases in human annotations are present, especially visual biases.

Such biases can affect the training and performance of machine learning models.

Preliminary evidence suggests the need for bias mitigation in annotation processes.

Abstract

Supervised systems require human labels for training. But, are humans themselves always impartial during the annotation process? We examine this question in the context of automated assessment of human behavioral tasks. Specifically, we investigate whether human ratings themselves can be trusted at their face value when scoring video-based structured interviews, and whether such ratings can impact machine learning models that use them as training data. We present preliminary empirical evidence that indicates there might be biases in such annotations, most of which are visual in nature.