Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback

TL;DR

This paper introduces a multimodal framework that analyzes interviewee behavior through facial expressions, speech, and prosody to provide automated feedback, aiming to improve interview performance assessment.

Contribution

It presents a novel multimodal analytical system that integrates video, audio, and transcript data for automated behavioral assessment and feedback in interviews.

Findings

Achieved promising classification accuracy in behavioral cue detection

Demonstrated effectiveness of multimodal data integration

Provided actionable feedback for interviewees

Abstract

Behavioral cues play a significant part in human communication and cognitive perception. In most professional domains, employee recruitment policies are framed such that both professional skills and personality traits are adequately assessed. Hiring interviews are structured to evaluate expansively a potential employee's suitability for the position - their professional qualifications, interpersonal skills, ability to perform in critical and stressful situations, in the presence of time and resource constraints, etc. Therefore, candidates need to be aware of their positive and negative attributes and be mindful of behavioral cues that might have adverse effects on their success. We propose a multimodal analytical framework that analyzes the candidate in an interview scenario and provides feedback for predefined labels such as engagement, speaking rate, eye contact, etc. We perform a comprehensive analysis that includes the interviewee's facial expressions, speech, and prosodic information, using the video, audio, and text transcripts obtained from the recorded interview. We use these multimodal data sources to construct a composite representation, which is used for training machine learning classifiers to predict the class labels. Such analysis is then used to provide constructive feedback to the interviewee for their behavioral cues and body language. Experimental validation showed that the proposed methodology achieved promising results.