Listening to the Unspoken: Exploring "365" Aspects of Multimodal Interview Performance Assessment

TL;DR

This paper introduces a comprehensive multimodal framework for interview performance assessment that integrates video, audio, and text data across multiple responses and evaluation dimensions, achieving state-of-the-art results in the AVI Challenge 2025.

Contribution

It presents a novel multimodal assessment framework with modality-specific feature extraction, shared compression, and ensemble learning, advancing automated interview evaluation methods.

Findings

Achieved a multi-dimensional average MSE of 0.1824.

Secured first place in the AVI Challenge 2025.

Demonstrated robustness and effectiveness in multimodal assessment.

Abstract

Interview performance assessment is essential for determining candidates' suitability for professional positions. To ensure holistic and fair evaluations, we propose a novel and comprehensive framework that explores ``365'' aspects of interview performance by integrating \textit{three} modalities (video, audio, and text), \textit{six} responses per candidate, and \textit{five} key evaluation dimensions. The framework employs modality-specific feature extractors to encode heterogeneous data streams and subsequently fused via a Shared Compression Multilayer Perceptron. This module compresses multimodal embeddings into a unified latent space, facilitating efficient feature interaction. To enhance prediction robustness, we incorporate a two-level ensemble learning strategy: (1) independent regression heads predict scores for each response, and (2) predictions are aggregated across responses using a mean-pooling mechanism to produce final scores for the five target dimensions. By listening to the unspoken, our approach captures both explicit and implicit cues from multimodal data, enabling comprehensive and unbiased assessments. Achieving a multi-dimensional average MSE of 0.1824, our framework secured first place in the AVI Challenge 2025, demonstrating its effectiveness and robustness in advancing automated and multimodal interview performance assessment. The full implementation is available at https://github.com/MSA-LMC/365Aspects.