Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb

TL;DR

This paper investigates how different multimodal fusion techniques affect fairness and accuracy in AI-based recruitment systems, highlighting early-fusion's advantages over late-fusion in handling demographic biases.

Contribution

It provides a comparative analysis of early and late fusion methods in multimodal AI recruitment, emphasizing early-fusion's superior fairness and accuracy.

Findings

Early-fusion achieves the lowest MAEs and closely matches ground truth.

Late-fusion results in higher MAEs and more generalized scores.

Early-fusion demonstrates potential for fairer, more accurate recruitment applications.

Abstract

Despite the large body of work on fairness-aware learning for individual modalities like tabular data, images, and text, less work has been done on multimodal data, which fuses various modalities for a comprehensive analysis. In this work, we investigate the fairness and bias implications of multimodal fusion techniques in the context of multimodal AI-based recruitment systems using the FairCVdb dataset. Our results show that early-fusion closely matches the ground truth for both demographics, achieving the lowest MAEs by integrating each modality's unique characteristics. In contrast, late-fusion leads to highly generalized mean scores and higher MAEs. Our findings emphasise the significant potential of early-fusion for accurate and fair applications, even in the presence of demographic biases, compared to late-fusion. Future research could explore alternative fusion strategies and incorporate modality-related fairness constraints to improve fairness. For code and additional insights, visit: https://github.com/Swati17293/Multimodal-AI-Based-Recruitment-FairCVdb