Multi-source Multimodal Progressive Domain Adaption for Audio-Visual Deception Detection

TL;DR

This paper introduces a novel multi-source multimodal domain adaptation framework for audio-visual deception detection, effectively reducing domain shift and achieving top-tier performance in a competitive challenge.

Contribution

The paper proposes the MMPDA framework that progressively aligns source and target domains at feature and decision levels across multiple modalities.

Findings

Achieved 60.43% accuracy and 56.99% F1-score in the challenge.

Outperformed the 1st place team by 5.59% on F1-score.

Secured Top-2 placement in the competition.

Abstract

This paper presents the winning approach for the 1st MultiModal Deception Detection (MMDD) Challenge at the 1st Workshop on Subtle Visual Computing (SVC). Aiming at the domain shift issue across source and target domains, we propose a Multi-source Multimodal Progressive Domain Adaptation (MMPDA) framework that transfers the audio-visual knowledge from diverse source domains to the target domain. By gradually aligning source and the target domain at both feature and decision levels, our method bridges domain shifts across diverse multimodal datasets. Extensive experiments demonstrate the effectiveness of our approach securing Top-2 place. Our approach reaches 60.43% on accuracy and 56.99\% on F1-score on competition stage 2, surpassing the 1st place team by 5.59% on F1-score and the 3rd place teams by 6.75% on accuracy. Our code is available at https://github.com/RH-Lin/MMPDA.