ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization

TL;DR

ModSelect is an unsupervised method that automatically selects the most robust modalities for synthetic-to-real domain generalization in activity recognition, improving performance without requiring labels.

Contribution

It introduces a systematic, label-free modality selection approach based on correlation and domain discrepancy, enhancing cross-domain activity recognition.

Findings

ModSelect effectively selects positive modalities for domain generalization.

It improves performance on synthetic-to-real activity recognition benchmarks.

The method narrows the domain gap without using ground-truth labels.

Abstract

Modality selection is an important step when designing multimodal systems, especially in the case of cross-domain activity recognition as certain modalities are more robust to domain shift than others. However, selecting only the modalities which have a positive contribution requires a systematic approach. We tackle this problem by proposing an unsupervised modality selection method (ModSelect), which does not require any ground-truth labels. We determine the correlation between the predictions of multiple unimodal classifiers and the domain discrepancy between their embeddings. Then, we systematically compute modality selection thresholds, which select only modalities with a high correlation and low domain discrepancy. We show in our experiments that our method ModSelect chooses only modalities with positive contributions and consistently improves the performance on a Synthetic-to-Real domain adaptation benchmark, narrowing the domain gap.