View-aware Cross-modal Distillation for Multi-view Action Recognition

TL;DR

This paper introduces ViCoKD, a novel framework for multi-view action recognition that effectively distills knowledge from fully supervised multi-modal teachers to students with limited modalities, addressing view misalignment and partial observations in real-world scenarios.

Contribution

It proposes a view-aware cross-modal distillation method with a cross-modal adapter and view consistency module, improving recognition accuracy under limited and partial view conditions.

Findings

Outperforms existing distillation methods on MultiSensor-Home dataset.

Achieves significant accuracy gains with limited modalities.

Surpasses teacher performance in constrained settings.

Abstract

The widespread use of multi-sensor systems has increased research in multi-view action recognition. While existing approaches in multi-view setups with fully overlapping sensors benefit from consistent view coverage, partially overlapping settings where actions are visible in only a subset of views remain underexplored. This challenge becomes more severe in real-world scenarios, as many systems provide only limited input modalities and rely on sequence-level annotations instead of dense frame-level labels. In this study, we propose View-aware Cross-modal Knowledge Distillation (ViCoKD), a framework that distills knowledge from a fully supervised multi-modal teacher to a modality- and annotation-limited student. ViCoKD employs a cross-modal adapter with cross-modal attention, allowing the student to exploit multi-modal correlations while operating with incomplete modalities. Moreover, we propose a View-aware Consistency module to address view misalignment, where the same action may appear differently or only partially across viewpoints. It enforces prediction alignment when the action is co-visible across views, guided by human-detection masks and confidence-weighted Jensen-Shannon divergence between their predicted class distributions. Experiments on the real-world MultiSensor-Home dataset show that ViCoKD consistently outperforms competitive distillation methods across multiple backbones and environments, delivering significant gains and surpassing the teacher model under limited conditions.