Day2Dark: Pseudo-Supervised Activity Recognition beyond Silent Daylight

TL;DR

This paper introduces a pseudo-supervised learning approach combined with adaptive audio-visual fusion to improve activity recognition in dark environments, addressing data scarcity and illumination challenges.

Contribution

It proposes a novel darkness-adaptive audio-visual recognizer and a pseudo-supervised learning scheme to enhance activity recognition in low-light conditions.

Findings

Outperforms image enhancement and domain adaptation methods

Improves robustness to occlusions and darkness

Effective across multiple datasets

Abstract

This paper strives to recognize activities in the dark, as well as in the day. We first establish that state-of-the-art activity recognizers are effective during the day, but not trustworthy in the dark. The main causes are the limited availability of labeled dark videos to learn from, as well as the distribution shift towards the lower color contrast at test-time. To compensate for the lack of labeled dark videos, we introduce a pseudo-supervised learning scheme, which utilizes easy to obtain unlabeled and task-irrelevant dark videos to improve an activity recognizer in low light. As the lower color contrast results in visual information loss, we further propose to incorporate the complementary activity information within audio, which is invariant to illumination. Since the usefulness of audio and visual features differs depending on the amount of illumination, we introduce our `darkness-adaptive' audio-visual recognizer. Experiments on EPIC-Kitchens, Kinetics-Sound, and Charades demonstrate our proposals are superior to image enhancement, domain adaptation and alternative audio-visual fusion methods, and can even improve robustness to local darkness caused by occlusions. Project page: https://xiaobai1217.github.io/Day2Dark/