Seeing Through Fog: Towards Fog-Invariant Action Recognition

TL;DR

This paper introduces FogAct, a new foggy action recognition dataset, and FogNet, a model that learns fog-invariant features to improve action recognition in foggy conditions.

Contribution

The paper presents the first foggy action recognition dataset and a novel two-stream CLIP model that enhances recognition robustness under foggy conditions.

Findings

FogNet achieves competitive performance on FogAct and other datasets.

The dataset includes nearly 10,000 video clips across 55 actions.

FogNet effectively captures shared cues between clean and foggy videos.

Abstract

Foggy conditions are commonly encountered in real-world applications; however, existing action recognition approaches typically assume favorable weather and high-quality video inputs. On foggy days, unpredictable visibility degradation and reduced contrast obstruct the extraction of semantic cues, posing significant challenges for current action recognition methods. In this paper, we mitigate the issues faced in action recognition under foggy conditions by employing two strategies. First, we present FogAct, the first benchmark dataset for foggy action recognition, consisting of paired clean and foggy videos captured with a stereo camera system. The dataset spans 10 scenes and 55 action categories, comprising nearly 10,000 video clips. Second, we propose FogNet, a two-stream CLIP model that discovers fog-invariant semantic information hidden behind the degraded videos. FogNet learns robust representations of foggy videos with guidance from clean videos, effectively capturing shared structural and motion cues between clean and foggy videos. Extensive experiments on FogAct and three other popular datasets demonstrate that our method achieves competitive performance compared with state-of-the-art (SOTA) approaches. Our FogAct and FogNet are given in our project page.