Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy

TL;DR

This paper introduces RefDense, a new approach to dense action detection that decomposes the task into unambiguous sub-concepts, employs separate sub-networks, and uses language-guided contrastive learning to improve detection of co-occurring actions.

Contribution

The paper proposes a novel paradigm shift by decomposing dense action detection into sub-tasks with dedicated networks and introduces a language-guided contrastive loss for better learning of action relationships.

Findings

Achieved 3.8% and 1.7% improvements on Charades and MultiTHUMOS datasets.

Demonstrated the effectiveness of sub-network decomposition in dense action detection.

Showed that explicit supervision on co-occurring concepts enhances performance.

Abstract

Dense action detection involves detecting multiple co-occurring actions while action classes are often ambiguous and represent overlapping concepts. We argue that handling the dual challenge of temporal and class overlaps is too complex to effectively be tackled by a single network. To address this, we propose to decompose the task of detecting dense ambiguous actions into detecting dense, unambiguous sub-concepts that form the action classes (i.e., action entities and action motions), and assigning these sub-tasks to distinct sub-networks. By isolating these unambiguous concepts, the sub-networks can focus exclusively on resolving a single challenge, dense temporal overlaps. Furthermore, simultaneous actions in a video often exhibit interrelationships, and exploiting these relationships can improve the method performance. However, current dense action detection networks fail to effectively learn these relationships due to their reliance on binary cross-entropy optimization, which treats each class independently. To address this limitation, we propose providing explicit supervision on co-occurring concepts during network optimization through a novel language-guided contrastive learning loss. Our extensive experiments demonstrate the superiority of our approach over state-of-the-art methods, achieving substantial improvements of 3.8% and 1.7% on average across all metrics on the challenging benchmark datasets, Charades and MultiTHUMOS.