Fine-grained Knowledge Graph-driven Video-Language Learning for Action Recognition

TL;DR

This paper introduces KG-CLIP, a knowledge graph-guided contrastive learning framework that enhances video action recognition by capturing fine-grained semantic relationships between actions and body movements, especially effective with limited data.

Contribution

The paper proposes a novel knowledge graph-driven contrastive learning approach that incorporates multi-grained action concepts into the CLIP model for improved fine-grained video understanding.

Findings

Outperforms baseline methods on Kinetics-TPS dataset.

Excels in action recognition with few sample frames.

Demonstrates strong data efficiency and learning capability.

Abstract

Recent work has explored video action recognition as a video-text matching problem and several effective methods have been proposed based on large-scale pre-trained vision-language models. However, these approaches primarily operate at a coarse-grained level without the detailed and semantic understanding of action concepts by exploiting fine-grained semantic connections between actions and body movements. To address this gap, we propose a contrastive video-language learning framework guided by a knowledge graph, termed KG-CLIP, which incorporates structured information into the CLIP model in the video domain. Specifically, we construct a multi-modal knowledge graph composed of multi-grained concepts by parsing actions based on compositional learning. By implementing a triplet encoder and deviation compensation to adaptively optimize the margin in the entity distance function, our model aims to improve alignment of entities in the knowledge graph to better suit complex relationship learning. This allows for enhanced video action recognition capabilities by accommodating nuanced associations between graph components. We comprehensively evaluate KG-CLIP on Kinetics-TPS, a large-scale action parsing dataset, demonstrating its effectiveness compared to competitive baselines. Especially, our method excels at action recognition with few sample frames or limited training data, which exhibits excellent data utilization and learning capabilities.