Disentangled Action Recognition with Knowledge Bases

TL;DR

This paper introduces DARK, a scalable, knowledge graph-based model for action recognition that generalizes well to unseen verb-noun combinations by disentangling features and leveraging external knowledge bases.

Contribution

DARK is a novel approach that factorizes action features and uses knowledge graphs to improve compositional action recognition, addressing scalability issues of previous methods.

Findings

Achieves state-of-the-art performance on Charades dataset.

Demonstrates improved generalization to unseen verb-noun combinations.

Proposes a new large-scale benchmark based on Epic-kitchen dataset.

Abstract

Action in video usually involves the interaction of human with objects. Action labels are typically composed of various combinations of verbs and nouns, but we may not have training data for all possible combinations. In this paper, we aim to improve the generalization ability of the compositional action recognition model to novel verbs or novel nouns that are unseen during training time, by leveraging the power of knowledge graphs. Previous work utilizes verb-noun compositional action nodes in the knowledge graph, making it inefficient to scale since the number of compositional action nodes grows quadratically with respect to the number of verbs and nouns. To address this issue, we propose our approach: Disentangled Action Recognition with Knowledge-bases (DARK), which leverages the inherent compositionality of actions. DARK trains a factorized model by first extracting disentangled feature representations for verbs and nouns, and then predicting classification weights using relations in external knowledge graphs. The type constraint between verb and noun is extracted from external knowledge bases and finally applied when composing actions. DARK has better scalability in the number of objects and verbs, and achieves state-of-the-art performance on the Charades dataset. We further propose a new benchmark split based on the Epic-kitchen dataset which is an order of magnitude bigger in the numbers of classes and samples, and benchmark various models on this benchmark.