Zero-shot Compositional Action Recognition with Neural Logic Constraints

TL;DR

This paper introduces LogicCAR, a neural framework that incorporates symbolic logic constraints to improve zero-shot compositional action recognition by modeling structure and hierarchy, leading to better generalization.

Contribution

It proposes a novel logic-driven approach that embeds compositional and hierarchical constraints into neural networks for zero-shot action recognition.

Findings

Outperforms baseline methods on Sth-com dataset

Effectively models compositional and hierarchical structures

Enhances reasoning capacity in zero-shot scenarios

Abstract

Zero-shot compositional action recognition (ZS-CAR) aims to identify unseen verb-object compositions in the videos by exploiting the learned knowledge of verb and object primitives during training. Despite compositional learning's progress in ZS-CAR, two critical challenges persist: 1) Missing compositional structure constraint, leading to spurious correlations between primitives; 2) Neglecting semantic hierarchy constraint, leading to semantic ambiguity and impairing the training process. In this paper, we argue that human-like symbolic reasoning offers a principled solution to these challenges by explicitly modeling compositional and hierarchical structured abstraction. To this end, we propose a logic-driven ZS-CAR framework LogicCAR that integrates dual symbolic constraints: Explicit Compositional Logic and Hierarchical Primitive Logic. Specifically, the former models the restrictions within the compositions, enhancing the compositional reasoning ability of our model. The latter investigates the semantical dependencies among different primitives, empowering the models with fine-to-coarse reasoning capacity. By formalizing these constraints in first-order logic and embedding them into neural network architectures, LogicCAR systematically bridges the gap between symbolic abstraction and existing models. Extensive experiments on the Sth-com dataset demonstrate that our LogicCAR outperforms existing baseline methods, proving the effectiveness of our logic-driven constraints.