ALGO: Object-Grounded Visual Commonsense Reasoning for Open-World Egocentric Action Recognition

TL;DR

ALGO is a neuro-symbolic framework that enhances open-world egocentric activity recognition by grounding objects and inferring actions using knowledge bases and evidence-based reasoning.

Contribution

It introduces a novel neuro-symbolic prompting method and a symbolic pattern theory framework for activity inference in egocentric videos with limited supervision.

Findings

Outperforms existing methods on EPIC-Kitchens, GTEA Gaze, and GTEA Gaze Plus datasets.

Effectively grounds objects and actions using knowledge-based reasoning.

Demonstrates robustness in open-world, zero-shot scenarios.

Abstract

Learning to infer labels in an open world, i.e., in an environment where the target "labels" are unknown, is an important characteristic for achieving autonomy. Foundation models pre-trained on enormous amounts of data have shown remarkable generalization skills through prompting, particularly in zero-shot inference. However, their performance is restricted to the correctness of the target label's search space. In an open world, this target search space can be unknown or exceptionally large, which severely restricts the performance of such models. To tackle this challenging problem, we propose a neuro-symbolic framework called ALGO - Action Learning with Grounded Object recognition that uses symbolic knowledge stored in large-scale knowledge bases to infer activities in egocentric videos with limited supervision using two steps. First, we propose a neuro-symbolic prompting approach that uses object-centric vision-language models as a noisy oracle to ground objects in the video through evidence-based reasoning. Second, driven by prior commonsense knowledge, we discover plausible activities through an energy-based symbolic pattern theory framework and learn to ground knowledge-based action (verb) concepts in the video. Extensive experiments on four publicly available datasets (EPIC-Kitchens, GTEA Gaze, GTEA Gaze Plus) demonstrate its performance on open-world activity inference.