A Probabilistic Jump-Diffusion Framework for Open-World Egocentric Activity Recognition

TL;DR

This paper introduces ProbRes, a probabilistic jump-diffusion framework that enhances open-world egocentric activity recognition by efficiently exploring large, partially observed activity spaces using structured priors and stochastic search, achieving state-of-the-art results.

Contribution

The paper presents a novel probabilistic jump-diffusion approach, ProbRes, integrating structured priors and adaptive search to improve open-world activity recognition in egocentric videos.

Findings

Achieves state-of-the-art performance on multiple benchmark datasets.

Demonstrates robustness across varying openness levels (L0--L3).

Provides a taxonomy for challenges in open-world egocentric activity recognition.

Abstract

Open-world egocentric activity recognition poses a fundamental challenge due to its unconstrained nature, requiring models to infer unseen activities from an expansive, partially observed search space. We introduce ProbRes, a Probabilistic Residual search framework based on jump-diffusion that efficiently navigates this space by balancing prior-guided exploration with likelihood-driven exploitation. Our approach integrates structured commonsense priors to construct a semantically coherent search space, adaptively refines predictions using Vision-Language Models (VLMs) and employs a stochastic search mechanism to locate high-likelihood activity labels while minimizing exhaustive enumeration efficiently. We systematically evaluate ProbRes across multiple openness levels (L0--L3), demonstrating its adaptability to increasing search space complexity. In addition to achieving state-of-the-art performance on benchmark datasets (GTEA Gaze, GTEA Gaze+, EPIC-Kitchens, and Charades-Ego), we establish a clear taxonomy for open-world recognition, delineating the challenges and methodological advancements necessary for egocentric activity understanding.