PALM: Predicting Actions through Language Models

TL;DR

PALM is a novel approach that combines action recognition, vision-language models, and large language models to improve long-term human activity prediction in egocentric videos, outperforming existing methods on multiple benchmarks.

Contribution

It introduces a new method that leverages context-aware prompting and in-context learning with LLMs for long-term action anticipation, addressing data scarcity issues.

Findings

Outperforms state-of-the-art in Ego4D benchmark

Demonstrates strong generalization across multiple datasets

Effectively utilizes context for long-term prediction

Abstract

Understanding human activity is a crucial yet intricate task in egocentric vision, a field that focuses on capturing visual perspectives from the camera wearer's viewpoint. Traditional methods heavily rely on representation learning that is trained on a large amount of video data. However, a major challenge arises from the difficulty of obtaining effective video representation. This difficulty stems from the complex and variable nature of human activities, which contrasts with the limited availability of data. In this study, we introduce PALM, an approach that tackles the task of long-term action anticipation, which aims to forecast forthcoming sequences of actions over an extended period. Our method PALM incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. By leveraging the context provided by these past events, we devise a prompting strategy for action anticipation using large language models (LLMs). Moreover, we implement maximal marginal relevance for example selection to facilitate in-context learning of the LLMs. Our experimental results demonstrate that PALM surpasses the state-of-the-art methods in the task of long-term action anticipation on the Ego4D benchmark. We further validate PALM on two additional benchmarks, affirming its capacity for generalization across intricate activities with different sets of taxonomies.