Bidirectional Action Sequence Learning for Long-term Action Anticipation with Large Language Models

TL;DR

This paper introduces BiAnt, a bidirectional action sequence learning method using large language models, significantly improving long-term action anticipation by capturing semantic sub-actions more effectively.

Contribution

It presents a novel bidirectional learning approach with large language models for enhanced long-term action anticipation in videos.

Findings

BiAnt outperforms baseline methods in edit distance on Ego4D dataset.

Bidirectional prediction captures semantic sub-actions more effectively.

Large language models improve long-term action anticipation accuracy.

Abstract

Video-based long-term action anticipation is crucial for early risk detection in areas such as automated driving and robotics. Conventional approaches extract features from past actions using encoders and predict future events with decoders, which limits performance due to their unidirectional nature. These methods struggle to capture semantically distinct sub-actions within a scene. The proposed method, BiAnt, addresses this limitation by combining forward prediction with backward prediction using a large language model. Experimental results on Ego4D demonstrate that BiAnt improves performance in terms of edit distance compared to baseline methods.