Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models

TL;DR

This paper presents PlausiVL, a large video-language model that anticipates plausible future action sequences by incorporating novel loss functions and logical constraints, improving diversity and realism in predictions.

Contribution

The work introduces a new plausibility-aware training framework for action anticipation using large video-language models, with novel loss functions and logical constraints.

Findings

Improved accuracy on Ego4D and EPIC-Kitchens-100 datasets.

Enhanced diversity and plausibility in generated action sequences.

Effective differentiation between plausible and implausible actions.

Abstract

We introduce PlausiVL, a large video-language model for anticipating action sequences that are plausible in the real-world. While significant efforts have been made towards anticipating future actions, prior approaches do not take into account the aspect of plausibility in an action sequence. To address this limitation, we explore the generative capability of a large video-language model in our work and further, develop the understanding of plausibility in an action sequence by introducing two objective functions, a counterfactual-based plausible action sequence learning loss and a long-horizon action repetition loss. We utilize temporal logical constraints as well as verb-noun action pair logical constraints to create implausible/counterfactual action sequences and use them to train the model with plausible action sequence learning loss. This loss helps the model to differentiate between plausible and not plausible action sequences and also helps the model to learn implicit temporal cues crucial for the task of action anticipation. The long-horizon action repetition loss puts a higher penalty on the actions that are more prone to repetition over a longer temporal window. With this penalization, the model is able to generate diverse, plausible action sequences. We evaluate our approach on two large-scale datasets, Ego4D and EPIC-Kitchens-100, and show improvements on the task of action anticipation.