Towards Streaming Egocentric Action Anticipation

TL;DR

This paper introduces a streaming evaluation protocol for egocentric action anticipation, emphasizing model runtime, and proposes a lightweight model that outperforms heavier models in this real-time scenario.

Contribution

It proposes a new streaming evaluation scheme considering runtime and introduces a lightweight, effective 3D CNN model optimized with knowledge distillation.

Findings

Smaller models outperform heavier ones in streaming scenarios.

The proposed lightweight model surpasses prior art in real-time settings.

Runtime-aware evaluation changes model rankings compared to offline assessments.

Abstract

Egocentric action anticipation is the task of predicting the future actions a camera wearer will likely perform based on past video observations. While in a real-world system it is fundamental to output such predictions before the action begins, past works have not generally paid attention to model runtime during evaluation. Indeed, current evaluation schemes assume that predictions can be made offline, and hence that computational resources are not limited. In contrast, in this paper, we propose a "streaming" egocentric action anticipation evaluation protocol which explicitly considers model runtime for performance assessment, assuming that predictions will be available only after the current video segment is processed, which depends on the processing time of a method. Following the proposed evaluation scheme, we benchmark different state-of-the-art approaches for egocentric action anticipation on two popular datasets. Our analysis shows that models with a smaller runtime tend to outperform heavier models in the considered streaming scenario, thus changing the rankings generally observed in standard offline evaluations. Based on this observation, we propose a lightweight action anticipation model consisting in a simple feed-forward 3D CNN, which we propose to optimize using knowledge distillation techniques and a custom loss. The results show that the proposed approach outperforms prior art in the streaming scenario, also in combination with other lightweight models.