Bringing Online Egocentric Action Recognition into the wild

TL;DR

This paper introduces a new realistic setting for egocentric action recognition in the wild, emphasizing deployment challenges like portability, real-time inference, and robustness, and proposes a model-agnostic technique for edge deployment on tiny devices.

Contribution

It defines a novel egocentric action recognition setting for real-world applications and presents a model-agnostic method enabling deployment on low-power edge devices.

Findings

Achieved real-time inference at 50 fps on Jetson Nano

Low energy consumption of 2.4W during operation

Demonstrated feasibility of deploying egocentric models on edge devices

Abstract

To enable a safe and effective human-robot cooperation, it is crucial to develop models for the identification of human activities. Egocentric vision seems to be a viable solution to solve this problem, and therefore many works provide deep learning solutions to infer human actions from first person videos. However, although very promising, most of these do not consider the major challenges that comes with a realistic deployment, such as the portability of the model, the need for real-time inference, and the robustness with respect to the novel domains (i.e., new spaces, users, tasks). With this paper, we set the boundaries that egocentric vision models should consider for realistic applications, defining a novel setting of egocentric action recognition in the wild, which encourages researchers to develop novel, applications-aware solutions. We also present a new model-agnostic technique that enables the rapid repurposing of existing architectures in this new context, demonstrating the feasibility to deploy a model on a tiny device (Jetson Nano) and to perform the task directly on the edge with very low energy consumption (2.4W on average at 50 fps). The code is publicly available at: https://github.com/EgocentricVision/EgoWild.