The Object at Hand: Automated Editing for Mixed Reality Video Guidance from Hand-Object Interactions

TL;DR

This paper presents an automated method using egocentric vision, CNNs, and FSMs to decompose real-world hand activities into steps for video guidance in Mixed Reality, achieving high precision in segmenting hand-object interactions.

Contribution

It introduces a novel approach combining HOI detection, object similarity, and FSMs to automatically edit and segment videos into steps, advancing real-time MR guidance.

Findings

High precision in segmenting hand-object interaction videos

Effective decomposition of real-world tasks into key steps

Validated on GTEA and a new Chinese Tea making dataset

Abstract

In this paper, we concern with the problem of how to automatically extract the steps that compose real-life hand activities. This is a key competence towards processing, monitoring and providing video guidance in Mixed Reality systems. We use egocentric vision to observe hand-object interactions in real-world tasks and automatically decompose a video into its constituent steps. Our approach combines hand-object interaction (HOI) detection, object similarity measurement and a finite state machine (FSM) representation to automatically edit videos into steps. We use a combination of Convolutional Neural Networks (CNNs) and the FSM to discover, edit cuts and merge segments while observing real hand activities. We evaluate quantitatively and qualitatively our algorithm on two datasets: the GTEA\cite{li2015delving}, and a new dataset we introduce for Chinese Tea making. Results show our method is able to segment hand-object interaction videos into key step segments with high levels of precision.