Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games

TL;DR

This paper introduces the Sims4Action dataset, created from life simulation video games, for training activity recognition models, and evaluates transferability to real-world videos, highlighting both potential and challenges.

Contribution

It presents a novel dataset from simulation games for ADL recognition and a benchmark for transferring models to real-world videos.

Findings

Simulation data improves activity recognition models

Transfer from gaming to real videos is challenging

Life simulation games are a cost-effective data source

Abstract

Recognizing Activities of Daily Living (ADL) is a vital process for intelligent assistive robots, but collecting large annotated datasets requires time-consuming temporal labeling and raises privacy concerns, e.g., if the data is collected in a real household. In this work, we explore the concept of constructing training examples for ADL recognition by playing life simulation video games and introduce the SIMS4ACTION dataset created with the popular commercial game THE SIMS 4. We build Sims4Action by specifically executing actions-of-interest in a "top-down" manner, while the gaming circumstances allow us to freely switch between environments, camera angles and subject appearances. While ADL recognition on gaming data is interesting from the theoretical perspective, the key challenge arises from transferring it to the real-world applications, such as smart-homes or assistive robotics. To meet this requirement, Sims4Action is accompanied with a GamingToReal benchmark, where the models are evaluated on real videos derived from an existing ADL dataset. We integrate two modern algorithms for video-based activity recognition in our framework, revealing the value of life simulation video games as an inexpensive and far less intrusive source of training data. However, our results also indicate that tasks involving a mixture of gaming and real data are challenging, opening a new research direction. We will make our dataset publicly available at https://github.com/aroitberg/sims4action.