Detecting Events of Daily Living Using Multimodal Data

TL;DR

This paper presents a novel multimodal data approach using semantic enrichment and knowledge graphs to automatically recognize and analyze daily living events from smartphone and wearable device data.

Contribution

It introduces an unobtrusive semantic context enrichment method with a knowledge graph for daily event recognition, including a new food recognition technique based on physical responses.

Findings

Successful event recognition with semantic enrichment.

Effective food classification from physical responses.

Demonstrated approach over 14 months of data from three users.

Abstract

Events are fundamental for understanding how people experience their lives. It is challenging, however, to automatically record all events in daily life. An understanding of multimedia signals allows recognizing events of daily living and getting their attributes as automatically as possible. In this paper, we consider the problem of recognizing a daily event by employing the commonly used multimedia data obtained from a smartphone and wearable device. We develop an unobtrusive approach to obtain latent semantic information from the data, and therefore an approach for daily event recognition based on semantic context enrichment. We represent the enrichment process through an event knowledge graph that semantically enriches a daily event from a low-level daily activity. To show a concrete example of this enrichment, we perform an experiment with eating activity, which may be one of the most complex events, by using 14 months of data for three users. In this process, to unobtrusively complement the lack of semantic information, we suggest a new food recognition/classification method that focuses only on a physical response to food consumption. Experimental results indicate that our approach is able to show automatic abstraction of life experience. These daily events can then be used to create a personal model that can capture how a person reacts to different stimuli under specific conditions.