Multimodal Emotion Recognition among Couples from Lab Settings to Daily Life using Smartwatches

TL;DR

This research develops automated multimodal emotion recognition systems for couples using smartwatch data, transitioning from lab-based to real-life daily monitoring to better support emotional well-being in chronic disease management.

Contribution

It introduces novel emotion recognition models based on extensive real-world data, filling a gap in couples' emotion recognition research beyond lab and observer-based methods.

Findings

Developed systems using 161 hours of multimodal data

Achieved accurate emotion recognition in daily life settings

Enabled continuous emotional monitoring for couples

Abstract

Couples generally manage chronic diseases together and the management takes an emotional toll on both patients and their romantic partners. Consequently, recognizing the emotions of each partner in daily life could provide an insight into their emotional well-being in chronic disease management. The emotions of partners are currently inferred in the lab and daily life using self-reports which are not practical for continuous emotion assessment or observer reports which are manual, time-intensive, and costly. Currently, there exists no comprehensive overview of works on emotion recognition among couples. Furthermore, approaches for emotion recognition among couples have (1) focused on English-speaking couples in the U.S., (2) used data collected from the lab, and (3) performed recognition using observer ratings rather than partner's self-reported / subjective emotions. In this body of work contained in this thesis (8 papers - 5 published and 3 currently under review in various journals), we fill the current literature gap on couples' emotion recognition, develop emotion recognition systems using 161 hours of data from a total of 1,051 individuals, and make contributions towards taking couples' emotion recognition from the lab which is the status quo, to daily life. This thesis contributes toward building automated emotion recognition systems that would eventually enable partners to monitor their emotions in daily life and enable the delivery of interventions to improve their emotional well-being.