Opportunistic and Context-aware Affect Sensing on Smartphones: The Concept, Challenges and Opportunities

TL;DR

This paper reviews recent advances and challenges in implementing opportunistic, context-aware affect sensing on smartphones, focusing on facial expression and voice detection enabled by new low-power hardware technologies.

Contribution

It identifies key barriers and potential solutions for deploying affect sensing systems that are both opportunistic and context-aware on smartphones.

Findings

Low-power DSP and GPU enable feasible audio-visual affect sensing

Context inference is crucial for accurate affect detection

Barriers include power consumption and contextual complexity

Abstract

Opportunistic affect sensing offers unprecedented potential for capturing spontaneous affect ubiquitously, obviating biases inherent in the laboratory setting. Facial expression and voice are two major affective displays, however most affect sensing systems on smartphone avoid them due to extensive power requirement. Encouragingly, due to the recent advent of low-power DSP (Digital Signal Processing) co-processor and GPU (Graphics Processing Unit) technology, audio and video sensing are becoming more feasible. To properly evaluate opportunistically captured facial expression and voice, contextual information about the dynamic audio-visual stimuli needs to be inferred. This paper discusses recent advances of affect sensing on the smartphone and identifies the key barriers and potential solutions of implementing opportunistic and context-aware affect sensing on smartphone platforms.