An analysis of observation length requirements for machine understanding of human behaviors from spoken language

TL;DR

This paper investigates how observation window length affects the accuracy of machine understanding of human behaviors from spoken language, proposing an evaluation framework and analyzing various behavior types.

Contribution

It introduces a framework for determining optimal observation lengths for behavior estimation and analyzes the relationship between behavior nature and observation duration.

Findings

Positive and negative affect behaviors are estimated accurately from short to medium observations.

Problem-solving and dysphoria behaviors require longer observations for accurate estimation.

Findings are consistent across different behavior modeling approaches.

Abstract

The task of quantifying human behavior by observing interaction cues is an important and useful one across a range of domains in psychological research and practice. Machine learning-based approaches typically perform this task by first estimating behavior based on cues within an observation window, such as a fixed number of words, and then aggregating the behavior over all the windows in that interaction. The length of this window directly impacts the accuracy of estimation by controlling the amount of information being used. The exact link between window length and accuracy, however, has not been well studied, especially in spoken language. In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation. Our proposed framework utilizes a two-pronged evaluation approach: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis to real-life conversations that are annotated for a large and diverse set of behavior codes and examine the relation between the nature of a behavior and how long it should be observed. We find that behaviors describing negative and positive affect can be accurately estimated from short to medium-length expressions whereas behaviors related to problem-solving and dysphoria require much longer observations and are difficult to quantify from language alone. These findings are found to be generally consistent across different behavior modeling approaches.