Challenges and Opportunities for Machine Learning Classification of Behavior and Mental State from Images

TL;DR

This paper reviews the challenges and opportunities in developing computer vision classifiers for detecting human behavior and mental states from images, emphasizing current research and future needs.

Contribution

It provides a comprehensive overview of the technical challenges and potential machine learning solutions for behavioral phenotyping using images.

Findings

Identifies key challenges such as data heterogeneity and bias.

Highlights state-of-the-art ML techniques like federated and meta-learning.

Discusses the need for improved representations and personalization.

Abstract

Computer Vision (CV) classifiers which distinguish and detect nonverbal social human behavior and mental state can aid digital diagnostics and therapeutics for psychiatry and the behavioral sciences. While CV classifiers for traditional and structured classification tasks can be developed with standard machine learning pipelines for supervised learning consisting of data labeling, preprocessing, and training a convolutional neural network, there are several pain points which arise when attempting this process for behavioral phenotyping. Here, we discuss the challenges and corresponding opportunities in this space, including handling heterogeneous data, avoiding biased models, labeling massive and repetitive data sets, working with ambiguous or compound class labels, managing privacy concerns, creating appropriate representations, and personalizing models. We discuss current state-of-the-art research endeavors in CV such as data curation, data augmentation, crowdsourced labeling, active learning, reinforcement learning, generative models, representation learning, federated learning, and meta-learning. We highlight at least some of the machine learning advancements needed for imaging classifiers to detect human social cues successfully and reliably.