Social Visual Behavior Analytics for Autism Therapy of Children Based on Automated Mutual Gaze Detection

TL;DR

This paper presents an automated framework for detecting mutual gaze in children with autism during therapy, providing reliable social visual behavior measures that can replace manual coding and assist in intervention evaluation.

Contribution

It introduces a novel automated mutual gaze detection method and validates its effectiveness in analyzing social visual behavior in autism therapy sessions.

Findings

Automated mutual gaze measures closely match human-coded data.

The framework effectively predicts social visual performance in children with autism.

Results support use in small-group social interaction assessments.

Abstract

Social visual behavior, as a type of non-verbal communication, plays a central role in studying social cognitive processes in interactive and complex settings of autism therapy interventions. However, for social visual behavior analytics in children with autism, it is challenging to collect gaze data manually and evaluate them because it costs a lot of time and effort for human coders. In this paper, we introduce a social visual behavior analytics approach by quantifying the mutual gaze performance of children receiving play-based autism interventions using an automated mutual gaze detection framework. Our analysis is based on a video dataset that captures and records social interactions between children with autism and their therapy trainers (N=28 observations, 84 video clips, 21 Hrs duration). The effectiveness of our framework was evaluated by comparing the mutual gaze ratio derived from the mutual gaze detection framework with the human-coded ratio values. We analyzed the mutual gaze frequency and duration across different therapy settings, activities, and sessions. We created mutual gaze-related measures for social visual behavior score prediction using multiple machine learning-based regression models. The results show that our method provides mutual gaze measures that reliably represent (or even replace) the human coders' hand-coded social gaze measures and effectively evaluates and predicts ASD children's social visual performance during the intervention. Our findings have implications for social interaction analysis in small-group behavior assessments in numerous co-located settings in (special) education and in the workplace.