Interaction Analysis by Humans and AI: A Comparative Perspective

TL;DR

This study compares children's communication in Mixed Reality and video conferencing, showing MR enhances interaction richness and engagement, while Zoom offers simplicity, with LLMs aiding analysis efficiency despite some limitations.

Contribution

It demonstrates the use of Large Language Models for analyzing children's interactions across MR and video conferencing platforms, highlighting both benefits and limitations.

Findings

MR fosters richer interaction and emotional expression

Zoom provides simplicity and accessibility

LLMs reduce analysis time despite annotation limitations

Abstract

This paper explores how Mixed Reality (MR) and 2D video conferencing influence children's communication during a gesture-based guessing game. Finnish-speaking participants engaged in a short collaborative task using two different setups: Microsoft HoloLens MR and Zoom. Audio-video recordings were transcribed and analyzed using Large Language Models (LLMs), enabling iterative correction, translation, and annotation. Despite limitations in annotations' accuracy and agreement, automated approaches significantly reduced processing time and allowed non-Finnish-speaking researchers to participate in data analysis. Evaluations highlight both the efficiency and constraints of LLM-based analyses for capturing children's interactions across these platforms. Initial findings indicate that MR fosters richer interaction, evidenced by higher emotional expression during annotation, and heightened engagement, while Zoom offers simplicity and accessibility. This study underscores the potential of MR to enhance collaborative learning experiences for children in distributed settings.