Evaluation scheme for children-centered language interaction competence of AI-driven robots

TL;DR

This paper proposes a comprehensive evaluation framework for assessing AI-driven robots' language communication skills with children, emphasizing children's active participation and empirical validation through natural interaction data.

Contribution

It introduces a novel, child-centered evaluation scheme for robot language interaction, incorporating empirical data and insights from children and parents.

Findings

Presence of parents affects evaluation outcomes.

Framework shows high inter-rater consistency.

Empirical data supports scheme's effectiveness.

Abstract

This article explores the evaluation method for the language communication proficiency of AI-driven robots engaging in interactive communication with children. The utilization of AI-driven robots in children's everyday communication is swiftly advancing, underscoring the importance of evaluating these robots'language communication skills. Based on 11 Chinese families' interviews and thematic analysis of the comment text from shopping websites, a framework is introduced in the article to assess five key dimensions of child-robot language communication: interactivity, specificity, development, sociality, and safety. We draw on the concept of "children's agency", viewing children as active participants in shaping society and cultural life alongside adults. Therefore, this article places particular emphasis on collecting data related to children. Whether through survey interviews or direct interactive experiments, we treat children as an independent object for data collection. The study involved empirical research following the mentioned framework, which involved capturing interaction videos in natural conversation settings among children from 6 families. Analysis was performed on quantitative data obtained from video recordings, alongside questionnaires and interviews carried out by parents acting as participants or observers. We found that the presence or absence of parents during children's interactions with robots can impact the evaluation of robots'language communication abilities. Ultimately, this article proposes an enhanced comprehensive evaluation framework incorporating insights from parents and children, supported by empirical evidence and inter-rater consistency assessments, showcasing the scheme's efficacy.