Evaluating LLMs on Generating Age-Appropriate Child-Like Conversations

TL;DR

This study evaluates how well five large language models generate authentic, age-appropriate child-like conversations in Norwegian, revealing challenges in modeling language suitable for children and highlighting data limitations.

Contribution

It provides a comparative analysis of LLMs' ability to produce age-appropriate dialogue for children, emphasizing the need for better training data for child language modeling.

Findings

GPT-4 and NorBloom-7b performed relatively well.

Models often generated language more advanced than children's speech.

High inter-rater reliability in evaluations (ICC=0.75).

Abstract

Large Language Models (LLMs), predominantly trained on adult conversational data, face significant challenges when generating authentic, child-like dialogue for specialized applications. We present a comparative study evaluating five different LLMs (GPT-4, RUTER-LLAMA-2-13b, GPTSW, NorMistral-7b, and NorBloom-7b) to generate age-appropriate Norwegian conversations for children aged 5 and 9 years. Through a blind evaluation by eleven education professionals using both real child interview data and LLM-generated text samples, we assessed authenticity and developmental appropriateness. Our results show that evaluators achieved strong inter-rater reliability (ICC=0.75) and demonstrated higher accuracy in age prediction for younger children (5-year-olds) compared to older children (9-year-olds). While GPT-4 and NorBloom-7b performed relatively well, most models generated language perceived as more linguistically advanced than the target age groups. These findings highlight critical data-related challenges in developing LLM systems for specialized applications involving children, particularly in low-resource languages where comprehensive age-appropriate lexical resources are scarce.