Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models

TL;DR

This paper introduces a child-inspired interactive training method for language models using communicative success as a reward, aiming to enhance language learning through single-turn dialogues.

Contribution

It operationalizes communicative success in an abstract language-only setting and demonstrates its potential as an indirect grammaticality signal for reinforcement learning.

Findings

Reward correlates with grammaticality.

Interpretable changes in speaker behavior observed.

No significant improvements in linguistic evaluation metrics.

Abstract

We propose a method for training language models in an interactive setting inspired by child language acquisition. In our setting, a speaker attempts to communicate some information to a listener in a single-turn dialogue and receives a reward if communicative success is achieved. Unlike earlier related work using image--caption data for interactive reference games, we operationalize communicative success in a more abstract language-only question--answering setting. First, we present a feasibility study demonstrating that our reward provides an indirect signal about grammaticality. Second, we conduct experiments using reinforcement learning to fine-tune language models. We observe that cognitively plausible constraints on the communication channel lead to interpretable changes in speaker behavior. However, we do not yet see improvements on linguistic evaluations from our training regime. We outline potential modifications to the task design and training configuration that could better position future work to use our methodology to observe the benefits of interaction on language learning in computational cognitive models.