Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs

TL;DR

This paper explores integrating Reinforcement Learning with Large Language Models to create more generalizable agents for open-ended text-based educational environments, demonstrating the benefits of hybrid approaches.

Contribution

It introduces a novel benchmark, PharmaSimText, and compares RL, LLM, and hybrid agents, highlighting the advantages of combining RL with LLMs for better generalization.

Findings

RL agents excel at task completion but ask poor diagnostic questions.

LLM agents ask better diagnostic questions but perform worse in task completion.

Hybrid LLM-RL agents outperform individual strategies in both aspects.

Abstract

There has been a growing interest in developing learner models to enhance learning and teaching experiences in educational environments. However, existing works have primarily focused on structured environments relying on meticulously crafted representations of tasks, thereby limiting the agent's ability to generalize skills across tasks. In this paper, we aim to enhance the generalization capabilities of agents in open-ended text-based learning environments by integrating Reinforcement Learning (RL) with Large Language Models (LLMs). We investigate three types of agents: (i) RL-based agents that utilize natural language for state and action representations to find the best interaction strategy, (ii) LLM-based agents that leverage the model's general knowledge and reasoning through prompting, and (iii) hybrid LLM-assisted RL agents that combine these two strategies to improve agents' performance and generalization. To support the development and evaluation of these agents, we introduce PharmaSimText, a novel benchmark derived from the PharmaSim virtual pharmacy environment designed for practicing diagnostic conversations. Our results show that RL-based agents excel in task completion but lack in asking quality diagnostic questions. In contrast, LLM-based agents perform better in asking diagnostic questions but fall short of completing the task. Finally, hybrid LLM-assisted RL agents enable us to overcome these limitations, highlighting the potential of combining RL and LLMs to develop high-performing agents for open-ended learning environments.