Can we use LLMs to bootstrap reinforcement learning? -- A case study in digital health behavior change

TL;DR

This study investigates whether large language models can generate useful user interaction data to train reinforcement learning models for digital health behavior change, potentially reducing reliance on costly real data.

Contribution

It demonstrates that LLM-generated samples can effectively substitute real data in training reinforcement learning models for health behavior change applications.

Findings

LLM-generated samples can match human rater performance.

Effectiveness varies with prompting strategies and LLM types.

LLMs can be a practical tool for data augmentation in digital health interventions.

Abstract

Personalizing digital applications for health behavior change is a promising route to making them more engaging and effective. This especially holds for approaches that adapt to users and their specific states (e.g., motivation, knowledge, wants) over time. However, developing such approaches requires making many design choices, whose effectiveness is difficult to predict from literature and costly to evaluate in practice. In this work, we explore whether large language models (LLMs) can be used out-of-the-box to generate samples of user interactions that provide useful information for training reinforcement learning models for digital behavior change settings. Using real user data from four large behavior change studies as comparison, we show that LLM-generated samples can be useful in the absence of real data. Comparisons to the samples provided by human raters further show that LLM-generated samples reach the performance of human raters. Additional analyses of different prompting strategies including shorter and longer prompt variants, chain-of-thought prompting, and few-shot prompting show that the relative effectiveness of different strategies depends on both the study and the LLM with also relatively large differences between prompt paraphrases alone. We provide recommendations for how LLM-generated samples can be useful in practice.