Sentiment Analysis for Reinforcement Learning

TL;DR

This paper explores using sentiment analysis to convert sparse textual rewards into dense signals in reinforcement learning, improving learning efficiency in NLP tasks like text-based games.

Contribution

It introduces a novel framework that leverages sentiment analysis for reward shaping, enabling RL agents to learn from intrinsic sentiment cues in text descriptions.

Findings

Sentiment-based rewards improve RL performance in text-based games.

Dense rewards from sentiment analysis facilitate faster convergence.

The approach enables learning without explicit external rewards.

Abstract

While reinforcement learning (RL) has been successful in natural language processing (NLP) domains such as dialogue generation and text-based games, it typically faces the problem of sparse rewards that leads to slow or no convergence. Traditional methods that use text descriptions to extract only a state representation ignore the feedback inherently present in them. In text-based games, for example, descriptions like "Good Job! You ate the food}" indicate progress, and descriptions like "You entered a new room" indicate exploration. Positive and negative cues like these can be converted to rewards through sentiment analysis. This technique converts the sparse reward problem into a dense one, which is easier to solve. Furthermore, this can enable reinforcement learning without rewards, in which the agent learns entirely from these intrinsic sentiment rewards. This framework is similar to intrinsic motivation, where the environment does not necessarily provide the rewards, but the agent analyzes and realizes them by itself. We find that providing dense rewards in text-based games using sentiment analysis improves performance under some conditions.