Language Decision Transformers with Exponential Tilt for Interactive Text Environments

TL;DR

This paper introduces Language Decision Transformers (LDTs), a novel framework combining transformer models and decision transformers to improve agent performance in complex text-based games with long sequences and sparse rewards.

Contribution

The paper presents LDTs with exponential tilt, new goal conditioning methods, and a future observation model, advancing offline reinforcement learning in text environments.

Findings

LDTs outperform other agents on challenging Jericho games.

LDTs achieve the highest scores on Enchanter.

The proposed methods improve goal guidance and observation modeling.

Abstract

Text-based game environments are challenging because agents must deal with long sequences of text, execute compositional actions using text and learn from sparse rewards. We address these challenges by proposing Language Decision Transformers (LDTs), a framework that is based on transformer language models and decision transformers (DTs). Our LDTs extend DTs with 3 components: (1) exponential tilt to guide the agent towards high obtainable goals, (2) novel goal conditioning methods yielding better results than the traditional return-to-go (sum of all future rewards), and (3) a model of future observations that improves agent performance. LDTs are the first to address offline RL with DTs on these challenging games. Our experiments show that LDTs achieve the highest scores among many different types of agents on some of the most challenging Jericho games, such as Enchanter.