Story Shaping: Teaching Agents Human-like Behavior with Stories

TL;DR

This paper introduces Story Shaping, a reinforcement learning technique where agents learn human-like behaviors by inferring and aligning their world state with exemplar stories, improving safety and role adherence.

Contribution

The paper presents a novel method for teaching agents human-like behaviors through story-based knowledge inference and intrinsic rewards.

Findings

Effective in text-based games requiring commonsense reasoning

Improves agent adherence to desired behaviors and safety

Demonstrates success in shaping virtual game characters

Abstract

Reward design for reinforcement learning agents can be difficult in situations where one not only wants the agent to achieve some effect in the world but where one also cares about how that effect is achieved. For example, we might wish for an agent to adhere to a tacit understanding of commonsense, align itself to a preference for how to behave for purposes of safety, or taking on a particular role in an interactive game. Storytelling is a mode for communicating tacit procedural knowledge. We introduce a technique, Story Shaping, in which a reinforcement learning agent infers tacit knowledge from an exemplar story of how to accomplish a task and intrinsically rewards itself for performing actions that make its current environment adhere to that of the inferred story world. Specifically, Story Shaping infers a knowledge graph representation of the world state from observations, and also infers a knowledge graph from the exemplar story. An intrinsic reward is generated based on the similarity between the agent's inferred world state graph and the inferred story world graph. We conducted experiments in text-based games requiring commonsense reasoning and shaping the behaviors of agents as virtual game characters.