Goals as Reward-Producing Programs

TL;DR

This paper introduces a novel approach to modeling human goals as reward-producing programs, enabling the generation of human-like goals and games that are evaluated as fun and authentic by human judges.

Contribution

It presents a method to model goals as reward-producing programs, learn a fitness function over these programs, and generate novel, human-like goals through program synthesis and quality-diversity sampling.

Findings

Generated goals are indistinguishable from human-created games.

Model's fitness scores predict fun and human-like qualities.

Human evaluators favor model-generated goals in experiments.

Abstract

People are remarkably capable of generating their own goals, beginning with child's play and continuing into adulthood. Despite considerable empirical and computational work on goals and goal-oriented behavior, models are still far from capturing the richness of everyday human goals. Here, we bridge this gap by collecting a dataset of human-generated playful goals (in the form of scorable, single-player games), modeling them as reward-producing programs, and generating novel human-like goals through program synthesis. Reward-producing programs capture the rich semantics of goals through symbolic operations that compose, add temporal constraints, and allow for program execution on behavioral traces to evaluate progress. To build a generative model of goals, we learn a fitness function over the infinite set of possible goal programs and sample novel goals with a quality-diversity algorithm. Human evaluators found that model-generated goals, when sampled from partitions of program space occupied by human examples, were indistinguishable from human-created games. We also discovered that our model's internal fitness scores predict games that are evaluated as more fun to play and more human-like.