Distilling Script Knowledge from Large Language Models for Constrained Language Planning

TL;DR

This paper introduces the task of constrained language planning, proposes an overgenerate-then-filter method to improve large language models, and creates a new dataset, CoScript, to enhance and evaluate constrained planning abilities.

Contribution

It defines the novel task of constrained language planning, develops a new dataset, and presents an effective distillation method to improve LLMs' planning with constraints.

Findings

Significant improvement in constraint faithfulness of LLMs.

Small LMs can effectively learn constrained planning from CoScript.

The overgenerate-then-filter approach enhances planning accuracy.

Abstract

In everyday life, humans often plan their actions by following step-by-step instructions in the form of goal-oriented scripts. Previous work has exploited language models (LMs) to plan for abstract goals of stereotypical activities (e.g., "make a cake"), but leaves more specific goals with multi-facet constraints understudied (e.g., "make a cake for diabetics"). In this paper, we define the task of constrained language planning for the first time. We propose an overgenerate-then-filter approach to improve large language models (LLMs) on this task, and use it to distill a novel constrained language planning dataset, CoScript, which consists of 55,000 scripts. Empirical results demonstrate that our method significantly improves the constrained language planning ability of LLMs, especially on constraint faithfulness. Furthermore, CoScript is demonstrated to be quite effective in endowing smaller LMs with constrained language planning ability.