TaskLAMA: Probing the Complex Task Understanding of Language Models

TL;DR

This paper evaluates how well large language models can decompose complex real-world tasks into steps and dependencies, introducing a new dataset and metrics, and highlighting their strengths and limitations in task understanding.

Contribution

It presents a high-quality human-annotated dataset and novel metrics for structured complex task decomposition, and assesses LLMs' performance with significant improvements over baselines.

Findings

LLMs effectively decompose tasks into steps with 15%-280% improvement over baselines.

Proposed methods further improve LLM performance by 7%-37%.

LLMs struggle to accurately predict pairwise temporal dependencies.

Abstract

Structured Complex Task Decomposition (SCTD) is the problem of breaking down a complex real-world task (such as planning a wedding) into a directed acyclic graph over individual steps that contribute to achieving the task, with edges specifying temporal dependencies between them. SCTD is an important component of assistive planning tools, and a challenge for commonsense reasoning systems. We probe how accurately SCTD can be done with the knowledge extracted from Large Language Models (LLMs). We introduce a high-quality human-annotated dataset for this problem and novel metrics to fairly assess performance of LLMs against several baselines. Our experiments reveal that LLMs are able to decompose complex tasks into individual steps effectively, with a relative improvement of 15% to 280% over the best baseline. We also propose a number of approaches to further improve their performance, with a relative improvement of 7% to 37% over the base model. However, we find that LLMs still struggle to predict pairwise temporal dependencies, which reveals a gap in their understanding of complex tasks.