Solving and Generating NPR Sunday Puzzles with Large Language Models

TL;DR

This paper evaluates large language models' ability to solve and generate NPR Sunday puzzles using a specialized dataset, finding they perform well in solving but struggle with puzzle creation.

Contribution

It introduces PUZZLEQA, a new dataset for puzzle solving, and assesses LLMs' capabilities in solving and generating puzzles, highlighting current limitations in puzzle generation.

Findings

GPT-3.5 achieves 50.2% accuracy in solving puzzles

Models excel at solving but struggle with puzzle generation

Puzzle generation remains a challenging open problem

Abstract

We explore the ability of large language models to solve and generate puzzles from the NPR Sunday Puzzle game show using PUZZLEQA, a dataset comprising 15 years of on-air puzzles. We evaluate four large language models using PUZZLEQA, in both multiple choice and free response formats, and explore two prompt engineering techniques to improve free response performance: chain-of-thought reasoning and prompt summarization. We find that state-of-the-art large language models can solve many PUZZLEQA puzzles: the best model, GPT-3.5, achieves 50.2% loose accuracy. However, in our few-shot puzzle generation experiment, we find no evidence that models can generate puzzles: GPT-3.5 generates puzzles with answers that do not conform to the generated rules. Puzzle generation remains a challenging task for future work.