Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling

TL;DR

This paper introduces a language-informed program sampling model that uses large language models to generate and evaluate questions in the game Battleship, effectively mirroring human question-asking behavior with limited resources.

Contribution

It presents a novel approach combining LLMs with symbolic program evaluation to improve question-asking in uncertain environments, highlighting the benefits of Bayesian language models.

Findings

LIPS model generates human-like questions with modest resources

Grounded question-asking outperforms LLM-only baselines

GPT-4V does not improve over non-visual models in this task

Abstract

Questions combine our mastery of language with our remarkable facility for reasoning about uncertainty. How do people navigate vast hypothesis spaces to pose informative questions given limited cognitive resources? We study these tradeoffs in a classic grounded question-asking task based on the board game Battleship. Our language-informed program sampling (LIPS) model uses large language models (LLMs) to generate natural language questions, translate them into symbolic programs, and evaluate their expected information gain. We find that with a surprisingly modest resource budget, this simple Monte Carlo optimization strategy yields informative questions that mirror human performance across varied Battleship board scenarios. In contrast, LLM-only baselines struggle to ground questions in the board state; notably, GPT-4V provides no improvement over non-visual baselines. Our results illustrate how Bayesian models of question-asking can leverage the statistics of language to capture human priors, while highlighting some shortcomings of pure LLMs as grounded reasoners.