LocationReasoner: Evaluating LLMs on Real-World Site Selection Reasoning

TL;DR

LocationReasoner is a benchmark for evaluating large language models' reasoning skills in real-world site selection, revealing current models' limited performance and challenges in complex spatial reasoning tasks.

Contribution

The paper introduces LocationReasoner, a novel benchmark with tools and verification for assessing LLMs' reasoning in real-world spatial and logistical scenarios.

Findings

State-of-the-art models show limited improvement over predecessors.

OpenAI o4 model fails on 30% of site selection tasks.

Agentic strategies like ReAct can worsen outcomes due to over-reasoning.

Abstract

Recent advances in large language models (LLMs), particularly those enhanced through reinforced post-training, have demonstrated impressive reasoning capabilities, as exemplified by models such as OpenAI o1 and DeepSeek-R1. However, these capabilities are predominantly benchmarked on domains like mathematical problem solving and code generation, leaving open the question of whether such reasoning skills generalize to complex real-world scenarios. In this paper, we introduce LocationReasoner, a benchmark designed to evaluate LLMs' reasoning abilities in the context of real-world site selection, where models must identify feasible locations by reasoning over diverse and complicated spatial, environmental, and logistic constraints. The benchmark covers carefully crafted queries of varying difficulty levels and is supported by a sandbox environment with in-house tools for constraint-based location search. Automated verification further guarantees the scalability of the benchmark, enabling the addition of arbitrary number of queries. Extensive evaluations on real-world site selection data from Boston, New York, and Tampa reveal that state-of-the-art reasoning models offer limited improvement over their non-reasoning predecessors in real-world contexts, with even the latest OpenAI o4 model failing on 30% of site selection tasks. Moreover, agentic strategies such as ReAct and Reflexion often suffer from over-reasoning, leading to worse outcomes than direct prompting. With key limitations of LLMs in holistic and non-linear reasoning highlighted, we release LocationReasoner to foster the development of LLMs and agents capable of robust, grounded reasoning in real-world decision-making tasks. Codes and data for our benchmark are available at https://github.com/miho-koda/LocationReasoner.