A Multi-Resolution Benchmark Framework for Spatial Reasoning Assessment in Neural Networks

TL;DR

This paper introduces a comprehensive benchmark framework for evaluating neural networks' spatial reasoning abilities, focusing on topology and geometry, revealing significant current limitations and guiding future hybrid approaches.

Contribution

The paper develops a multi-resolution benchmark framework for systematic spatial reasoning assessment in neural networks, integrating synthetic datasets, standardized training, and evaluation protocols.

Findings

Neural networks show significant challenges in basic geometric tasks.

Systematic failures observed in topological and spatial distance reasoning.

Framework enables reproducible evaluation of spatial reasoning capabilities.

Abstract

This paper presents preliminary results in the definition of a comprehensive benchmark framework designed to systematically evaluate spatial reasoning capabilities in neural networks, with a particular focus on morphological properties such as connectivity and distance relationships. The framework is currently being used to study the capabilities of nnU-Net, exploiting the spatial model checker VoxLogicA to generate two distinct categories of synthetic datasets: maze connectivity problems for topological analysis and spatial distance computation tasks for geometric understanding. Each category is evaluated across multiple resolutions to assess scalability and generalization properties. The automated pipeline encompasses a complete machine learning workflow including: synthetic dataset generation, standardized training with cross-validation, inference execution, and comprehensive evaluation using Dice coefficient and IoU (Intersection over Union) metrics. Preliminary experimental results demonstrate significant challenges in neural network spatial reasoning capabilities, revealing systematic failures in basic geometric and topological understanding tasks. The framework provides a reproducible experimental protocol, enabling researchers to identify specific limitations. Such limitations could be addressed through hybrid approaches combining neural networks with symbolic reasoning methods for improved spatial understanding in clinical applications, establishing a foundation for ongoing research into neural network spatial reasoning limitations and potential solutions.