Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysis

TL;DR

Landscaper is a novel Python tool that uses topological data analysis and Hessian-based methods to analyze high-dimensional loss landscapes, revealing geometric features and training dynamics.

Contribution

It introduces a new open-source package combining topological analysis with Hessian methods for comprehensive loss landscape exploration in neural networks.

Findings

SMAD quantifies landscape smoothness and training transitions.

Landscaper reveals geometric structures like basin hierarchy and connectivity.

SMAD correlates with out-of-distribution generalization in scientific ML tasks.

Abstract

Loss landscapes are a powerful tool for understanding neural network optimization and generalization, yet traditional low-dimensional analyses often miss complex topological features. We present Landscaper, an open-source Python package for arbitrary-dimensional loss landscape analysis. Landscaper combines Hessian-based subspace construction with topological data analysis to reveal geometric structures such as basin hierarchy and connectivity. A key component is the Saddle-Minimum Average Distance (SMAD) for quantifying landscape smoothness. We demonstrate Landscaper's effectiveness across various architectures and tasks, including those involving pre-trained language models, showing that SMAD captures training transitions, such as landscape simplification, that conventional metrics miss. We also illustrate Landscaper's performance in challenging chemical property prediction tasks, where SMAD can serve as a metric for out-of-distribution generalization, offering valuable insights for model diagnostics and architecture design in data-scarce scientific machine learning scenarios.