Complexity Scaling Laws for Neural Models using Combinatorial Optimization

TL;DR

This paper develops complexity-based scaling laws for neural models using combinatorial optimization, demonstrating predictable growth in suboptimality with problem size and drawing analogies to local search methods.

Contribution

It introduces complexity measures like solution and representation space size to derive scaling laws, using TSP as a case study, applicable even without interpretable loss functions.

Findings

Suboptimality grows predictably with problem size.

Scaling laws hold across different training methods.

Gradient descent on cost landscapes shows similar trends.

Abstract

Recent work on neural scaling laws demonstrates that model performance scales predictably with compute budget, model size, and dataset size. In this work, we develop scaling laws based on problem complexity. We analyze two fundamental complexity measures: solution space size and representation space size. Using the Traveling Salesman Problem (TSP) as a case study, we show that combinatorial optimization promotes smooth cost trends, and therefore meaningful scaling laws can be obtained even in the absence of an interpretable loss. We then show that suboptimality grows predictably for fixed-size models when scaling the number of TSP nodes or spatial dimensions, independent of whether the model was trained with reinforcement learning or supervised fine-tuning on a static dataset. We conclude with an analogy to problem complexity scaling in local search, showing that a much simpler gradient descent of the cost landscape produces similar trends.