Algorithmic Barriers to Detecting and Repairing Structural Overspecification in Adaptive Data-Structure Selection

TL;DR

This paper investigates the fundamental algorithmic challenges in identifying and fixing structural overspecification in adaptive data-structure selection, revealing undecidability in general and fixed points in conservative repairs.

Contribution

It establishes undecidability of detecting overspecification in unbounded domains and shows fixed points exist for conservative repair operators, highlighting fundamental barriers.

Findings

Undecidability of structural overspecification detection on infinite domains.

Existence of overspecified fixed points under conservative repair constraints.

Barriers differ from classical data-structure lower bounds, affecting pipeline detectability.

Abstract

We study algorithmic barriers to detecting and repairing a systematic form of structural overspecification in adaptive data-structure selection. An input instance induces an implied workload signature, such as ordering, sparsity, dynamism, locality, or substring structure, and candidate implementations may be preferred because they match that full signature even when the measured workload evidence supports only a strict subset of it. Under a model in which pairwise evaluators favor implementations that realize the implied signature, we show that this preference propagates through both benchmark aggregation and Bradley-Terry-Luce fitting. We then establish two main results. First, determining whether a representation-selection pipeline exhibits structural commitment beyond measured warrant is undecidable on unbounded input domains, by reduction from the halting problem, but decidable by exhaustive enumeration on finite domains. Second, under a conservative repair constraint requiring already evidence-aligned pipelines to remain unchanged, any total computable repair operator admits an overspecified fixed point via Kleene's recursion theorem. These barriers are qualitatively different from classical lower bounds in data-structure design: they do not limit efficiency on finite workloads, but the possibility of uniformly detecting and repairing overspecification across pipeline families.