A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup

TL;DR

This paper demonstrates that structure proxies used to interpret out-of-distribution performance can be misleading, showing cases where proxy rankings do not align with actual OOD accuracy rankings in a controlled setup.

Contribution

It provides a formal counterexample and experimental evidence that proxy-based explanations of OOD performance can fail, establishing a boundary for their reliability.

Findings

Proxy rankings can reverse OOD accuracy rankings in controlled experiments.

A formal construction separates structure proxies from task-relevant structures.

Counterexamples highlight limits of proxy-based explanations for OOD transfer.

Abstract

Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another, but such explanations require the proxy to track the structure that matters for the downstream task. We test this requirement in a fixed pretraining-and-probing setup motivated by computationally bounded notions of learned structure, including epiplexity. The core question is whether a proxy ranking of two pretraining datasets must agree with their ranking by OOD probe accuracy. We show that it need not. First, we give a controlled construction in which a formal structure quantity, its operational proxy, and the task-relevant structure for a target family separate. We then instantiate the same mechanism in a synthetic sequence-model experiment: under the primary all-sample evaluation, the OOD accuracy ranking reverses the proxy ranking in two of three seeds, with auxiliary diagnostics and ablations supporting the same interpretation. The counterexample does not reject structure-based explanations in general; it identifies a boundary on strong proxy-based explanations. A proxy for total learned structure can fail to track the task-relevant structure that drives OOD performance, even in a controlled setting.