Contextures: Representations from Contexts

TL;DR

This paper introduces the contexture theory, a framework that characterizes how many representation learning methods capture input-context associations, revealing implications for model scaling and the importance of better contexts.

Contribution

The paper establishes the contexture theory, unifying various learning paradigms under a common framework and analyzing the impact of model size and context quality on representations.

Findings

Representation learning approximates top singular functions of the expectation operator.

Scaling models beyond a certain size yields diminishing returns without better contexts.

Proposed a metric to evaluate context usefulness independent of downstream tasks.

Abstract

Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the contexture theory. It shows that a large class of representation learning methods can be characterized as learning from the association between the input and a context variable. Specifically, we show that many popular methods aim to approximate the top-d singular functions of the expectation operator induced by the context, in which case we say that the representation learns the contexture. We demonstrate the generality of the contexture theory by proving that representation learning within various learning paradigms -- supervised, self-supervised, and manifold learning -- can all be studied from such a perspective. We also prove that the representations that learn the contexture are optimal on those tasks that are compatible with the context. One important implication of the contexture theory is that once the model is large enough to approximate the top singular functions, further scaling up the model size yields diminishing returns. Therefore, scaling is not all we need, and further improvement requires better contexts. To this end, we study how to evaluate the usefulness of a context without knowing the downstream tasks. We propose a metric and show by experiments that it correlates well with the actual performance of the encoder on many real datasets.