Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law

TL;DR

This paper introduces the Capability Salience Vector, a method to better align validation loss with downstream task capabilities by weighting tokens according to their importance, improving performance prediction accuracy.

Contribution

It proposes a novel Capability Salience Vector that decomposes loss to reflect token importance for specific capabilities, bridging the gap between validation loss and downstream performance.

Findings

Improves downstream task performance prediction accuracy

Effectively aligns validation loss with model capabilities

Enhances understanding of token importance in loss modeling

Abstract

Scaling law builds the relationship between training computation and validation loss, enabling researchers to effectively predict the loss trending of models across different levels of computation. However, a gap still remains between validation loss and the model's downstream capabilities, making it untrivial to apply scaling law to direct performance prediction for downstream tasks. The loss typically represents a cumulative penalty for predicted tokens, which are implicitly considered to have equal importance. Nevertheless, our studies have shown evidence that when considering different training data distributions, we cannot directly model the relationship between downstream capability and computation or token loss. To bridge the gap between validation loss and downstream task capabilities, in this work, we introduce Capability Salience Vector, which decomposes the overall loss and assigns different importance weights to tokens to assess a specific meta-capability, aligning the validation loss with downstream task performance in terms of the model's capabilities. Experiments on various popular benchmarks demonstrate that our proposed Capability Salience Vector could significantly improve the predictability of language model performance on downstream tasks.