The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units

TL;DR

This paper introduces the condition number as a scale-invariant measure of information encoding in neural units, linking it to entropy and geometric properties, and demonstrates its practical application in fine-tuning large language models to prevent catastrophic forgetting.

Contribution

It formalizes the relationship between the condition number and information encoding in neural networks, providing a theoretical basis and a practical method for selective fine-tuning.

Findings

High condition number correlates with reduced information transfer.

The proposed KappaTune method mitigates catastrophic forgetting effectively.

The approach does not require access to pre-training statistics.

Abstract

This paper explores the relationship between the condition number of a neural network's weight tensor and the extent of information encoded by the associated processing unit, viewed through the lens of information theory. It argues that a high condition number, though not sufficient for effective knowledge encoding, may indicate that the unit has learned to selectively amplify and compress information. This intuition is formalized for linear units with Gaussian inputs, linking the condition number and the transformation's log-volume scaling factor to the characteristics of the output entropy and the geometric properties of the learned transformation. The analysis demonstrates that for a fixed weight norm, a concentrated distribution of singular values (high condition number) corresponds to reduced overall information transfer, indicating a specialized and efficient encoding strategy. Furthermore, the linear stage entropy bound provides an upper limit on post-activation information for contractive, element-wise nonlinearities, supporting the condition number as a scale-invariant proxy for encoding capacity in practical neural networks. An empirical case study applies these principles to guide selective fine-tuning of Large Language Models for both a new task and a new input modality. The experiments show that the proposed method, named KappaTune, effectively mitigates catastrophic forgetting. Unlike many existing catastrophic forgetting mitigation methods that rely on access to pre-training statistics, which are often unavailable, this selective fine-tuning approach offers a way to bypass this common requirement.