LQF: Linear Quadratic Fine-Tuning

TL;DR

LQF introduces a linearization method for pre-trained models that achieves performance comparable to non-linear fine-tuning in image classification, combining interpretability with practical effectiveness.

Contribution

The paper presents a novel linearization approach for pre-trained models that maintains high performance, bridging the gap between linear models' interpretability and deep neural networks' accuracy.

Findings

Achieves comparable performance to non-linear fine-tuning on image classification tasks.

Effective in low-data regimes, outperforming traditional methods.

Uses simple modifications like Leaky-ReLU, mean squared loss, and Kronecker pre-conditioning.

Abstract

Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization. Such desirable properties are absent in deep neural networks (DNNs), typically trained by non-linear fine-tuning of a pre-trained model. Previous attempts to linearize DNNs have led to interesting theoretical insights, but have not impacted the practice due to the substantial performance gap compared to standard non-linear optimization. We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning on most of real-world image classification tasks tested, thus enjoying the interpretability of linear models without incurring punishing losses in performance. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification: Leaky-ReLU instead of ReLU, mean squared loss instead of cross-entropy, and pre-conditioning using Kronecker factorization. None of these changes in isolation is sufficient to approach the performance of non-linear fine-tuning. When used in combination, they allow us to reach comparable performance, and even superior in the low-data regime, while enjoying the simplicity, robustness and interpretability of linear-quadratic optimization.