Hessian Surgery: Class-Targeted Post-Hoc Rebalancing via Hessian Spike Perturbation

TL;DR

Hessian Surgery is a post-hoc method that perturbs model weights along Hessian eigenvectors to improve class balance in deep networks without retraining.

Contribution

It introduces a novel Hessian-based perturbation technique and optimization framework to rebalance class accuracy after training.

Findings

Improves balanced accuracy on CIFAR-10 and ISIC-2019 datasets.

Effectively targets weak classes without degrading strong ones.

Operates post-hoc without retraining.

Abstract

The Hessian spectrum of trained deep networks exhibits a characteristic structure: a continuous bulk of near-zero eigenvalues and a small number of large outlier eigenvalues (spikes), confirming the relevance of Random Matrix Theory in deep learning. The spike count matches the number of classes minus one. While prior work has described this structure, no method has exploited it operationally to improve classification performance. We propose Hessian Surgery, a post-hoc optimization method that directly perturbs model weights along spike eigenvectors to rebalance per-class accuracy without retraining. We introduce (i) a spike-class sensitivity matrix that quantifies the directional derivative of each class's accuracy along each spike eigenvector, (ii) a constrained optimization of perturbation coefficients that targets weak classes while preserving strong ones, and (iii) an adaptive amplitude control that raises or lowers the perturbation budget based on iteration-level improvement signals. We obtain encouraging results on CIFAR-10 and ISIC-2019 on both balanced accuracy and standard deviation.