ModHiFi: Identifying High Fidelity predictive components for Model Modification

TL;DR

ModHiFi introduces a novel method to identify critical model components for modification tasks like pruning and unlearning without needing training data or loss access, based on local reconstruction errors and subset fidelity.

Contribution

The paper presents ModHiFi, a new algorithm that leverages local reconstruction errors to determine component importance, enabling model modification without training data or gradient access.

Findings

ModHiFi-P achieves 11% speedup over state-of-the-art pruning methods on ImageNet.

ModHiFi-U successfully unlearns classes on CIFAR-10 without fine-tuning.

Theoretical demonstration of Lipschitz continuity in CNNs and Transformers.

Abstract

Open weight models, which are ubiquitous, rarely provide access to their training data or loss function. This makes modifying such models for tasks such as pruning or unlearning, which are constrained by this unavailability, an active area of research. Existing techniques typically require gradients or ground-truth labels, rendering them infeasible in settings with limited computational resources. In this work, we investigate the fundamental question of identifying components that are critical to the model's predictive performance, without access to either gradients or the loss function, and with only distributional access such as synthetic data. We theoretically demonstrate that the global error is linearly bounded by local reconstruction errors for Lipschitz-continuous networks such as CNNs and well-trained Transformers (which, contrary to existing literature, we find exhibit Lipschitz continuity). This motivates using the locally reconstructive behavior of component subsets to quantify their global importance, via a metric that we term Subset Fidelity. In the uncorrelated features setting, selecting individual components based on their Subset Fidelity scores is optimal, which we utilize to propose ModHiFi, an algorithm for model modification that requires neither training data nor access to a loss function. ModHiFi-P, for structured pruning, achieves an 11\% speedup over the current state of the art on ImageNet models and competitive performance on language models. ModHiFi-U, for classwise unlearning, achieves complete unlearning on CIFAR-10 without fine-tuning and demonstrates competitive performance on Swin Transformers.