Interpolating Compressed Parameter Subspaces

TL;DR

This paper introduces Compressed Parameter Subspaces (CPS), a geometric framework for interpolating trained neural network parameters across shifted input distributions, enabling robust performance in diverse and unseen task variations.

Contribution

The paper proposes CPS as a novel method for parameter subspace sampling that captures shifted distributions, improving robustness and continual learning capabilities.

Findings

Ensembling within CPS achieves high accuracy across various distribution shifts.

CPS contains low-loss points for different task shifts, including unseen and perturbed labels.

Demonstrated effectiveness in continual learning with CIFAR100.

Abstract

Inspired by recent work on neural subspaces and mode connectivity, we revisit parameter subspace sampling for shifted and/or interpolatable input distributions (instead of a single, unshifted distribution). We enforce a compressed geometric structure upon a set of trained parameters mapped to a set of train-time distributions, denoting the resulting subspaces as Compressed Parameter Subspaces (CPS). We show the success and failure modes of the types of shifted distributions whose optimal parameters reside in the CPS. We find that ensembling point-estimates within a CPS can yield a high average accuracy across a range of test-time distributions, including backdoor, adversarial, permutation, stylization and rotation perturbations. We also find that the CPS can contain low-loss point-estimates for various task shifts (albeit interpolated, perturbed, unseen or non-identical coarse labels). We further demonstrate this property in a continual learning setting with CIFAR100.