Rethinking Parameter Sharing as Graph Coloring for Structured Compression

TL;DR

This paper introduces a systematic, graph-theoretic approach to parameter sharing in deep models, using a geometric criterion based on the Hessian spectrum to optimize structured compression and improve performance.

Contribution

It recasts parameter sharing as a graph coloring problem and proposes a scalable, principled method using Hessian analysis for cross-layer sharing configuration.

Findings

Outperforms heuristic sharing strategies across various architectures.

Achieves higher compression ratios with less accuracy loss.

Provides a theoretical framework linking sharing to structural symmetries.

Abstract

Modern deep models have massive parameter sizes, leading to high inference-time memory usage that limits practical deployment. Parameter sharing, a form of structured compression, effectively reduces redundancy, but existing approaches remain heuristic-restricted to adjacent layers and lacking a systematic analysis for cross-layer sharing. However, extending sharing across multiple layers leads to an exponentially expanding configuration space, making exhaustive search computationally infeasible and forming a critical bottleneck for parameter sharing. We recast parameter sharing from a group-theoretic perspective as introducing structural symmetries in the model's parameter space. A sharing configuration can be described by a coloring function $\alpha:L\rightarrow C$ (L: layer indices and C: sharing classes), which determines inter-layer sharing groups while preserving structural symmetry. To determine the coloring function, we propose a second-order geometric criterion based on Taylor expansion and the Hessian spectrum. By projecting perturbations onto the Hessian's low-curvature eigensubspace, the criterion provides an analytic rule for selecting sharing groups that minimize performance impact, yielding a principled and scalable configuration procedure. Across diverse architectures and tasks, Geo-Sharing consistently outperforms state-of-the-art heuristic sharing strategies, achieving higher compression ratios with smaller accuracy degradation.