Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering

TL;DR

This paper develops a theoretical and algorithmic framework for clustering matrices for joint SVD compression under explicit error constraints, providing spectral bounds and efficient incremental estimation methods.

Contribution

It introduces a principled, error-aware clustering approach for matrix compression, with new spectral bounds and scalable algorithms based on incremental SVD estimation.

Findings

Derived spectral bounds for concatenated matrices' SVD error

Developed an efficient incremental SVD estimator for error prediction

Proposed clustering algorithms with explicit error control

Abstract

Large collections of matrices arise throughout modern machine learning, signal processing, and scientific computing, where they are commonly compressed by concatenation followed by truncated singular value decomposition (SVD). This strategy enables parameter sharing and efficient reconstruction and has been widely adopted across domains ranging from multi-view learning and signal processing to neural network compression. However, it leaves a fundamental question unanswered: which matrices can be safely concatenated and compressed together under explicit reconstruction error constraints? Existing approaches rely on heuristic or architecture-specific grouping and provide no principled guarantees on the resulting SVD approximation error. In the present work, we introduce a theory-driven framework for compression-aware clustering of matrices under SVD compression constraints. Our analysis establishes new spectral bounds for horizontally concatenated matrices, deriving global upper bounds on the optimal rank-$r$ SVD reconstruction error from lower bounds on singular value growth. The first bound follows from Weyl-type monotonicity under blockwise extensions, while the second leverages singular values of incremental residuals to yield tighter, per-block guarantees. We further develop an efficient approximate estimator based on incremental truncated SVD that tracks dominant singular values without forming the full concatenated matrix. Therefore, we propose three clustering algorithms that merge matrices only when their predicted joint SVD compression error remains below a user-specified threshold. The algorithms span a trade-off between speed, provable accuracy, and scalability, enabling compression-aware clustering with explicit error control. Code is available online.