Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory

TL;DR

This paper investigates optimal singular value thresholds for low-rank approximations of DNN weight matrices using random matrix theory, proposing a cosine similarity-based metric for evaluation.

Contribution

It introduces a novel evaluation metric and compares threshold estimation methods for singular value-based weight matrix approximation in neural networks.

Findings

The proposed metric effectively assesses threshold quality.

Random matrix theory-based thresholds improve approximation accuracy.

Comparison results favor certain threshold estimation methods.

Abstract

This study evaluates thresholds for removing singular values from singular value decomposition-based low-rank approximations of deep neural network weight matrices. Each weight matrix is modeled as the sum of signal and noise matrices. The low-rank approximation is obtained by removing noise-related singular values using a threshold based on random matrix theory. To assess the adequacy of this threshold, we propose an evaluation metric based on the cosine similarity between the singular vectors of the signal and original weight matrices. The proposed metric is used in numerical experiments to compare two threshold estimation methods.