Batch-efficient EigenDecomposition for Small and Medium Matrices

TL;DR

This paper introduces a GPU-efficient QR-based eigen decomposition method tailored for small to medium matrices, significantly reducing computation time and enhancing performance in computer vision tasks.

Contribution

The paper presents a novel batched eigen decomposition technique using QR iterations with Givens rotations, optimized for GPU processing and small to medium matrices.

Findings

Our method outperforms PyTorch SVD for matrices smaller than 32 in dimension.

It reduces QR iteration complexity from O(n^5) to O(n^3).

Experimental results show competitive performance in vision applications.

Abstract

EigenDecomposition (ED) is at the heart of many computer vision algorithms and applications. One crucial bottleneck limiting its usage is the expensive computation cost, particularly for a mini-batch of matrices in the deep neural networks. In this paper, we propose a QR-based ED method dedicated to the application scenarios of computer vision. Our proposed method performs the ED entirely by batched matrix/vector multiplication, which processes all the matrices simultaneously and thus fully utilizes the power of GPUs. Our technique is based on the explicit QR iterations by Givens rotation with double Wilkinson shifts. With several acceleration techniques, the time complexity of QR iterations is reduced from $O{(}n^5{)}$ to $O{(}n^3{)}$. The numerical test shows that for small and medium batched matrices (\emph{e.g.,} $dim{<}32$) our method can be much faster than the Pytorch SVD function. Experimental results on visual recognition and image generation demonstrate that our methods also achieve competitive performances.