An Improved Incremental Singular Value Decomposition and New Error Bounds

TL;DR

This paper introduces an improved incremental SVD algorithm that reduces reorthogonalization frequency, preserves exact output, and offers tighter error bounds, resulting in significantly faster computation.

Contribution

The authors restructure incremental SVD to accumulate updates implicitly, reducing orthogonal multiplications and providing new error bounds with proven accuracy.

Findings

The new algorithm runs 4.5 to 34 times faster than competitors.

Sharpened operator-norm truncation bound from n*tol to sqrt(n)*tol.

Loss of orthogonality depends only on the column length m, not stream length n.

Abstract

The incremental singular value decomposition (SVD) updates a truncated SVD as new columns arrive, replacing a single large SVD with a sequence of small ones. In floating-point arithmetic, each update multiplies the running singular basis by a small orthogonal factor, and the accumulated product loses orthogonality unless the basis is reorthogonalized periodically. How often this reorthogonalization is needed has been an open question; we answer it by restructuring the algorithm so that rank-preserving updates are accumulated implicitly and applied in batches, reducing the number of large orthogonal multiplications from $n$, the stream length, to $r$, the numerical rank. We prove that this restructuring preserves the exact-arithmetic output of the original algorithm and establish two forward-error bounds. First, we sharpen the existing operator-norm truncation bound from $n\,\texttt{tol}$ to $\sqrt{n}\,\texttt{tol}$, and show the new rate is attained on a constructive example. Second, under a standard probabilistic rounding-error model, we prove that the loss of orthogonality of the computed left factor is independent of the stream length $n$ and depends on $m$, the length of each incoming column, only through a single $\sqrt{m}$ factor. Numerical experiments confirm both bounds and demonstrate that the proposed algorithm runs $4.5\times$ to $34\times$ faster than its closest competitors.