A Kaczmarz-Inspired Method for Orthogonalization

TL;DR

This paper introduces a stochastic iterative method inspired by Kaczmarz's algorithm that converges to an orthonormal basis for a set of vectors, with quantifiable convergence rates based on volume measures.

Contribution

The paper proposes a novel random iterative procedure for approximately orthogonalizing vectors and proves its almost sure convergence to an orthonormal basis.

Findings

The method converges almost surely to an orthonormal basis.

The volume of the parallelepiped approaches 1, indicating near orthogonality.

Convergence occurs within a number of iterations proportional to n^2 log(1/det|A|).

Abstract

This paper asks if the following iterative procedure approximately orthogonalizes a set of $n$ linearly independent unit vectors while preserving their span: in each iteration, access a random pair of vectors and replace one with the component perpendicular to the other, renormalized to be a unit vector. We provide a positive answer: any given set of starting vectors converges almost surely to an orthonormal basis of their span. We specifically argue that the $n$-volume of the parallelepiped generated by the vectors approaches 1 (i.e. the parallelepiped approaches a hypercube). If $A$ is the matrix formed by taking these vectors as columns, this volume is simply $\det(|A|)$ where $|A|=(A^*A)^{1/2}$. We show that $O(n^2\log(1/(\det(|A|)\varepsilon)))$ iterations suffice to bring ${\det(|A|)}$ above $1-\varepsilon$ with constant probability.