# Hamiltonian Monte-Carlo for Orthogonal Matrices

## TL;DR

This paper introduces a novel Hamiltonian Monte Carlo method for sampling orthogonal matrices that leverages Riemannian optimization, avoiding exact geodesic computations, and demonstrates improved efficiency in Bayesian neural network applications.

## Contribution

It proposes a new HMC scheme for orthogonal matrices based on Riemannian optimization, which is faster and more sample-efficient than existing methods, and provides theoretical and empirical validation.

## Key findings

- Comparable or faster per iteration than existing methods
- More sample-efficient in Bayesian neural network applications
- Effective in low-rank matrix factorization tasks

## Abstract

We consider the problem of sampling from posterior distributions for Bayesian models where some parameters are restricted to be orthogonal matrices. Such matrices are sometimes used in neural networks models for reasons of regularization and stabilization of training procedures, and also can parameterize matrices of bounded rank, positive-definite matrices and others. In \citet{byrne2013geodesic} authors have already considered sampling from distributions over manifolds using exact geodesic flows in a scheme similar to Hamiltonian Monte Carlo (HMC). We propose new sampling scheme for a set of orthogonal matrices that is based on the same approach, uses ideas of Riemannian optimization and does not require exact computation of geodesic flows. The method is theoretically justified by proof of symplecticity for the proposed iteration. In experiments we show that the new scheme is comparable or faster in time per iteration and more sample-efficient comparing to conventional HMC with explicit orthogonal parameterization and Geodesic Monte-Carlo. We also provide promising results of Bayesian ensembling for orthogonal neural networks and low-rank matrix factorization.