DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

TL;DR

This paper introduces DP-Muon, a differentially private matrix optimizer with orthogonalization, providing privacy guarantees and improved utility in private training tasks.

Contribution

The paper formulates DP-Muon, analyzes its privacy and optimization properties, and proposes a bias-corrected variant DP-MuonBC that enhances utility without extra privacy cost.

Findings

DP-Muon inherits privacy guarantees from Gaussian accountant without extra cost.

Theoretical bounds separate optimization error, noise, and approximation errors.

DP-MuonBC reduces bias and improves utility in private fine-tuning experiments.

Abstract

We study differentially private (DP) training with Muon, a matrix-valued optimizer that updates hidden-layer weights using momentum followed by Newton--Schulz orthogonalization. While DP-SGD is well understood, the interaction between per-example clipping, Gaussian noise, momentum, and nonlinear orthogonalization in Muon has not been systematically analyzed. We formulate DP-Muon, a private Muon procedure that clips per-example matrix gradients, adds Gaussian noise to the clipped lot average, and then applies momentum and Newton--Schulz orthogonalization as post-processing. We prove that DP-Muon inherits the privacy guarantee certified by the corresponding same-lot subsampled Gaussian accountant, with no additional privacy cost from Muon-specific post-processing. On the optimization side, we establish finite-horizon and vanishing stationarity guarantees under per-matrix clipping, with bounds that separate optimization error, clipping residual, privacy noise, and Newton--Schulz approximation error. We further show that the DP-induced bias in Muon arises not in the linear momentum buffer itself, but after the nonlinear Newton--Schulz map, where Gaussian noise induces a matrix-valued heat-smoothing bias. This motivates DP-MuonBC, a bias-corrected variant that removes the leading output-level bias term while preserving the same privacy guarantee. Experiments on E2E and DART show that Muon-style matrix updates improve private fine-tuning, and that DP-MuonBC further improves utility without increasing the privacy budget.