Input-Side Variance Suppression under Non-Normal Transient Amplification in Continuous-Control Reinforcement Learning

TL;DR

This paper investigates how non-normal transient amplification in control loops causes high variance in reinforcement learning and proposes an input-side variance suppression layer to mitigate this effect.

Contribution

It introduces a novel input-side variance suppression layer and provides control-theoretic analysis to reduce variance caused by non-normal amplification in RL systems.

Findings

Input-side suppression reduces control jitter and variance.

Non-normal amplification significantly contributes to variance in RL.

Control interventions validate the mechanism on quadrotor tasks.

Abstract

Continuous-control reinforcement learning (RL) often exhibits large closed-loop variance, high-frequency control jitter, and sensitivity to disturbance injection. Existing explanations usually emphasize disturbance sources such as action noise, exploration perturbations, or policy nonsmoothness. This letter studies a complementary amplifier-side perspective: in nominally stable yet strongly non-normal closed loops, small input perturbations can undergo transient amplification and lead to disproportionately large state covariance. Motivated by this source--amplifier decomposition, we introduce an input-side variance suppression layer that operates between the learned policy and the plant input to reduce applied-input variance and step-to-step jitter. To separate mechanism from correlation, we use two control-theoretic interventions: one varies only eigenvector geometry under fixed eigenvalues and spectral radius, and the other varies only applied-input statistics under fixed strongly non-normal geometry. We then provide mechanism-consistent external validation on planar quadrotor tasks. Throughout, Koopman/ALE surrogates are used only as analysis and certification tools, not as direct performance paths. Taken together, the results support a narrower claim: in the studied settings, non-normal transient amplification is an important and under-emphasized contributor to execution-time closed-loop variance, and source-side suppression can reduce downstream covariance without changing the structural peak gain.