Spectral Regularization: an Inductive Bias for Sequence Modeling

TL;DR

This paper introduces a spectral regularization method for sequence modeling that leverages the trace norm of Hankel matrices to encode simplicity, validated through experiments on formal grammars.

Contribution

It proposes a novel spectral regularization technique based on Hankel matrix trace norm, with an unbiased stochastic estimator for bi-infinite matrices, enhancing sequence modeling.

Findings

Spectral regularization improves sequence modeling performance.

The stochastic estimator effectively handles bi-infinite Hankel matrices.

Experimental results on Tomita grammars validate the approach.

Abstract

Various forms of regularization in learning tasks strive for different notions of simplicity. This paper presents a spectral regularization technique, which attaches a unique inductive bias to sequence modeling based on an intuitive concept of simplicity defined in the Chomsky hierarchy. From fundamental connections between Hankel matrices and regular grammars, we propose to use the trace norm of the Hankel matrix, the tightest convex relaxation of its rank, as the spectral regularizer. To cope with the fact that the Hankel matrix is bi-infinite, we propose an unbiased stochastic estimator for its trace norm. Ultimately, we demonstrate experimental results on Tomita grammars, which exhibit the potential benefits of spectral regularization and validate the proposed stochastic estimator.