Target-Embedding Autoencoders for Supervised Representation Learning

TL;DR

This paper introduces target-embedding autoencoders (TEA), a supervised learning framework that improves generalization by learning intermediate representations predictive of high-dimensional targets, with theoretical guarantees and empirical validation across various architectures.

Contribution

The paper formalizes TEA for supervised prediction, providing a generalization guarantee and demonstrating its effectiveness in sequence forecasting tasks beyond static classification.

Findings

TEA offers a regularization effect through auxiliary reconstruction.

TEA improves performance in multivariate sequence forecasting.

Theoretical stability guarantees support TEA's generalization ability.

Abstract

Autoencoder-based learning has emerged as a staple for disciplining representations in unsupervised and semi-supervised settings. This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets---encoding the prior that variations in targets are driven by a compact set of underlying factors. As our theoretical contribution, we provide a guarantee of generalization for linear TEAs by demonstrating uniform stability, interpreting the benefit of the auxiliary reconstruction task as a form of regularization. As our empirical contribution, we extend validation of this approach beyond existing static classification applications to multivariate sequence forecasting, verifying their advantage on both linear and nonlinear recurrent architectures---thereby underscoring the further generality of this framework beyond feedforward instantiations.