Autoencoder Trees

TL;DR

This paper introduces autoencoder trees, a novel model where decision trees are used for encoding and decoding, trained with stochastic gradient descent, capturing hierarchical data representations effectively.

Contribution

It presents a new autoencoder architecture using soft decision trees for encoding and decoding, enabling hierarchical data representation and efficient training.

Findings

Autoencoder trees achieve good reconstruction error on digit and news data.

They capture hierarchical and local structures in the data.

The model is trained effectively with stochastic gradient descent.

Abstract

We discuss an autoencoder model in which the encoding and decoding functions are implemented by decision trees. We use the soft decision tree where internal nodes realize soft multivariate splits given by a gating function and the overall output is the average of all leaves weighted by the gating values on their path. The encoder tree takes the input and generates a lower dimensional representation in the leaves and the decoder tree takes this and reconstructs the original input. Exploiting the continuity of the trees, autoencoder trees are trained with stochastic gradient descent. On handwritten digit and news data, we see that the autoencoder trees yield good reconstruction error compared to traditional autoencoder perceptrons. We also see that the autoencoder tree captures hierarchical representations at different granularities of the data on its different levels and the leaves capture the localities in the input space.