Improved Auto-Encoding using Deterministic Projected Belief Networks

TL;DR

This paper introduces a deterministic projected belief network (D-PBN) auto-encoder that leverages trainable complex activation functions to improve data reconstruction, outperforming standard auto-encoders like variational auto-encoders.

Contribution

The paper presents a novel D-PBN auto-encoder framework that effectively utilizes invertible trainable activation functions for enhanced data analysis and reconstruction.

Findings

D-PBN auto-encoder outperforms standard auto-encoders.

Utilizes invertible trainable activation functions.

Achieves better data distribution restoration.

Abstract

In this paper, we exploit the unique properties of a deterministic projected belief network (D-PBN) to take full advantage of trainable compound activation functions (TCAs). A D-PBN is a type of auto-encoder that operates by "backing up" through a feed-forward neural network. TCAs are activation functions with complex monotonic-increasing shapes that change the distribution of the data so that the linear transformation that follows is more effective. Because a D-PBN operates by "backing up", the TCAs are inverted in the reconstruction process, restoring the original distribution of the data, thus taking advantage of a given TCA in both analysis and reconstruction. In this paper, we show that a D-PBN auto-encoder with TCAs can significantly out-perform standard auto-encoders including variational auto-encoders.