Theoretically informed selection of latent activation in autoencoder based recommender systems

TL;DR

This paper provides a theoretical framework for selecting activation functions in autoencoder-based recommender systems, highlighting sigmoid-like activations as optimal for preserving key properties and improving recommendation accuracy.

Contribution

It identifies three essential properties for autoencoder encoders in recommender systems and shows that sigmoid-like activations are theoretically best suited to satisfy these properties.

Findings

ReLU and tanh cannot satisfy all key properties simultaneously.

Sigmoid-like activations fulfill the identified mathematical properties.

The approach guides systematic hyperparameter selection for better model efficiency.

Abstract

Autoencoders may lend themselves to the design of more accurate and computationally efficient recommender systems by distilling sparse high-dimensional data into dense lower-dimensional latent representations. However, designing these systems remains challenging due to the lack of theoretical guidance. This work addresses this by identifying three key mathematical properties that the encoder in an autoencoder should exhibit to improve recommendation accuracy: (1) dimensionality reduction, (2) preservation of similarity ordering in dot product comparisons, and (3) preservation of non-zero vectors. Through theoretical analysis, we demonstrate that common activation functions, such as ReLU and tanh, cannot fulfill these properties jointly within a generalizable framework. In contrast, sigmoid-like activations emerge as suitable choices for latent activations. This theoretically informed approach offers a more systematic method for hyperparameter selection, enhancing the efficiency of model design.