# Bivariate Beta-LSTM

## TL;DR

This paper introduces a novel bivariate Beta distribution-based gate structure for LSTM, enabling probabilistic modeling of gates, which enhances flexibility and gradient flow, improving performance across various tasks.

## Contribution

It proposes a new probabilistic gate structure using bivariate Beta distribution for LSTM, allowing for flexible modeling and better gradient flow, which was not addressed in prior models.

## Key findings

- Higher gradient upper bound than sigmoid gates
- Improved performance in sentence and image classification
- Effective in music modeling and caption generation

## Abstract

Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing multi-modality or skewness. Besides, the previous models lack modeling on the correlation between the gates, which would be a new method to adopt inductive bias for a relationship between previous and current input. This paper proposes a new gate structure with the bivariate Beta distribution. The proposed gate structure enables probabilistic modeling on the gates within the LSTM cell so that the modelers can customize the cell state flow with priors and distributions. Moreover, we theoretically show the higher upper bound of the gradient compared to the sigmoid function, and we empirically observed that the bivariate Beta distribution gate structure provides higher gradient values in training. We demonstrate the effectiveness of bivariate Beta gate structure on the sentence classification, image classification, polyphonic music modeling, and image caption generation.