An Analysis of Dropout for Matrix Factorization

TL;DR

This paper provides a theoretical analysis of dropout in matrix factorization, revealing its equivalence to a deterministic regularization model and highlighting issues with fixed dropout rates affecting factorization size.

Contribution

It introduces a theoretical framework connecting dropout to a regularizer in matrix factorization and proposes increasing dropout rates with factor size to address size control issues.

Findings

Dropout is equivalent to a deterministic regularizer involving column norms.

Using a fixed dropout rate can lead to trivial solutions with larger factorization.

Increasing dropout rate with factor size mitigates the trivial solution problem.

Abstract

Dropout is a simple yet effective algorithm for regularizing neural networks by randomly dropping out units through Bernoulli multiplicative noise, and for some restricted problem classes, such as linear or logistic regression, several theoretical studies have demonstrated the equivalence between dropout and a fully deterministic optimization problem with data-dependent Tikhonov regularization. This work presents a theoretical analysis of dropout for matrix factorization, where Bernoulli random variables are used to drop a factor, thereby attempting to control the size of the factorization. While recent work has demonstrated the empirical effectiveness of dropout for matrix factorization, a theoretical understanding of the regularization properties of dropout in this context remains elusive. This work demonstrates the equivalence between dropout and a fully deterministic model for matrix factorization in which the factors are regularized by the sum of the product of the norms of the columns. While the resulting regularizer is closely related to a variational form of the nuclear norm, suggesting that dropout may limit the size of the factorization, we show that it is possible to trivially lower the objective value by doubling the size of the factorization. We show that this problem is caused by the use of a fixed dropout rate, which motivates the use of a rate that increases with the size of the factorization. Synthetic experiments validate our theoretical findings.