# Self-Balanced Dropout

## TL;DR

This paper introduces Self-Balanced Dropout, a new method that addresses residual co-adaptation issues in dropout by using a trainable variable to improve model generalization across various tasks.

## Contribution

It provides a theoretical proof of co-adaptation persistence after dropout and proposes a novel, trainable dropout mechanism to mitigate this problem.

## Key findings

- Effective in reducing co-adaptation
- Significantly improves performance across tasks
- Works with simple and complex models

## Abstract

Dropout is known as an effective way to reduce overfitting via preventing co-adaptations of units. In this paper, we theoretically prove that the co-adaptation problem still exists after using dropout due to the correlations among the inputs. Based on the proof, we further propose Self-Balanced Dropout, a novel dropout method which uses a trainable variable to balance the influence of the input correlation on parameter update. We evaluate Self-Balanced Dropout on a range of tasks with both simple and complex models. The experimental results show that the mechanism can effectively solve the co-adaption problem to some extent and significantly improve the performance on all tasks.