Density Fixing: Simple yet Effective Regularization Method based on the Class Prior

TL;DR

Density fixing is a simple regularization technique that improves model generalization by aligning predictions with class priors, applicable to both supervised and semi-supervised learning, supported by theoretical analysis and benchmark results.

Contribution

The paper introduces a novel density-fixing regularization method based on class priors, with theoretical justification and demonstrated effectiveness on benchmark datasets.

Findings

Improves generalization by aligning with class priors

Theoretically justified with asymptotic analysis

Effective on multiple benchmark datasets

Abstract

Machine learning models suffer from overfitting, which is caused by a lack of labeled data. To tackle this problem, we proposed a framework of regularization methods, called density-fixing, that can be used commonly for supervised and semi-supervised learning. Our proposed regularization method improves the generalization performance by forcing the model to approximate the class's prior distribution or the frequency of occurrence. This regularization term is naturally derived from the formula of maximum likelihood estimation and is theoretically justified. We further provide the several theoretical analyses of the proposed method including asymptotic behavior. Our experimental results on multiple benchmark datasets are sufficient to support our argument, and we suggest that this simple and effective regularization method is useful in real-world machine learning problems.