On the Optimization of Margin Distribution

TL;DR

This paper explores the theoretical and practical benefits of optimizing margin distribution in learning algorithms, introducing a new generalization bound and an effective method called MSVMAv that improves performance.

Contribution

It provides a new theoretical generalization error bound based on margin distribution and proposes MSVMAv, a novel approach to optimize margin distribution for better accuracy.

Findings

MSVMAv outperforms existing methods in experiments.

Theoretical bounds relate margin distribution to generalization error.

Optimizing margin distribution improves learning performance.

Abstract

Margin has played an important role on the design and analysis of learning algorithms during the past years, mostly working with the maximization of the minimum margin. Recent years have witnessed the increasing empirical studies on the optimization of margin distribution according to different statistics such as medium margin, average margin, margin variance, etc., whereas there is a relative paucity of theoretical understanding. In this work, we take one step on this direction by providing a new generalization error bound, which is heavily relevant to margin distribution by incorporating ingredients such as average margin and semi-variance, a new margin statistics for the characterization of margin distribution. Inspired by the theoretical findings, we propose the MSVMAv, an efficient approach to achieve better performance by optimizing margin distribution in terms of its empirical average margin and semi-variance. We finally conduct extensive experiments to show the superiority of the proposed MSVMAv approach.