Frank-Wolfe algorithm for learning SVM-type multi-category classifiers

TL;DR

This paper introduces a new Frank-Wolfe based optimization algorithm for multi-category SVMs that converges rapidly and enhances pattern recognition performance across various datasets.

Contribution

The study discovers closed form solutions for the subproblems in the Frank-Wolfe algorithm applied to dual MC-SVM problems, applicable to multiple variants.

Findings

Rapid convergence demonstrated on large datasets

Improved pattern recognition accuracy

Closed form solutions for subproblems in the Frank-Wolfe framework

Abstract

Multi-category support vector machine (MC-SVM) is one of the most popular machine learning algorithms. There are lots of variants of MC-SVM, although different optimization algorithms were developed for different learning machines. In this study, we developed a new optimization algorithm that can be applied to many of MC-SVM variants. The algorithm is based on the Frank-Wolfe framework that requires two subproblems, direction finding and line search, in each iteration. The contribution of this study is the discovery that both subproblems have a closed form solution if the Frank-Wolfe framework is applied to the dual problem. Additionally, the closed form solutions on both for the direction finding and for the line search exist even for the Moreau envelopes of the loss functions. We use several large datasets to demonstrate that the proposed optimization algorithm converges rapidly and thereby improves the pattern recognition performance.