A Simple but Effective Closed-form Solution for Extreme Multi-label Learning

TL;DR

This paper introduces a simple, hyperparameter-efficient ridge regression-based method for extreme multi-label learning that achieves competitive performance and improves low-frequency label prediction.

Contribution

It presents the first application of ridge regression to XML, providing a closed-form solution with a single hyperparameter, simplifying implementation and tuning.

Findings

Achieves comparable or better performance than complex models.

Significantly improves low-frequency label prediction.

Simplifies XML modeling with a closed-form, hyperparameter-efficient approach.

Abstract

Extreme multi-label learning (XML) is a task of assigning multiple labels from an extremely large set of labels to each data instance. Many current high-performance XML models are composed of a lot of hyperparameters, which complicates the tuning process. Additionally, the models themselves are adapted specifically to XML, which complicates their reimplementation. To remedy this problem, we propose a simple method based on ridge regression for XML. The proposed method not only has a closed-form solution but also is composed of a single hyperparameter. Since there are no precedents on applying ridge regression to XML, this paper verified the performance of the method by using various XML benchmark datasets. Furthermore, we enhanced the prediction of low-frequency labels in XML, which hold informative content. This prediction is essential yet challenging because of the limited amount of data. Here, we employed a simple frequency-based weighting. This approach greatly simplifies the process compared with existing techniques. Experimental results revealed that it can achieve levels of performance comparable to, or even exceeding, those of models with numerous hyperparameters. Additionally, we found that the frequency-based weighting significantly improved the predictive performance for low-frequency labels, while requiring almost no changes in implementation. The source code for the proposed method is available on github at https://github.com/cars1015/XML-ridge.