Exploring Weight Balancing on Long-Tailed Recognition Problem
Naoya Hasegawa, Issei Sato
TL;DR
This paper analyzes the effectiveness of weight balancing in long-tailed recognition, revealing its underlying mechanisms related to neural collapse and Fisher's discriminant ratio, and proposes a simplified training method.
Contribution
It provides a theoretical understanding of weight balancing's success and introduces a simplified one-stage training approach for long-tailed recognition.
Findings
Weight balancing increases Fisher's discriminant ratio.
Implicit logit adjustment improves class balance.
Simplified one-stage training maintains high accuracy.
Abstract
Recognition problems in long-tailed data, in which the sample size per class is heavily skewed, have gained importance because the distribution of the sample size per class in a dataset is generally exponential unless the sample size is intentionally adjusted. Various methods have been devised to address these problems.Recently, weight balancing, which combines well-known classical regularization techniques with two-stage training, has been proposed. Despite its simplicity, it is known for its high performance compared with existing methods devised in various ways. However, there is a lack of understanding as to why this method is effective for long-tailed data. In this study, we analyze weight balancing by focusing on neural collapse and the cone effect at each training stage and found that it can be decomposed into an increase in Fisher's discriminant ratio of the feature extractor…
Peer Reviews
Decision·ICLR 2024 poster
1. The problem of imbalanced classification is undeniably a highly practical and crucial research issue in the field of machine learning. 2. The authors provided an analysis of weight balancing to a certain extent and offered insightful perspectives on the topic.
1. This paper appears to resemble an appendix on Weight Balancing to some extent and the technical innovation is rather limited. 2. Given that Weight Balancing is not the best-performing method in the field of imbalanced learning, the significance of this paper in the field remains debatable. 3. Considering that Weight Balancing involves implicit constraints at the parameter level (compared to direct correction in other long-tail classification methods), its extension to address broader distribu
1. The problem formulation is well motivated and sensible. Developing a theory for weight balancing in LTR has been under-researched in the literature. This work makes a timely contribution to this important topic. 2. The technical contributions in Sec. 4 and 5 are solid. Both theorems 1 and 2 are well presented and their rigorous proof have been included in the Appendix. The generalized result of Theorem 2 (Theorem 3 in Appendix) is commendable. 3. In addition to the theoretical analysis, this
1. The difference between weight balancing (WB) and weight decay (WD) needs to be make clearer. Sec. 3 only reviews WB and overlooks WD. Historically, WD was proposed much earlier than WB. It will be a good idea to include some review of WD in Sec. 3, I think. Note that WD is already present in Table 1 on page 4 (right after Sec. 3). 2. For those who are less familiar with two-stage training of LTR, it might be a good idea to include a concise review of two-stage training methods in the Appendix
1. This paper provides an in-depth analysis of the reasons behind the success of WD in long-tail scenarios, demonstrating thoughtful insights. From the perspective of neural collapse and the cone effect, it explains the WD well. 2. This paper has a well-organized structure which makes it easy for readers to understand the research. 3. Extensive experimental results confirm the validity of the analysis.
1. The paper only discusses the related work of NC and WD but the related work of the long-tail is also necessary. 2. Some concerns which I will mention in the following section.
Code & Models
Videos
Taxonomy
TopicsNeural Networks and Applications · Machine Learning and ELM · Domain Adaptation and Few-Shot Learning
MethodsWeight Decay