Are Sparse Neural Networks Better Hard Sample Learners?

TL;DR

This paper investigates the effectiveness of sparse neural networks in learning hard, noisy, and intricate samples, revealing they can match or outperform dense models at certain sparsity levels, especially with limited data.

Contribution

It provides the first extensive analysis of sparse neural networks' performance on challenging samples, highlighting the importance of layer-wise density ratios and training strategies.

Findings

SNNs can match or surpass dense models on hard samples at specific sparsity levels.

Layer-wise density ratios significantly influence SNN performance, especially without pre-training.

SNNs are effective in data-limited scenarios for complex and noisy data.

Abstract

While deep learning has demonstrated impressive progress, it remains a daunting challenge to learn from hard samples as these samples are usually noisy and intricate. These hard samples play a crucial role in the optimal performance of deep neural networks. Most research on Sparse Neural Networks (SNNs) has focused on standard training data, leaving gaps in understanding their effectiveness on complex and challenging data. This paper's extensive investigation across scenarios reveals that most SNNs trained on challenging samples can often match or surpass dense models in accuracy at certain sparsity levels, especially with limited data. We observe that layer-wise density ratios tend to play an important role in SNN performance, particularly for methods that train from scratch without pre-trained initialization. These insights enhance our understanding of SNNs' behavior and potential for efficient learning approaches in data-centric AI. Our code is publicly available at: \url{https://github.com/QiaoXiao7282/hard_sample_learners}.