Semantically Redundant Training Data Removal and Deep Model Classification Performance: A Study with Chest X-rays

TL;DR

This study introduces an entropy-based method to identify and remove semantically redundant chest X-ray images, improving deep learning model performance by focusing on more informative training data.

Contribution

It proposes a novel entropy-based sample scoring approach for selecting informative training data, demonstrating improved model performance on chest X-ray classification tasks.

Findings

Model trained on selected data outperforms full dataset training in recall.

Removing redundant data enhances generalizability to external datasets.

Entropy-based selection improves training efficiency and effectiveness.

Abstract

Deep learning (DL) has demonstrated its innate capacity to independently learn hierarchical features from complex and multi-dimensional data. A common understanding is that its performance scales up with the amount of training data. Another data attribute is the inherent variety. It follows, therefore, that semantic redundancy, which is the presence of similar or repetitive information, would tend to lower performance and limit generalizability to unseen data. In medical imaging data, semantic redundancy can occur due to the presence of multiple images that have highly similar presentations for the disease of interest. Further, the common use of augmentation methods to generate variety in DL training may be limiting performance when applied to semantically redundant data. We propose an entropy-based sample scoring approach to identify and remove semantically redundant training data. We demonstrate using the publicly available NIH chest X-ray dataset that the model trained on the resulting informative subset of training data significantly outperforms the model trained on the full training set, during both internal (recall: 0.7164 vs 0.6597, p<0.05) and external testing (recall: 0.3185 vs 0.2589, p<0.05). Our findings emphasize the importance of information-oriented training sample selection as opposed to the conventional practice of using all available training data.