Revealing the Underlying Patterns: Investigating Dataset Similarity, Performance, and Generalization

TL;DR

This paper investigates dataset similarity and model generalization in supervised deep learning, proposing distance metrics and methods to improve performance with minimal additional data, thereby reducing costs.

Contribution

It introduces new dataset-distance metrics and demonstrates how small amounts of unseen data can enhance model generalization and performance estimation.

Findings

Dataset-dataset distances correlate with model performance.

Adding few unseen images improves generalization.

Proposed method reduces training and annotation costs.

Abstract

Supervised deep learning models require significant amount of labeled data to achieve an acceptable performance on a specific task. However, when tested on unseen data, the models may not perform well. Therefore, the models need to be trained with additional and varying labeled data to improve the generalization. In this work, our goal is to understand the models, their performance and generalization. We establish image-image, dataset-dataset, and image-dataset distances to gain insights into the model's behavior. Our proposed distance metric when combined with model performance can help in selecting an appropriate model/architecture from a pool of candidate architectures. We have shown that the generalization of these models can be improved by only adding a small number of unseen images (say 1, 3 or 7) into the training set. Our proposed approach reduces training and annotation costs while providing an estimate of model performance on unseen data in dynamic environments.