Practical Insights of Repairing Model Problems on Image Classification

TL;DR

This paper analyzes methods to mitigate model degradation in image classification, emphasizing the importance of choosing appropriate techniques based on dataset characteristics and system lifecycle considerations.

Contribution

It provides practical insights and a comparative analysis of methods for reducing degradation, tailored to industrial use cases and dataset arrangements.

Findings

Degradation can occur due to additional training in real-world scenarios.

Choosing the right method depends on dataset availability and AI system lifecycle.

Trade-offs exist between accuracy and degradation prevention.

Abstract

Additional training of a deep learning model can cause negative effects on the results, turning an initially positive sample into a negative one (degradation). Such degradation is possible in real-world use cases due to the diversity of sample characteristics. That is, a set of samples is a mixture of critical ones which should not be missed and less important ones. Therefore, we cannot understand the performance by accuracy alone. While existing research aims to prevent a model degradation, insights into the related methods are needed to grasp their benefits and limitations. In this talk, we will present implications derived from a comparison of methods for reducing degradation. Especially, we formulated use cases for industrial settings in terms of arrangements of a data set. The results imply that a practitioner should care about better method continuously considering dataset availability and life cycle of an AI system because of a trade-off between accuracy and preventing degradation.