Data Complexity-aware Deep Model Performance Forecasting

TL;DR

This paper introduces a lightweight, two-stage framework for predicting deep model performance before training, leveraging dataset properties and model details to improve model selection and data quality assessment.

Contribution

The proposed method offers a generalizable, efficient approach to forecast model performance pre-training, aiding in architecture selection and dataset evaluation.

Findings

Dataset variance can guide model choice.

The framework predicts performance accurately across datasets.

It helps identify data issues before training begins.

Abstract

Deep learning models are widely used across computer vision and other domains. When working on the model induction, selecting the right architecture for a given dataset often relies on repetitive trial-and-error procedures. This procedure is time-consuming, resource-intensive, and difficult to automate. While previous work has explored performance prediction using partial training or complex simulations, these methods often require significant computational overhead or lack generalizability. In this work, we propose an alternative approach: a lightweight, two-stage framework that can estimate model performance before training given the understanding of the dataset and the focused deep model structures. The first stage predicts a baseline based on the analysis of some measurable properties of the dataset, while the second stage adjusts the estimation with additional information on the model's architectural and hyperparameter details. The setup allows the framework to generalize across datasets and model types. Moreover, we find that some of the underlying features used for prediction - such as dataset variance - can offer practical guidance for model selection, and can serve as early indicators of data quality. As a result, the framework can be used not only to forecast model performance, but also to guide architecture choices, inform necessary preprocessing procedures, and detect potentially problematic datasets before training begins.