MD tree: a model-diagnostic tree grown on loss landscape

TL;DR

This paper introduces MD tree, a novel model diagnosis method that leverages loss landscape metrics to identify failure sources in pre-trained neural networks, outperforming traditional validation-based approaches in various transfer scenarios.

Contribution

The paper proposes MD tree, a new diagnosis approach based on loss landscape analysis, providing more accurate failure source identification without training configuration details.

Findings

MD tree achieves 87.7% accuracy in dataset transfer diagnosis.

MD tree outperforms validation-based methods by 14.88%.

Effective in scale and dataset transfer scenarios.

Abstract

This paper considers "model diagnosis", which we formulate as a classification problem. Given a pre-trained neural network (NN), the goal is to predict the source of failure from a set of failure modes (such as a wrong hyperparameter, inadequate model size, and insufficient data) without knowing the training configuration of the pre-trained NN. The conventional diagnosis approach uses training and validation errors to determine whether the model is underfitting or overfitting. However, we show that rich information about NN performance is encoded in the optimization loss landscape, which provides more actionable insights than validation-based measurements. Therefore, we propose a diagnosis method called MD tree based on loss landscape metrics and experimentally demonstrate its advantage over classical validation-based approaches. We verify the effectiveness of MD tree in multiple practical scenarios: (1) use several models trained on one dataset to diagnose a model trained on another dataset, essentially a few-shot dataset transfer problem; (2) use small models (or models trained with small data) to diagnose big models (or models trained with big data), essentially a scale transfer problem. In a dataset transfer task, MD tree achieves an accuracy of 87.7%, outperforming validation-based approaches by 14.88%. Our code is available at https://github.com/YefanZhou/ModelDiagnosis.