A Priori Generalizability Estimate for a CNN

TL;DR

This paper introduces a diagnostic method using singular value decompositions of CNNs to predict which images the model may perform poorly on, aiding in understanding and improving model generalizability.

Contribution

It develops a novel approach using truncated SVD of CNNs and defines new metrics to estimate model performance and identify class imbalance issues.

Findings

Right and Left Projection Ratios identify class imbalance.

Right Projection Ratio correlates with segmentation performance.

Unlabeled data can be used to estimate model reliability.

Abstract

We formulate truncated singular value decompositions of entire convolutional neural networks. We demonstrate the computed left and right singular vectors are useful in identifying which images the convolutional neural network is likely to perform poorly on. To create this diagnostic tool, we define two metrics: the Right Projection Ratio and the Left Projection Ratio. The Right (Left) Projection Ratio evaluates the fidelity of the projection of an image (label) onto the computed right (left) singular vectors. We observe that both ratios are able to identify the presence of class imbalance for an image classification problem. Additionally, the Right Projection Ratio, which only requires unlabeled data, is found to be correlated to the model's performance when applied to image segmentation. This suggests the Right Projection Ratio could be a useful metric to estimate how likely the model is to perform well on a sample.