Examining convolutional feature extraction using Maximum Entropy (ME) and Signal-to-Noise Ratio (SNR) for image classification

TL;DR

This paper investigates how the feature extraction abilities of CNNs are influenced by input data quality, using ME and SNR measures on datasets like MNIST and CIFAR-10, highlighting the importance of tailoring CNNs to data complexity.

Contribution

It introduces the use of Maximum Entropy and Signal-to-Noise Ratio measures to evaluate CNN feature extraction capabilities and suggests customizing CNN architecture based on data signal quality.

Findings

CNN performance depends on signal quality in input data.

ME and SNR effectively measure information content in datasets.

Tailoring CNN size to data complexity improves classification accuracy.

Abstract

Convolutional Neural Networks (CNNs) specialize in feature extraction rather than function mapping. In doing so they form complex internal hierarchical feature representations, the complexity of which gradually increases with a corresponding increment in neural network depth. In this paper, we examine the feature extraction capabilities of CNNs using Maximum Entropy (ME) and Signal-to-Noise Ratio (SNR) to validate the idea that, CNN models should be tailored for a given task and complexity of the input data. SNR and ME measures are used as they can accurately determine in the input dataset, the relative amount of signal information to the random noise and the maximum amount of information respectively. We use two well known benchmarking datasets, MNIST and CIFAR-10 to examine the information extraction and abstraction capabilities of CNNs. Through our experiments, we examine convolutional feature extraction and abstraction capabilities in CNNs and show that the classification accuracy or performance of CNNs is greatly dependent on the amount, complexity and quality of the signal information present in the input data. Furthermore, we show the effect of information overflow and underflow on CNN classification accuracies. Our hypothesis is that the feature extraction and abstraction capabilities of convolutional layers are limited and therefore, CNN models should be tailored to the input data by using appropriately sized CNNs based on the SNR and ME measures of the input dataset.