Optimizing Convolutional Neural Network Architecture

TL;DR

This paper introduces OCNNA, a novel CNN optimization method using pruning and knowledge distillation, which improves deployment efficiency on resource-limited devices while maintaining high accuracy.

Contribution

The paper presents a new CNN optimization technique, OCNNA, that outperforms over 20 existing algorithms in reducing complexity and energy consumption.

Findings

OCNNA achieves competitive accuracy with fewer parameters.

It outperforms 20+ existing CNN simplification algorithms.

Effective on datasets like CIFAR-10, CIFAR-100, and ImageNet.

Abstract

Convolutional Neural Networks (CNN) are widely used to face challenging tasks like speech recognition, natural language processing or computer vision. As CNN architectures get larger and more complex, their computational requirements increase, incurring significant energetic costs and challenging their deployment on resource-restricted devices. In this paper, we propose Optimizing Convolutional Neural Network Architecture (OCNNA), a novel CNN optimization and construction method based on pruning and knowledge distillation designed to establish the importance of convolutional layers. The proposal has been evaluated though a thorough empirical study including the best known datasets (CIFAR-10, CIFAR-100 and Imagenet) and CNN architectures (VGG-16, ResNet-50, DenseNet-40 and MobileNet), setting Accuracy Drop and Remaining Parameters Ratio as objective metrics to compare the performance of OCNNA against the other state-of-art approaches. Our method has been compared with more than 20 convolutional neural network simplification algorithms obtaining outstanding results. As a result, OCNNA is a competitive CNN constructing method which could ease the deployment of neural networks into IoT or resource-limited devices.