Analyzing Compression Techniques for Computer Vision

TL;DR

This paper evaluates the effectiveness of knowledge distillation, pruning, and quantization, both individually and combined, for compressing deep networks in computer vision tasks using MNIST and CIFAR-10 datasets.

Contribution

It systematically compares basic and combined compression techniques for small-scale recognition tasks in computer vision.

Findings

Knowledge distillation, pruning, and quantization each improve model efficiency.

Combining techniques yields better compression than individual methods.

Results provide insights into effective compression strategies for small datasets.

Abstract

Compressing deep networks is highly desirable for practical use-cases in computer vision applications. Several techniques have been explored in the literature, and research has been done in finding efficient strategies for combining them. For this project, we aimed to explore three different basic compression techniques - knowledge distillation, pruning, and quantization for small-scale recognition tasks. Along with the basic methods, we also test the efficacy of combining them in a sequential manner. We analyze them using MNIST and CIFAR-10 datasets and present the results along with few observations inferred from them.