Evaluation Metrics for DNNs Compression

TL;DR

This paper reviews and standardizes evaluation metrics for neural network compression, introduces two new metrics, and demonstrates their application across different hardware platforms and tasks.

Contribution

It provides a standardized framework, NetZIP, for evaluating neural network compression techniques and introduces two novel metrics to address existing evaluation gaps.

Findings

NetZIP enables consistent comparison of compression methods.

The new metrics CHATS and OCS fill evaluation gaps.

Case studies validate the framework across hardware and tasks.

Abstract

There is a lot of ongoing research effort into developing different techniques for neural networks compression. However, the community lacks standardised evaluation metrics, which are key to identifying the most suitable compression technique for different applications. This paper reviews existing neural network compression evaluation metrics and implements them into a standardisation framework called NetZIP. We introduce two novel metrics to cover existing gaps of evaluation in the literature: 1) Compression and Hardware Agnostic Theoretical Speed (CHATS) and 2) Overall Compression Success (OCS). We demonstrate the use of NetZIP using two case studies on two different hardware platforms (a PC and a Raspberry Pi 4) focusing on object classification and object detection.