Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment

TL;DR

This paper develops a combined compression methodology for VAE-based out-of-distribution detectors, enabling real-time deployment on embedded systems by reducing memory and inference time with minimal performance loss.

Contribution

It introduces a novel combined approach of quantization, pruning, and knowledge distillation for VAE-based OOD detectors, maintaining detection performance while optimizing for embedded deployment.

Findings

20% reduction in GPU inference time

28% reduction in CPU inference time

AUROC within 5% of baseline

Abstract

Out-of-distribution (OOD) detectors can act as safety monitors in embedded cyber-physical systems by identifying samples outside a machine learning model's training distribution to prevent potentially unsafe actions. However, OOD detectors are often implemented using deep neural networks, which makes it difficult to meet real-time deadlines on embedded systems with memory and power constraints. We consider the class of variational autoencoder (VAE) based OOD detectors where OOD detection is performed in latent space, and apply quantization, pruning, and knowledge distillation. These techniques have been explored for other deep models, but no work has considered their combined effect on latent space OOD detection. While these techniques increase the VAE's test loss, this does not correspond to a proportional decrease in OOD detection performance and we leverage this to develop lean OOD detectors capable of real-time inference on embedded CPUs and GPUs. We propose a design methodology that combines all three compression techniques and yields a significant decrease in memory and execution time while maintaining AUROC for a given OOD detector. We demonstrate this methodology with two existing OOD detectors on a Jetson Nano and reduce GPU and CPU inference time by 20% and 28% respectively while keeping AUROC within 5% of the baseline.