Test Automation with Grad-CAM Heatmaps -- A Future Pipe Segment in MLOps for Vision AI?

TL;DR

This paper explores using Grad-CAM heatmaps to improve explainability in vision AI models, proposing their integration into MLOps pipelines to detect biases and ensure trustworthy AI in safety-critical applications.

Contribution

It introduces automated heatmap analysis as a new MLOps component for model explainability and bias detection in vision AI systems.

Findings

Grad-CAM heatmaps enhance model explainability.

Automated heatmap analysis can detect biased activations.

Proposed pipeline supports compliance with Trustworthy AI requirements.

Abstract

Machine Learning (ML) is a fundamental part of modern perception systems. In the last decade, the performance of computer vision using trained deep neural networks has outperformed previous approaches based on careful feature engineering. However, the opaqueness of large ML models is a substantial impediment for critical applications such as in the automotive context. As a remedy, Gradient-weighted Class Activation Mapping (Grad-CAM) has been proposed to provide visual explanations of model internals. In this paper, we demonstrate how Grad-CAM heatmaps can be used to increase the explainability of an image recognition model trained for a pedestrian underpass. We argue how the heatmaps support compliance to the EU's seven key requirements for Trustworthy AI. Finally, we propose adding automated heatmap analysis as a pipe segment in an MLOps pipeline. We believe that such a building block can be used to automatically detect if a trained ML-model is activated based on invalid pixels in test images, suggesting biased models.