Monitizer: Automating Design and Evaluation of Neural Network Monitors
Muqsit Azeem, Marta Grobelna, Sudeep Kanav, Jan Kretinsky and, Stefanie Mohr, Sabine Rieder

TL;DR
This paper introduces Monitizer, a comprehensive tool that automates the application, optimization, and comparison of neural network monitors to improve out-of-distribution detection in safety-critical systems.
Contribution
The paper presents Monitizer, a novel tool that streamlines the deployment, tuning, and evaluation of neural network monitors, addressing scalability and reproducibility challenges.
Findings
Effective comparison of different monitoring approaches
Demonstrated usability across multiple use cases
Facilitated development of new monitoring methods
Abstract
The behavior of neural networks (NNs) on previously unseen types of data (out-of-distribution or OOD) is typically unpredictable. This can be dangerous if the network's output is used for decision-making in a safety-critical system. Hence, detecting that an input is OOD is crucial for the safe application of the NN. Verification approaches do not scale to practical NNs, making runtime monitoring more appealing for practical use. While various monitors have been suggested recently, their optimization for a given problem, as well as comparison with each other and reproduction of results, remain challenging. We present a tool for users and developers of NN monitors. It allows for (i) application of various types of monitors from the literature to a given input NN, (ii) optimization of the monitor's hyperparameters, and (iii) experimental evaluation and comparison to other approaches.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications
