Drift detection on feature attributions for monitoring visual reinforcement learning models in maritime port surveillance

TL;DR

This paper introduces FADMON, a new method to monitor AI models in port surveillance by detecting changes in how the models interpret data, improving security and understanding.

Contribution

FADMON introduces XAI-driven drift detection on feature attributions for image-based models, enabling concept-level monitoring of DRL systems.

Findings

FADMON consistently detects drift in all drifted scenarios with mean p-values of 0.000 across 30 repetitions.

FADMON provides semi-supervised explainable model monitoring, detecting changes in model interpretation rather than just data.

The method achieves comparable performance to established drift detection techniques while offering interpretability.

Abstract

Maritime activity is expanding globally, increasing the demand for robust port security systems capable of detecting illegal trafficking. Due to the growing sophistication of smuggling methods, law enforcement agencies require advanced surveillance and prevention technologies such as those developed in the SMAUG project. In this context, initiatives such as the SMAUG project aim to deliver integrated surveillance capabilities coordinated by a high-level deep reinforcement learning (DRL) decision-making system that operates on image-based environmental representations. Despite their effectiveness, DRL models are closed-boxes, complicating continuous model monitoring (CMM). Conventional drift detection captures shifts in input or output distributions yet often fails to explain underlying problems. Explainable AI (XAI) techniques can provide a complementary approach with insights into the agent’s inner workings, enabling monitoring of the concept rather than just the data.

We propose FADMON, an XAI-driven concept drift detection method for image-based models. FADMON performs statistical drift tests on feature attributions to detect deviations in learned policies. We demonstrate how FADMON can enhance CMM with a three-stage model monitoring architecture that enables semi-supervised explainable model monitoring. We validate our approach with SMAUG’s decision-making DRL model on a simulated maritime port surveillance environment under multiple unforeseen scenarios.

FADMON consistently flags drift on all drifted scenarios with mean p-values of 0.000 with no variance trough 30 repetitions, with lower mean p-values (0.553±0.215) on non-drifted scenarios with respect to other established drift detection methodologies such as prior probability shift detection (0.65 ± 0.000), though well above the standard 0.05 threshold.

FADMON can add an explainability layer to the monitoring system while also supporting detection of changes in the underlying interpretation of the input data by the model, monitoring the concept rather than the data, while matching established drift detection methods metrics-wise.

This article shows how it is possible to monitor an AI model interpretation of the data it is being fed, rather than monitoring only the data itself, to improve our capability to understand how an AI model is behaving while it works. This allows us to detect changes in the way it is behaving, as well as understand these changes better, in order to detect more quickly when the model is not behaving correctly and fix it. In this article, we apply this approach to a maritime port surveillance environment, testing it under different simulated scenarios, such as overabundance of obstacles in the port or malicious attacks on the vessels, to see if the method meets its potential. We conclude that it does, with certain caveats such as the need of stronger hardware, resulting in a good balance between detection of irregular AI model behavior and interpretability of said behavior.

Figures