Fanoos: Multi-Resolution, Multi-Strength, Interactive Explanations for Learned Systems

TL;DR

Fanoos is a framework that combines formal verification, heuristic search, and user interaction to generate customizable, multi-resolution explanations for learned systems, improving interpretability for diverse stakeholders.

Contribution

It introduces a novel approach to explanation that allows dynamic adjustment of explanation granularity and fidelity, addressing limitations of existing methods.

Findings

Fanoos effectively produces explanations at various abstraction levels.

It adapts explanations based on user requests in real-time.

Demonstrated on a neural controller and CPU usage model.

Abstract

Machine learning is becoming increasingly important to control the behavior of safety and financially critical components in sophisticated environments, where the inability to understand learned components in general, and neural nets in particular, poses serious obstacles to their adoption. Explainability and interpretability methods for learned systems have gained considerable academic attention, but the focus of current approaches on only one aspect of explanation, at a fixed level of abstraction, and limited if any formal guarantees, prevents those explanations from being digestible by the relevant stakeholders (e.g., end users, certification authorities, engineers) with their diverse backgrounds and situation-specific needs. We introduce Fanoos, a framework for combining formal verification techniques, heuristic search, and user interaction to explore explanations at the desired level of granularity and fidelity. We demonstrate the ability of Fanoos to produce and adjust the abstractness of explanations in response to user requests on a learned controller for an inverted double pendulum and on a learned CPU usage model.