Spotting Hallucinations in Inverse Problems with Data-Driven Priors

TL;DR

This paper introduces a scalable, efficient method to identify regions in inverse problem solutions that are prone to hallucinations caused by data-driven priors, enhancing trustworthiness of neural network reconstructions.

Contribution

It presents a novel diagnostic technique based on Fisher information to detect hallucination-prone regions in inverse problem solutions with deep neural networks.

Findings

Method efficiently flags hallucination-prone regions in high-dimensional inverse problems.

The approach scales linearly with parameters, suitable for large datasets.

Enables users to assess the robustness of measurements in reconstructed solutions.

Abstract

Hallucinations are an inescapable consequence of solving inverse problems with deep neural networks. The expressiveness of recent generative models is the reason why they can yield results far superior to conventional regularizers; it can also lead to realistic-looking but incorrect features, potentially undermining the trust in important aspects of the reconstruction. We present a practical and computationally efficient method to determine, which regions in the solutions of inverse problems with data-driven priors are prone to hallucinations. By computing the diagonal elements of the Fisher information matrix of the likelihood and the data-driven prior separately, we can flag regions where the information is prior-dominated. Our diagnostic can directly be compared to the reconstructed solutions and enables users to decide if measurements in such regions are robust for their application. Our method scales linearly with the number of parameters and is thus applicable in high-dimensional settings, allowing it to be rolled out broadly for the large-volume data products of future wide-field surveys.