Gappy local conformal auto-encoders for heterogeneous data fusion: in praise of rigidity

TL;DR

This paper introduces a neural network-based pipeline that fuses heterogeneous, partial sensor data into a consistent latent space by leveraging multiple measurements to estimate local distortions, demonstrated across various complex scenarios.

Contribution

It presents a novel end-to-end auto-encoder architecture that effectively fuses heterogeneous data sources using local perturbation measurements to achieve global consistency.

Findings

Successfully fused data in 2D, Wi-Fi localization, and PDE observations.

Achieved globally consistent latent representations from partial, heterogeneous data.

Validated approach across multiple complex real-world examples.

Abstract

Fusing measurements from multiple, heterogeneous, partial sources, observing a common object or process, poses challenges due to the increasing availability of numbers and types of sensors. In this work we propose, implement and validate an end-to-end computational pipeline in the form of a multiple-auto-encoder neural network architecture for this task. The inputs to the pipeline are several sets of partial observations, and the result is a globally consistent latent space, harmonizing (rigidifying, fusing) all measurements. The key enabler is the availability of multiple slightly perturbed measurements of each instance:, local measurement, "bursts", that allows us to estimate the local distortion induced by each instrument. We demonstrate the approach in a sequence of examples, starting with simple two-dimensional data sets and proceeding to a Wi-Fi localization problem and to the solution of a "dynamical puzzle" arising in spatio-temporal observations of the solutions of Partial Differential Equations.