Regression analysis with missing data and unknown colored noise: application to the MICROSCOPE space mission

TL;DR

This paper introduces a regression method that effectively handles missing data and colored noise in physical measurements, demonstrated on MICROSCOPE space mission data, significantly improving parameter estimation precision.

Contribution

The paper presents a novel AR-based generalized least squares regression technique that mitigates the effects of data gaps and unknown colored noise, approaching optimal estimation accuracy.

Findings

Method reduces uncertainty by a factor of 60 compared to ordinary least squares.

Approach performs well with various gap patterns and unknown noise PSD.

Results meet the precision requirements of the MICROSCOPE mission.

Abstract

The analysis of physical measurements often copes with highly correlated noises and interruptions caused by outliers, saturation events or transmission losses. We assess the impact of missing data on the performance of linear regression analysis involving the fit of modeled or measured time series. We show that data gaps can significantly alter the precision of the regression parameter estimation in the presence of colored noise, due to the frequency leakage of the noise power. We present a regression method which cancels this effect and estimates the parameters of interest with a precision comparable to the complete data case, even if the noise power spectral density (PSD) is not known a priori. The method is based on an autoregressive (AR) fit of the noise, which allows us to build an approximate generalized least squares estimator approaching the minimal variance bound. The method, which can be applied to any similar data processing, is tested on simulated measurements of the MICROSCOPE space mission, whose goal is to test the Weak Equivalence Principle (WEP) with a precision of $10^{-15}$. In this particular context the signal of interest is the WEP violation signal expected to be found around a well defined frequency. We test our method with different gap patterns and noise of known PSD and find that the results agree with the mission requirements, decreasing the uncertainty by a factor 60 with respect to ordinary least squares methods. We show that it also provides a test of significance to assess the uncertainty of the measurement.