Matrix Factorization-Based Solar Spectral Irradiance Missing Data Imputation with Uncertainty Quantification

TL;DR

This paper introduces a novel matrix factorization method for imputing missing solar spectral irradiance data, incorporating uncertainty quantification, and demonstrates its superior performance over existing approaches.

Contribution

It develops a low-rank matrix factorization technique with autoregressive and spectral regularizations, tailored for SSI data with complex missingness patterns, and provides calibrated uncertainty intervals.

Findings

The proposed method outperforms Gaussian process and linear smoothing in imputation accuracy.

It produces well-calibrated uncertainty intervals for reconstructed SSI.

The approach is computationally efficient and suitable for climate science applications.

Abstract

The solar spectral irradiance (SSI) depicts the spectral distribution of solar energy flux reaching the top of the Earth's atmosphere. Daily SSI measurements constitute a matrix with spectrally (rows) and temporally (columns) resolved solar energy flux measurements. The most recent SSI measurements have been made by NASA's Total and Spectral Solar Irradiance Sensor-1 (TSIS-1) Spectral Irradiance Monitor (SIM) since March 2018. This data has considerable missing data due to both random factors and instrument downtime, a periodic trend related to the Sun's cyclical magnetic activity, and varying degrees of correlation among the spectra, some approaching unity. We propose a low-rank matrix factorization method for SSI reconstruction that incorporates autoregressive temporal regularization, periodic spline detrending, and cross-spectral covariance information. The method is implemented as a two-stage procedure designed to address scattered missingness and extended downtime missingness, respectively, and is fitted using efficient alternating optimization algorithms. We further accompany the reconstructed SSI values with a distribution-free interval estimation procedure based on conformal prediction. Through synthetic experiments and real-data analyses, we compare this method with Gaussian process regression, linear time series smoothing, and existing matrix-completion approaches in terms of imputation accuracy, interval coverage, interval length, and computational efficiency. The results show that exploiting the periodic, temporal, and cross-spectral structure of SSI substantially improves reconstruction performance and yields calibrated uncertainty intervals, producing a reconstructed SSI data product suitable for downstream climate science studies.