Probabilistic data quality assessment for structural monitoring data via outlier-resistant conditional diffusion model

TL;DR

This paper introduces a novel probabilistic framework using a conditional diffusion model for outlier-resistant data quality assessment in structural health monitoring, improving accuracy and robustness over existing methods.

Contribution

It develops a conditional diffusion model with outlier detection capabilities, integrating temporal context and robust loss functions for enhanced data quality evaluation.

Findings

The proposed method outperforms baseline approaches in real-world case studies.

It effectively identifies outliers and assesses overall data quality.

Ablation and hyperparameter analyses confirm robustness and effectiveness.

Abstract

Data quality assessment is an essential step that ensures the reliability of the subsequent structural health monitoring (SHM) tasks. This study proposes a prediction deviation-based SHM data quality assessment method using a univariate implicit auto-regressive model, enabling outlier diagnosis and data cleaning. The proposed conditional diffusion model (CDM) augments the standard diffusion model with a conditional embedding module to incorporate temporal context, quartile normalization to mitigate distribution skew, and a Huber loss to enhance robustness against outliers. Within this univariate implicit autoregressive framework, each data point is assigned an outlier probability, quantifying its degree of "outlier-ness", and a global quality evaluation score is computed to characterize the overall dataset quality. Extensive case studies utilizing operational data from real-world structures demonstrate that the proposed framework significantly improves the accuracy of data quality assessment, outperforming other strong baselines representative of clustering, isolation-based, and deep reconstruction methods. The effectiveness and robustness of the proposed framework are further demonstrated by the findings of ablation experiments and hyperparameter analysis.