Big Data and Reliability Applications: The Complexity Dimension

TL;DR

This paper reviews how complex big data types like sensor, functional, and image data can be used in reliability analysis, highlighting modern statistical and machine learning methods to address associated challenges.

Contribution

It extends prior discussions by focusing on complex data structures in big data reliability analysis and explores analytical methods to handle their challenges.

Findings

Review of recent developments in complex data types for reliability

Discussion of modern statistical and machine learning methods

Insights into analytical challenges and solutions for complex big data

Abstract

Big data features not only large volumes of data but also data with complicated structures. Complexity imposes unique challenges in big data analytics. Meeker and Hong (2014, Quality Engineering, pp. 102-116) provided an extensive discussion of the opportunities and challenges in big data and reliability, and described engineering systems that can generate big data that can be used in reliability analysis. Meeker and Hong (2014) focused on large scale system operating and environment data (i.e., high-frequency multivariate time series data), and provided examples on how to link such data as covariates to traditional reliability responses such as time to failure, time to recurrence of events, and degradation measurements. This paper intends to extend that discussion by focusing on how to use data with complicated structures to do reliability analysis. Such data types include high-dimensional sensor data, functional curve data, and image streams. We first provide a review of recent development in those directions, and then we provide a discussion on how analytical methods can be developed to tackle the challenging aspects that arise from the complexity feature of big data in reliability applications. The use of modern statistical methods such as variable selection, functional data analysis, scalar-on-image regression, spatio-temporal data models, and machine learning techniques will also be discussed.