An Online Updating Approach for Estimating and Testing Mediation Effects with Big Data Streams

TL;DR

This paper introduces an online updating method for efficient estimation and testing of mediation effects in large-scale streaming data, improving computational speed for various mediation tests.

Contribution

It proposes a novel online updating algorithm that enhances computational efficiency for mediation analysis in big data streams, applicable to linear and logistic models.

Findings

Significant speed improvements in mediation tests.

Effective application demonstrated on real-world data.

Robust performance shown through extensive simulations.

Abstract

The use of mediation analysis has become increasingly popular in various research fields in recent years. The primary objective of mediation analysis is to examine the indirect effects along the pathways from exposure to outcome. Meanwhile, the advent of data collection technology has sparked a surge of interest in the burgeoning field of big data analysis, where mediation analysis of streaming data sets has recently garnered significant attention. The enormity of the data, however, results in an augmented computational burden. The present study proposes an online updating approach to address this issue, aiming to estimate and test mediation effects in the context of linear and logistic mediation models with massive data streams. The proposed algorithm significantly enhances the computational efficiency of Sobel test, adjusted Sobel test, joint significance test, and adjusted joint significance test. We conduct a substantial number of numerical simulations to evaluate the performance of the renewable method. Two real-world examples are employed to showcase the practical applicability of this approach.