# Distributed sequential method for analyzing massive data

## TL;DR

This paper introduces a distributed sequential estimation method for analyzing massive datasets, combining parallel processing, adaptive sampling, and variable selection to improve efficiency and accuracy.

## Contribution

It proposes a novel parallel divide-and-conquer approach with adaptive sampling and shrinkage estimation, maintaining statistical properties for large-scale data analysis.

## Key findings

- Method achieves efficient estimation on large datasets.
- Adaptive sampling accelerates the estimation process.
- Successfully applied to real-world energy and pollution data.

## Abstract

To analyse a very large data set containing lengthy variables, we adopt a sequential estimation idea and propose a parallel divide-and-conquer method. We conduct several conventional sequential estimation procedures separately, and properly integrate their results while maintaining the desired statistical properties. Additionally, using a criterion from the statistical experiment design, we adopt an adaptive sample selection, together with an adaptive shrinkage estimation method, to simultaneously accelerate the estimation procedure and identify the effective variables. We confirm the cogency of our methods through theoretical justifications and numerical results derived from synthesized data sets. We then apply the proposed method to three real data sets, including those pertaining to appliance energy use and particulate matter concentration.