An Efficient Data Analysis Method for Big Data using Multiple-Model Linear Regression

TL;DR

This paper presents a novel multiple-model linear regression approach for big data that improves efficiency and flexibility, with an approximate algorithm that guarantees correctness and linear time complexity, validated on synthetic and real datasets.

Contribution

Introduces the MMLR model and an efficient $(,0)$-estimator based algorithm with proven correctness and linear time complexity for big data analysis.

Findings

MMLR achieves comparable prediction accuracy to existing methods.

The algorithm operates with linear time complexity relative to dataset size.

Empirical results demonstrate high efficiency and accuracy on real-world data.

Abstract

This paper introduces a new data analysis method for big data using a newly defined regression model named multiple model linear regression(MMLR), which separates input datasets into subsets and construct local linear regression models of them. The proposed data analysis method is shown to be more efficient and flexible than other regression based methods. This paper also proposes an approximate algorithm to construct MMLR models based on $(\epsilon,\delta)$-estimator, and gives mathematical proofs of the correctness and efficiency of MMLR algorithm, of which the time complexity is linear with respect to the size of input datasets. This paper also empirically implements the method on both synthetic and real-world datasets, the algorithm shows to have comparable performance to existing regression methods in many cases, while it takes almost the shortest time to provide a high prediction accuracy.