MapReduce is Good Enough? If All You Have is a Hammer, Throw Away Everything That's Not a Nail!

TL;DR

This paper argues that MapReduce is sufficiently versatile for big data analysis, advocating for simplifying problem approaches rather than developing new models for non-nail algorithms.

Contribution

The paper challenges the need for alternative models by emphasizing problem reformulation over new programming paradigms for non-iterative algorithms.

Findings

Iterative algorithms are poorly suited for MapReduce.

Replacing algorithms with non-iterative solutions can be effective.

Simplifying problem approaches can reduce complexity in big data processing.

Abstract

Hadoop is currently the large-scale data analysis "hammer" of choice, but there exist classes of algorithms that aren't "nails", in the sense that they are not particularly amenable to the MapReduce programming model. To address this, researchers have proposed MapReduce extensions or alternative programming models in which these algorithms can be elegantly expressed. This essay espouses a very different position: that MapReduce is "good enough", and that instead of trying to invent screwdrivers, we should simply get rid of everything that's not a nail. To be more specific, much discussion in the literature surrounds the fact that iterative algorithms are a poor fit for MapReduce: the simple solution is to find alternative non-iterative algorithms that solve the same problem. This essay captures my personal experiences as an academic researcher as well as a software engineer in a "real-world" production analytics environment. From this combined perspective I reflect on the current state and future of "big data" research.