LSST is Not "Big Data"

TL;DR

Despite its large data collection, LSST will be manageable with current computing resources, challenging the notion that it constitutes 'big data' in terms of storage and analysis complexity.

Contribution

The paper argues that LSST's data volume is manageable and not 'big data' by current standards, contrasting it with common perceptions and historical data challenges.

Findings

LSST data will not exceed 22 hard drives in size.

Individual researchers can maintain personal data copies.

LSST's data volume is small relative to Moore's Law growth.

Abstract

LSST promises to be the largest optical imaging survey of the sky. If we were fortunate enough to have the equivalent of LSST today, it would represent a "fire hose" of data that would be difficult to store, transfer, and analyze with available compute resources. LSST parallels the SDSS compute task which was ambitious yet tractable. By almost any measure relative to computers that will be available (thanks to the steady progression of Moore's Law), LSST will be a small data set. LSST will never fill more than 22 hard drives. Individual investigators will be able to maintain their own data copies to analyze as they choose.