TeraScale SneakerNet: Using Inexpensive Disks for Backup, Archiving, and Data Exchange

TL;DR

This paper presents TeraScale SneakerNet, a cost-effective system using inexpensive disks and specialized hardware bricks for efficient transfer, backup, and archiving of large datasets like those from the Sloan Digital Sky Survey.

Contribution

It introduces a novel hardware and software approach using 3GT storage bricks for large dataset transfer and storage at a significantly reduced cost.

Findings

Cost-effective data transfer solution for terabyte datasets

Successful deployment for Sloan Digital Sky Survey data

Discussion of hardware and software challenges

Abstract

Large datasets are most economically trnsmitted via parcel post given the current economics of wide-area networking. This article describes how the Sloan Digital Sky Survey ships terabyte scale datasets both within the US and to Europe and Asia. We 3GT storage bricks (Ghz processor, GB ram, GbpsEthernet, TB disk) for about 2k$ each. These bricks act as database servers on the LAN. They are loaded at one site and read at the second site. The paper describes the bricks, their economics, and some software issues that they raise.