Random Access Concatenated Libraries and dd enable a short-latency high-content website on an inexpensive shared server

TL;DR

This paper presents a method using concatenated libraries and the Linux utility dd to host large, content-rich websites efficiently on inexpensive shared servers by minimizing resource demands and maintaining short latency.

Contribution

It introduces a novel approach combining concatenated libraries with dd to enable high-content websites on limited-resource shared servers, demonstrating scalability across diverse applications.

Findings

Efficient hosting of 8000+ genealogical books with 3.5 million images.

Management of 86 million death records in a website database.

Utilization of 800 neuroimaging databases in cluster computing.

Abstract

Content-rich websites typically house their images as individual files or as more costly binary database objects. Both methods place high demands on storage resources with commensurate high monetary cost. Inexpensive shared service accounts come with strict limits on disk and computing resources which prevent their use for hosting content-rich websites. To minimize the load on these limited resources, large numbers of images or other records may be concatenated into a single file and delivered with short latency independent of location within the file using Linux utility dd. This solution and its performance are presented for (a) a website which houses 8000+ genealogical reference books with more than 3.5 million 250-Kbyte page images, (b) a website-based database of 86 million death records, and (c) a cluster computing application which utilizes 800 neuroimaging databases, each with 1.5 million records. These disparate applications demonstrate the efficacy, scalability, and general utility of the solution.