To BLOB or Not To BLOB: Large Object Storage in a Database or a Filesystem?

TL;DR

This paper investigates the optimal storage method for large objects, analyzing fragmentation and performance implications of storing objects in databases versus filesystems based on size and usage patterns.

Contribution

It provides empirical analysis of fragmentation and performance for large object storage, offering guidelines based on object size and access patterns.

Findings

Objects smaller than 256KB are best stored in a database.

Objects larger than 1MB are best stored in a filesystem.

Between 256KB and 1MB, storage choice depends on read/write ratio and overwrite rate.

Abstract

Application designers often face the question of whether to store large objects in a filesystem or in a database. Often this decision is made for application design simplicity. Sometimes, performance measurements are also used. This paper looks at the question of fragmentation - one of the operational issues that can affect the performance and/or manageability of the system as deployed long term. As expected from the common wisdom, objects smaller than 256KB are best stored in a database while objects larger than 1M are best stored in the filesystem. Between 256KB and 1MB, the read:write ratio and rate of object overwrite or replacement are important factors. We used the notion of "storage age" or number of object overwrites as way of normalizing wall clock time. Storage age allows our results or similar such results to be applied across a number of read:write ratios and object replacement rates.