# SimFS: A Simulation Data Virtualizing File System Interface

## TL;DR

SimFS is a virtualized file system interface that reduces storage needs for large-scale simulation data by re-simulating missing data on demand, enabling cost-effective long-term data analysis.

## Contribution

We introduce SimFS, a novel file system interface that virtualizes simulation data, managing re-simulations and access patterns to optimize storage and computation trade-offs.

## Key findings

- SimFS effectively reduces storage requirements for large simulation datasets.
- It employs access pattern monitoring for intelligent data prefetching.
- SimFS enables scalable, cost-effective long-term simulation data analysis.

## Abstract

Nowadays simulations can produce petabytes of data to be stored in parallel filesystems or large-scale databases. This data is accessed over the course of decades often by thousands of analysts and scientists. However, storing these volumes of data for long periods of time is not cost effective and, in some cases, practically impossible. We propose to transparently virtualize the simulation data, relaxing the storage requirements by not storing the full output and re-simulating the missing data on demand. We develop SimFS, a file system interface that exposes a virtualized view of the simulation output to the analysis applications and manages the re-simulations. SimFS monitors the access patterns of the analysis applications in order to (1) decide the data to keep stored for faster accesses and (2) to employ prefetching strategies to reduce the access time of missing data. Virtualizing simulation data allows us to trade storage for computation: this paradigm becomes similar to traditional on-disk analysis (all data is stored) or in situ (no data is stored) according with the storage resources that are assigned to SimFS. Overall, by exploiting the growing computing power and relaxing the storage capacity requirements, SimFS offers a viable path towards exa-scale simulations.