Model Input Verification of Large Scale Simulations

TL;DR

This paper introduces FabGuard, a data validation tool for large-scale simulations that ensures input correctness, integrates into workflows, and leverages LLMs for automating constraint inference, demonstrated across diverse simulation domains.

Contribution

It presents a formalism and pipeline for model input verification in simulations, incorporating LLMs for automating constraint generation and validation.

Findings

FabGuard efficiently processes 12,000 files in 140 seconds.

LLMs correctly inferred 22 out of 23 constraints in a migration case study.

MIV is feasible for large datasets and diverse simulation domains.

Abstract

Reliable simulations are critical for analyzing and understanding complex systems, but their accuracy depends on correct input data. Incorrect inputs such as invalid or out-of-range values, missing data, and format inconsistencies can cause simulation crashes or unnoticed result distortions, ultimately undermining the validity of the conclusions. This paper presents a methodology for verifying the validity of input data in simulations, a process we term model input verification (MIV). We implement this approach in FabGuard, a toolset that uses established data schema and validation tools for the specific needs of simulation modeling. We introduce a formalism for categorizing MIV patterns and offer a streamlined verification pipeline that integrates into existing simulation workflows. FabGuard's applicability is demonstrated across three diverse domains: conflict-driven migration, disaster evacuation, and disease spread models. We also explore the use of Large Language Models (LLMs) for automating constraint generation and inference. In a case study with a migration simulation, LLMs not only correctly inferred 22 out of 23 developer-defined constraints, but also identified errors in existing constraints and proposed new, valid constraints. Our evaluation demonstrates that MIV is feasible on large datasets, with FabGuard efficiently processing 12,000 input files in 140 seconds and maintaining consistent performance across varying file sizes.