Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

TL;DR

This paper explores using vision language models to generate plant simulation configurations from drone images, aiming to improve scalability in agricultural digital twins by leveraging in-context learning with open-source VLMs.

Contribution

It introduces a novel approach employing VLMs to produce JSON simulation parameters from images, marking the first use of VLMs for plant simulation configuration generation.

Findings

VLMs can interpret structural metadata and estimate plant parameters.

Performance degrades due to contextual bias or dataset mean reliance.

Validation shows potential and limitations of VLMs in real-world scenarios.

Abstract

This paper introduces a synthetic benchmark to evaluate the performance of vision language models (VLMs) in generating plant simulation configurations for digital twins. While functional-structural plant models (FSPMs) are useful tools for simulating biophysical processes in agricultural environments, their high complexity and low throughput create bottlenecks for deployment at scale. We propose a novel approach that leverages state-of-the-art open-source VLMs -- Gemma 3 and Qwen3-VL -- to directly generate simulation parameters in JSON format from drone-based remote sensing images. Using a synthetic cowpea plot dataset generated via the Helios 3D procedural plant generation library, we tested five in-context learning methods and evaluated the models across three categories: JSON integrity, geometric evaluations, and biophysical evaluations. Our results show that while VLMs can interpret structural metadata and estimate parameters like plant count and sun azimuth, they often exhibit performance degradation due to contextual bias or rely on dataset means when visual cues are insufficient. Validation on a real-world drone orthophoto dataset and an ablation study using a blind baseline further characterize the models' reasoning capabilities versus their reliance on contextual priors. To the best of our knowledge, this is the first study to utilize VLMs to generate structural JSON configurations for plant simulations, providing a scalable framework for reconstruction 3D plots for digital twin in agriculture.