Data Analysis and Performance Evaluation of Simulation Deduction Based on LLMs

TL;DR

This paper presents a structured multi-step approach using large language models to improve the quality and efficiency of simulation deduction data analysis and performance evaluation in military contexts.

Contribution

It introduces a novel method that decomposes complex analysis tasks into sub-tasks with tailored prompts, multi-round interactions, and custom tools for enhanced structured reporting.

Findings

Generated reports have higher quality scores than baseline methods.

The approach effectively handles diverse data types and scenarios.

Multi-round LLM interactions improve analysis accuracy.

Abstract

Data analysis and performance evaluation of simulation deduction plays a pivotal role in modern warfare, which enables military personnel to gain invaluable insights into the potential effectiveness of different strategies, tactics, and operational plans. Traditional manual analysis approach is time-consuming and limited by human errors. To enhance efficiency and accuracy, large language models (LLMs) with strong analytical and inferencing capabilities can be employed. However, high-quality analysis reports with well-structured formatting cannot be obtained through a single instruction input to the LLM. To tackle this issue, we propose a method that first decomposes the complex task into several sub-tasks and designs effective system prompts and user prompts for each sub-task. Multi-round interactions with the LLM incorporating self-check and reflection are then conducted to enable structured data extraction as well as multi-step analysis and evaluation. Furthermore, custom tools are defined and invoked to generate figures and compute metrics. We also design multiple report templates, each tailored to a specific application and input data type, ensuring their adaptability across a variety of scenarios. Extensive evaluation results demonstrate that the reports generated by our method exhibit higher quality, therefore obtaining higher scores than the baseline method.