FRIDA to the Rescue! Analyzing Synthetic Data Effectiveness in Object-Based Common Sense Reasoning for Disaster Response

TL;DR

This paper presents FRIDA, a pipeline for creating synthetic data to enhance small language models' physical reasoning in disaster scenarios, demonstrating effective incorporation of common sense with minimal data.

Contribution

The paper introduces a novel synthetic data generation pipeline, FRIDA, tailored for improving physical reasoning in small language models for disaster response.

Findings

Ablated FRIDA models outperform others in physical reasoning tasks.

Synthetic data focusing on objects' physical states and functions is most effective.

FRIDA enables small models to acquire physical common sense with minimal data.

Abstract

During Human Robot Interactions in disaster relief scenarios, Large Language Models (LLMs) have the potential for substantial physical reasoning to assist in mission objectives. However, these reasoning capabilities are often found only in larger models, which are not currently reasonable to deploy on robotic systems due to size constraints. To meet our problem space requirements, we introduce a dataset and pipeline to create Field Reasoning and Instruction Decoding Agent (FRIDA) models. In our pipeline, domain experts and linguists combine their knowledge to make high-quality, few-shot prompts used to generate synthetic data for fine-tuning. We hand-curate datasets for this few-shot prompting and for evaluation to improve LLM reasoning on both general and disaster-specific objects. We concurrently run an ablation study to understand which kinds of synthetic data most affect performance. We fine-tune several small instruction-tuned models and find that ablated FRIDA models only trained on objects' physical state and function data outperformed both the FRIDA models trained on all synthetic data and the base models in our evaluation. We demonstrate that the FRIDA pipeline is capable of instilling physical common sense with minimal data.