DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Tasks

TL;DR

DeepThink3D improves large language models' ability to perform complex reasoning in 3D scenes by generating more intricate questions and fine-tuning tool usage strategies, leading to better performance on 3D reasoning tasks.

Contribution

The paper introduces a novel evolutionary approach and fine-tuning method to enhance LLMs' tool usage in complex 3D reasoning tasks, addressing limitations of simple question datasets.

Findings

Enhanced question complexity via evolutionary methods

Improved tool usage accuracy through fine-tuning and DPO

Better performance on the SQA3D benchmark

Abstract

This work enhances the ability of large language models (LLMs) to perform complex reasoning in 3D scenes. Recent work has addressed the 3D situated reasoning task by invoking tool usage through large language models. Large language models call tools via APIs and integrate the generated programs through a chain of thought to solve problems based on the program results. However, due to the simplicity of the questions in the dataset, the generated program reasoning chains are relatively short. To solve this main challenge, in this paper, we introduce DeepThink3D to enhance the tool usage of LLMs in complex 3D situated reasoning tasks. Our work proposes a combinatorial and iterative evolutionary approach on the SQA3D benchmark to generate more complex questions. Building on this foundation, we fine-tune the large language model to make it more proficient in using 3D tools. By employing Direct Preference Optimization (DPO), we directly optimize the toolchain strategies generated by models, thereby enhancing their accuracy in complex tasks.