Botfip-LLM: An Enhanced Multimodal Scientific Computing Framework Leveraging Knowledge Distillation from Large Language Models

TL;DR

Botfip-LLM enhances a multimodal scientific computing framework by integrating large language models through knowledge distillation, significantly improving understanding and handling of formula strings and related tasks.

Contribution

This work introduces Botfip-LLM, a novel framework that leverages pre-trained LLMs to improve multimodal scientific computing, especially in processing formula strings.

Findings

ChatGLM-2 outperforms other LLMs in this framework.

Botfip-LLM improves performance, generalization, and task applicability.

Enhanced handling of formula string-related tasks.

Abstract

In recent years, the introduction of AI technologies has brought transformative changes to scientific computing. However, AI models typically focus on single-task and single-modal data processing, limiting their application. To address this, multimodal scientific computing frameworks have become a trend. The Botfip framework aligns function images with symbolic operation trees through multimodal training, extracting deep scientific information. However, Botfip struggles with processing Formula Strings, leading to inadequate understanding in multimodal learning. To enhance Botfip's learning of Formula Strings and expand its applicability to related tasks, we propose the Botfip-LLM framework based on knowledge distillation, incorporating pre-trained large language models for aligning symbolic tree data. Experimental analysis shows that the choice of LLM is crucial, with ChatGLM-2 outperforming others in training and testing. Botfip-LLM not only improves performance, generalization, and extrapolation over the original Botfip model but also significantly enhances applicability to Formula String-related tasks, enabling more diverse task handling.