Through the Thicket: A Study of Number-Oriented LLMs derived from Random Forest Models

TL;DR

This paper introduces a novel method for training large language models by transferring knowledge from random forest models, converting decision paths into natural language to improve numerical data interpretation and decision explanation.

Contribution

It presents a new approach to fine-tuning LLMs using RF-derived rules, enhancing their numerical reasoning and interpretability capabilities.

Findings

RF-based rule conversion improves LLM classification accuracy

Preprocessing techniques significantly affect numerical data representation

The method enhances LLM's ability to explain decisions based on numerical data

Abstract

Large Language Models (LLMs) have shown exceptional performance in text processing. Notably, LLMs can synthesize information from large datasets and explain their decisions similarly to human reasoning through a chain of thought (CoT). An emerging application of LLMs is the handling and interpreting of numerical data, where fine-tuning enhances their performance over basic inference methods. This paper proposes a novel approach to training LLMs using knowledge transfer from a random forest (RF) ensemble, leveraging its efficiency and accuracy. By converting RF decision paths into natural language statements, we generate outputs for LLM fine-tuning, enhancing the model's ability to classify and explain its decisions. Our method includes verifying these rules through established classification metrics, ensuring their correctness. We also examine the impact of preprocessing techniques on the representation of numerical data and their influence on classification accuracy and rule correctness