Random Forest-of-Thoughts: Uncertainty-aware Reasoning for Computational Social Science

TL;DR

This paper introduces Random Forest of Thoughts (RFoT), a novel prompting method for large language models that enhances their reasoning capabilities in computational social science by enabling diverse, uncertainty-aware decision-making processes.

Contribution

The paper proposes RFoT, a new prompting approach that allows LLMs to explore multiple reasoning paths and handle uncertainty, improving social survey analysis tasks.

Findings

RFoT significantly improves LLM performance on social survey analysis problems.

RFoT enables diverse thought generation and uncertainty modeling in LLM reasoning.

Experiments demonstrate RFoT's effectiveness on two real-world social science datasets.

Abstract

Social surveys in computational social science are well-designed by elaborate domain theories that can effectively reflect the interviewee's deep thoughts without concealing their true feelings. The candidate questionnaire options highly depend on the interviewee's previous answer, which results in the complexity of social survey analysis, the time, and the expertise required. The ability of large language models (LLMs) to perform complex reasoning is well-enhanced by prompting learning such as Chain-of-thought (CoT) but still confined to left-to-right decision-making processes or limited paths during inference. This means they can fall short in problems that require exploration and uncertainty searching. In response, a novel large language model prompting method, called Random Forest of Thoughts (RFoT), is proposed for generating uncertainty reasoning to fit the area of computational social science. The RFoT allows LLMs to perform deliberate decision-making by generating diverse thought space and randomly selecting the sub-thoughts to build the forest of thoughts. It can extend the exploration and prediction of overall performance, benefiting from the extensive research space of response. The method is applied to optimize computational social science analysis on two datasets covering a spectrum of social survey analysis problems. Our experiments show that RFoT significantly enhances language models' abilities on two novel social survey analysis problems requiring non-trivial reasoning.