Steps are all you need: Rethinking STEM Education with Prompt Engineering

TL;DR

This paper introduces a novel prompting technique called Analogical CoT prompting that enhances the mathematical reasoning abilities of LLMs, especially smaller models, by leveraging analogical reasoning and Mixture of Experts, improving performance on physics question answering tasks.

Contribution

It proposes Analogical CoT prompting and demonstrates its effectiveness in improving LLM performance, particularly for smaller models, by addressing hallucination and mathematical reasoning limitations.

Findings

Improved performance of LLMs on physics QA tasks with Mixture of Experts and analogical prompting.

Analogical CoT prompting enables smaller models to better leverage analogical reasoning.

Survey of prompting techniques and their impact on model performance.

Abstract

Few shot and Chain-of-Thought prompting have shown promise when applied to Physics Question Answering Tasks, but are limited by the lack of mathematical ability inherent to LLMs, and are prone to hallucination. By utilizing a Mixture of Experts (MoE) Model, along with analogical prompting, we are able to show improved model performance when compared to the baseline on standard LLMs. We also survey the limits of these prompting techniques and the effects they have on model performance. Additionally, we propose Analogical CoT prompting, a prompting technique designed to allow smaller, open source models to leverage Analogical prompting, something they have struggled with, possibly due to a lack of specialist training data.