LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ

TL;DR

LLaMa-SciQ is an educational chatbot designed to help students solve science MCQs, using fine-tuned LLaMa-8B with retrieval augmentation and quantization, achieving high accuracy and efficiency in mathematical reasoning tasks.

Contribution

The paper introduces LLaMa-SciQ, a fine-tuned and aligned LLaMa-based chatbot with retrieval augmentation and quantization, tailored for STEM MCQ assistance, and evaluates its performance.

Findings

LLaMa-SciQ achieved 74.5% accuracy on GSM8k dataset.

Quantization caused only a 5% performance loss.

Retrieval-Augmented Generation did not improve accuracy.

Abstract

Large Language Models (LLMs) often struggle with tasks requiring mathematical reasoning, particularly multiple-choice questions (MCQs). To address this issue, we developed LLaMa-SciQ, an educational chatbot designed to assist college students in solving and understanding MCQs in STEM fields. We begin by fine-tuning and aligning the models to human preferences. After comparing the performance of Mistral-7B and LLaMa-8B, we selected the latter as the base model due to its higher evaluation accuracy. To further enhance accuracy, we implement Retrieval-Augmented Generation (RAG) and apply quantization to compress the model, reducing inference time and increasing accessibility for students. For mathematical reasoning, LLaMa-SciQ achieved 74.5% accuracy on the GSM8k dataset and 30% on the MATH dataset. However, RAG does not improve performance and even reduces it, likely due to retriever issues or the model's unfamiliarity with context. Despite this, the quantized model shows only a 5% loss in performance, demonstrating significant efficiency improvements.