Not All Options Are Created Equal: Textual Option Weighting for Token-Efficient LLM-Based Knowledge Tracing

TL;DR

This paper introduces LOKT, a novel framework for knowledge tracing using LLMs that encodes interaction histories with textual option weights, improving scalability, interpretability, and performance in token-constrained environments.

Contribution

The paper proposes TCOW, a semantic labeling method for interaction history encoding, and demonstrates LOKT's superior performance and efficiency over existing models in knowledge tracing tasks.

Findings

LOKT outperforms existing models in multiple-choice datasets.

LOKT enables scalable inference under token constraints.

TCOW enhances interpretability of LLM-based KT.

Abstract

Large Language Models (LLMs) have recently emerged as promising tools for knowledge tracing (KT) due to their strong reasoning and generalization abilities. While recent LLM-based KT methods have proposed new prompt formats, they struggle to represent the full interaction histories of example learners within a single prompt during in-context learning (ICL), resulting in limited scalability and high computational cost under token constraints. In this work, we present \textit{LLM-based Option-weighted Knowledge Tracing (LOKT)}, a simple yet effective framework that encodes the interaction histories of example learners in context as \textit{textual categorical option weights (TCOW)}. TCOW are semantic labels (e.g., ``inadequate'') assigned to the options selected by learners when answering questions, enhancing the interpretability of LLMs. Experiments on multiple-choice datasets show that LOKT outperforms existing non-LLM and LLM-based KT models in both cold-start and warm-start settings. Moreover, LOKT enables scalable and cost-efficient inference, achieving strong performance even under strict token constraints. Our code is available at \href{https://anonymous.4open.science/r/LOKT_model-3233}{https://anonymous.4open.science/r/LOKT\_model-3233}.