Teaching and Learning under Deductive Errors

TL;DR

This paper introduces a framework for machine teaching and learning that accounts for deductive errors, analyzing theoretical and computational aspects, and validating with experiments on large language models.

Contribution

It develops a PAC-based model incorporating deductive errors, studies the complexity of finding optimal teaching sets, and empirically evaluates protocols with LLMs.

Findings

Theoretical bounds on PAC teaching set computation under error

XP algorithms with tight runtime bounds based on set size

Experimental insights into teaching protocols for LLMs

Abstract

Most models of machine teaching and learning assume the learner makes no errors in its internal deductive inference. However, humans and large language models in few-shot learning regimes are two important examples of learners where this does not hold. They fail on some consistency checks, and they can fail stochastically. In this paper we introduce a teaching and learning framework that takes these deductive errors into account. We specifically study the case of machine teaching, as different characterizations of the teacher can account for both machine teaching and learning. In an overhauled Probably Approximately Correct (PAC) setting, we study theoretically that, for some estimated error level, the teacher must find a PAC teaching set that with high probability will lead the learner to guess a hypothesis that is approximately correct. We study the computational complexity of six different problems related to computing optimal PAC teaching sets. We give XP algorithms parametrized by size of teaching set, with tight runtime bounds under standard complexity assumptions like ETH. These results are complemented with a small experimental study of which teaching and learning protocols can best represent the observed behavior in some LLM teaching sessions.