Structured Thoughts Automaton: First Formalized Execution Model for Auto-Regressive Language Models

TL;DR

This paper introduces the first formalized execution model for language models, providing a reliable algorithm and a low-level language to enhance understanding, inspection, and further research in LM execution processes.

Contribution

It presents the first formal execution model for LMs, including a new sampling algorithm and a low-level language for writing cognitive programs.

Findings

Developed a reliable sampling algorithm for LMs

Created a low-level language for cognitive programming

Highlighted the need for formal execution models in LMs

Abstract

In recent months, Language Models (LMs) have become a part of daily discourse, with focus on OpenAI and the potential of Artificial General Intelligence (AGI). Furthermore, the leaking of LLama's weights to the public has led to an influx of innovations demonstrating the impressive capabilities of generative LMs. While we believe that AGI is still a distant goal, we recognize the potential of LMs in solving tasks such as searching complex documents, compiling reports with basic analysis, and providing assistance in problem-solving. In this paper, we propose formalizing the execution model of language models. We investigate current execution models, to find that this formalism has received little attention, and present our contribution: the first formalized execution model for LMs. We introduce a new algorithm for sampling the predictions of LMs, which we use to build a reliable and inspectable execution model. We introduce a low-level language to write "cognitive program" for this execution model. We hope to shed light on the need for execution models for LMs and encourage further research in this area.