Symbolic and Language Agnostic Large Language Models

TL;DR

This paper proposes a novel approach to large language models by integrating bottom-up reverse engineering with symbolic, language-agnostic, and ontologically grounded methods, addressing limitations of subsymbolic models.

Contribution

It introduces a symbolic framework for large language models that is language-agnostic and ontologically grounded, moving beyond purely subsymbolic approaches.

Findings

Symbolic models can effectively capture language knowledge.

Ontologically grounded models improve interpretability.

Proposed approach addresses inferential limitations of subsymbolic models.

Abstract

We argue that the relative success of large language models (LLMs) is not a reflection on the symbolic vs. subsymbolic debate but a reflection on employing an appropriate strategy of bottom-up reverse engineering of language at scale. However, due to the subsymbolic nature of these models whatever knowledge these systems acquire about language will always be buried in millions of microfeatures (weights) none of which is meaningful on its own. Moreover, and due to their stochastic nature, these models will often fail in capturing various inferential aspects that are prevalent in natural language. What we suggest here is employing the successful bottom-up strategy in a symbolic setting, producing symbolic, language agnostic and ontologically grounded large language models.