Adaptive Engram Memory System for Indonesian Language Model: Generative AI Based on TOBA LM for Batak and Minang Language

TL;DR

This paper introduces TOBA-LM, a trilingual GPT-2-based model with an adaptive Engram Memory system that significantly improves training efficiency for Indonesian, Batak, and Minangkabau languages, reducing computational costs.

Contribution

The paper presents a novel Engram Memory mechanism integrated into a GPT-2 architecture, enhancing training efficiency for regional languages with limited resources.

Findings

Training efficiency improved by 80%

Loss reduced from 6.4 to 1.7996 in fewer steps

External memory reduces computational requirements

Abstract

This study presents TOBA-LM, a trilingual language model based on GPT-2 architecture with 1.2 billion parameters, trained on a corpus encompassing Indonesian, Batak, and Minangkabau using syllabic-agglutinative tokenization. The architecture integrates an Engram Memory mechanism, an adaptive n-gram-based memory system with a 500,000 x 768 embedding table that captures morphological dependencies through bigram and trigram pathways. Empirical results demonstrate a training efficiency of 80%, with the loss value dropping from 6.4 to 1.7996 in only 12,973 steps -- significantly faster than the conventional transformer architecture, which required over 70,000 steps to achieve comparable convergence. These findings confirm that the integration of external statistical memory substantially reduces computational requirements for developing regional language models under limited resources.