Idea-Gated Transformers: Enforcing Semantic Coherence via Differentiable Vocabulary Pruning

TL;DR

This paper introduces the Idea-Gated Transformer, which uses a differentiable gating mechanism based on semantic planning to improve topic coherence and domain retention in language generation, addressing the limitations of autoregressive models.

Contribution

The paper presents a novel architecture that separates semantic planning from syntactic generation using an auxiliary Idea Head and a gating mechanism, enhancing controllability and domain retention.

Findings

Achieves comparable perplexity to GPT-2 baseline.

Significantly improves domain retention and semantic coherence.

Effectively locks generation into specific semantic clusters.

Abstract

Autoregressive Language Models (LLMs) trained on Next-Token Prediction (NTP) often suffer from Topic Drift where the generation wanders away from the initial prompt due to a reliance on local associations rather than global planning. While scaling model size mitigates this, the fundamental myopia of the NTP objective remains. In this work, we introduce the Idea-Gated Transformer, a novel architecture that separates semantic planning from syntactic generation. We introduce an auxiliary Idea Head trained to predict the bag-of-words distribution for a future context window, creating a latent ``Concept Vector'' that actively gates the main vocabulary during generation. We propose a differentiable gating mechanism that suppresses semantically irrelevant tokens, effectively pruning the search space in real-time. Experiments on WikiText-103 demonstrate that while the Idea-Gated model achieves comparable validation perplexity to a standard GPT-2 baseline, it exhibits significantly superior Domain Retention. Qualitative and quantitative analysis reveals that the gating mechanism successfully locks generation into specific semantic clusters (e.g., Finance, Science) and resists associative drift, offering a parameter-efficient path toward more controllable language modeling.