DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM

TL;DR

DeTiME introduces a novel framework combining encoder-decoder LLMs and diffusion models to produce highly clusterable embeddings and enable topic-based text generation, addressing limitations of existing neural topic models.

Contribution

The paper presents a new framework that integrates diffusion models with encoder-decoder LLMs for improved clustering and text generation in topic modeling.

Findings

Produces embeddings with superior clusterability and semantic coherence

Enables efficient topic-based text generation

Demonstrates high adaptability to various LLMs and diffusion models

Abstract

In the burgeoning field of natural language processing (NLP), Neural Topic Models (NTMs) , Large Language Models (LLMs) and Diffusion model have emerged as areas of significant research interest. Despite this, NTMs primarily utilize contextual embeddings from LLMs, which are not optimal for clustering or capable for topic based text generation. NTMs have never been combined with diffusion model for text generation. Our study addresses these gaps by introducing a novel framework named Diffusion-Enhanced Topic Modeling using Encoder-Decoder-based LLMs (DeTiME). DeTiME leverages Encoder-Decoder-based LLMs to produce highly clusterable embeddings that could generate topics that exhibit both superior clusterability and enhanced semantic coherence compared to existing methods. Additionally, by exploiting the power of diffusion model, our framework also provides the capability to do topic based text generation. This dual functionality allows users to efficiently produce highly clustered topics and topic based text generation simultaneously. DeTiME's potential extends to generating clustered embeddings as well. Notably, our proposed framework(both encoder-decoder based LLM and diffusion model) proves to be efficient to train and exhibits high adaptability to other LLMs and diffusion model, demonstrating its potential for a wide array of applications.