FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation

TL;DR

FuDoBa is a Bayesian optimization method that combines LLM-based embeddings with domain knowledge to create efficient, interpretable, and task-relevant document representations that improve classification performance.

Contribution

It introduces a novel fusion approach that integrates LLM embeddings with structured knowledge, reducing dimensionality and complexity for domain-specific tasks.

Findings

Outperforms LLM-only embeddings on six datasets across two domains.

Produces low-dimensional, interpretable representations.

Enhances classification accuracy with AutoML classifiers.

Abstract

Building on the success of Large Language Models (LLMs), LLM-based representations have dominated the document representation landscape, achieving great performance on the document embedding benchmarks. However, the high-dimensional, computationally expensive embeddings from LLMs tend to be either too generic or inefficient for domain-specific applications. To address these limitations, we introduce FuDoBa a Bayesian optimisation-based method that integrates LLM-based embeddings with domain-specific structured knowledge, sourced both locally and from external repositories like WikiData. This fusion produces low-dimensional, task-relevant representations while reducing training complexity and yielding interpretable early-fusion weights for enhanced classification performance. We demonstrate the effectiveness of our approach on six datasets in two domains, showing that when paired with robust AutoML-based classifiers, our proposed representation learning approach performs on par with, or surpasses, those produced solely by the proprietary LLM-based embedding baselines.