Benchmarking pre-trained text embedding models in aligning built asset information

TL;DR

This paper benchmarks various pre-trained text embedding models to evaluate their effectiveness in aligning complex built asset technical data with domain-specific classifications, aiming to automate asset information mapping.

Contribution

It introduces a comprehensive benchmark and open-source library for evaluating text embeddings in built asset data alignment, addressing a gap in domain-specific semantic representation assessment.

Findings

Benchmarking across six datasets shows varied model performance.

Results highlight the need for domain adaptation techniques.

Open-source library supports future research in this area.

Abstract

Accurate mapping of the built asset information to established data classification systems and taxonomies is crucial for effective asset management, whether for compliance at project handover or ad-hoc data integration scenarios. Due to the complex nature of built asset data, which predominantly comprises technical text elements, this process remains largely manual and reliant on domain expert input. Recent breakthroughs in contextual text representation learning (text embedding), particularly through pre-trained large language models, offer promising approaches that can facilitate the automation of cross-mapping of the built asset data. However, no comprehensive evaluation has yet been conducted to assess these models' ability to effectively represent the complex semantics specific to built asset technical terminology. This study presents a comparative benchmark of state-of-the-art text embedding models to evaluate their effectiveness in aligning built asset information with domain-specific technical concepts. Our proposed datasets are derived from two renowned built asset data classification dictionaries. The results of our benchmarking across six proposed datasets, covering three tasks of clustering, retrieval, and reranking, highlight the need for future research on domain adaptation techniques. The benchmarking resources are published as an open-source library, which will be maintained and extended to support future evaluations in this field.