Comparing how Large Language Models perform against keyword-based searches for social science research data discovery

TL;DR

This study compares large language model-based semantic search with traditional keyword search for social science data discovery, showing LLM's advantages in handling complex, misspelled, and place-based queries, thus improving data retrieval effectiveness.

Contribution

The paper provides an empirical evaluation demonstrating that LLM-based semantic search enhances social science data discovery by effectively handling complex and misspelled queries, complementing traditional keyword search.

Findings

Semantic search returns more results, especially for complex queries.

High semantic similarity between datasets retrieved by both methods.

Semantic search is robust to spelling errors and interprets geographic relevance.

Abstract

This paper evaluates the performance of a large language model (LLM) based semantic search tool relative to a traditional keyword-based search for data discovery. Using real-world search behaviour, we compare outputs from a bespoke semantic search system applied to UKRI data services with the Consumer Data Research Centre (CDRC) keyword search. Analysis is based on 131 of the most frequently used search terms extracted from CDRC search logs between December 2023 and October 2024. We assess differences in the volume, overlap, ranking, and relevance of returned datasets using descriptive statistics, qualitative inspection, and quantitative similarity measures, including exact dataset overlap, Jaccard similarity, and cosine similarity derived from BERT embeddings. Results show that the semantic search consistently returns a larger number of results than the keyword search and performs particularly well for place based, misspelled, obscure, or complex queries. While the semantic search does not capture all keyword based results, the datasets returned are overwhelmingly semantically similar, with high cosine similarity scores despite lower exact overlap. Rankings of the most relevant results differ substantially between tools, reflecting contrasting prioritisation strategies. Case studies demonstrate that the LLM based tool is robust to spelling errors, interprets geographic and contextual relevance effectively, and supports natural-language queries that keyword search fails to resolve. Overall, the findings suggest that LLM driven semantic search offers a substantial improvement for data discovery, complementing rather than fully replacing traditional keyword-based approaches.