Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights

TL;DR

This paper presents a human-in-the-loop method using LLMs to identify similar data points across various non-text data types by summarization and hidden state analysis, enhancing scalability and interpretability.

Contribution

The study introduces a novel two-step LLM-based approach combining summarization and hidden state extraction for cross-domain data similarity detection.

Findings

Effective identification of similar data points across diverse datasets

Enables non-technical users to perform similarity analysis

Demonstrates scalability and practical utility in real-world scenarios

Abstract

This study introduces a simple yet effective method for identifying similar data points across non-free text domains, such as tabular and image data, using Large Language Models (LLMs). Our two-step approach involves data point summarization and hidden state extraction. Initially, data is condensed via summarization using an LLM, reducing complexity and highlighting essential information in sentences. Subsequently, the summarization sentences are fed through another LLM to extract hidden states, serving as compact, feature-rich representations. This approach leverages the advanced comprehension and generative capabilities of LLMs, offering a scalable and efficient strategy for similarity identification across diverse datasets. We demonstrate the effectiveness of our method in identifying similar data points on multiple datasets. Additionally, our approach enables non-technical domain experts, such as fraud investigators or marketing operators, to quickly identify similar data points tailored to specific scenarios, demonstrating its utility in practical applications. In general, our results open new avenues for leveraging LLMs in data analysis across various domains