Method for Aggregating Unstructured Data Using Large Language Models

TL;DR

This paper introduces a robust method combining web scraping, LLMs, and verification techniques to automate and improve the aggregation of unstructured web data, addressing instability and manual effort issues.

Contribution

It presents a novel two-stage verification process for LLM-generated data, enhancing accuracy and robustness in dynamic web content aggregation.

Findings

High accuracy in key field completion

Robustness to webpage structure changes

Scalable for real-time news and log analysis

Abstract

This paper presents a method for the automated collection and aggregation of unstructured data from diverse web sources, utilizing Large Language Models (LLMs). The primary challenge with existing techniques is their instability when the structure of webpages changes, their limited support for dynamically loaded content during information collection, and the requirement for labor-intensive manual design of data pre-processing processes. The proposed algorithm integrates hybrid web scraping (Goose3 for static pages and Selenium+WebDriver for dynamic ones), data storage in a non-relational MongoDB database management system (DBMS), and intelligent extraction and normalization of information using LLMs into a predetermined JSON schema. A key scientific contribution of this study is a two-stage verification process for the generated data, designed to eliminate potential hallucinations byy comparing the embeddings of multiple LLM outputs obtained with different temperature parameter values, combined with formalized rules for monitoring data consistency and integrity. The experimental findings indicate a high level of accuracy in the completion of key fields, as well as the robustness of the proposed methodology to changes in web page structures. This makes it suitable for use in tasks such as news content aggregation, monitoring, and log analysis in near real-time mode, with the capacity to scale rapidly in terms of the number of sources.