DeepDiveAI: Identifying AI Related Documents in Large Scale Literature Data

TL;DR

This paper introduces DeepDiveAI, a method combining expert knowledge and advanced models to automatically classify AI-related documents in large literature datasets, resulting in an accurate and efficient AI literature dataset.

Contribution

The paper presents a novel two-stage classification approach integrating LSTM and BERT models with expert input to create a comprehensive AI-related literature dataset.

Findings

High classification accuracy demonstrated

Efficient processing of large-scale datasets achieved

DeepDiveAI dataset enables better AI literature analysis

Abstract

In this paper, we propose a method to automatically classify AI-related documents from large-scale literature databases, leading to the creation of an AI-related literature dataset, named DeepDiveAI. The dataset construction approach integrates expert knowledge with the capabilities of advanced models, structured across two global stages. In the first stage, expert-curated classification datasets are used to train an LSTM model, which classifies coarse AI related records from large-scale datasets. In the second stage, we use Qwen2.5 Plus to annotate a random 10% of the coarse AI-related records, which are then used to train a BERT binary classifier. This step further refines the coarse AI related record set to obtain the final DeepDiveAI dataset. Evaluation results demonstrate that the entire workflow can efficiently and accurately identify AI-related literature from large-scale datasets.