FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs

TL;DR

This paper introduces FinReflectKG, a large-scale, open-source financial knowledge graph built from SEC filings, along with a novel construction framework that uses intelligent parsing, iterative extraction, and reflection-driven evaluation to improve quality.

Contribution

It presents a comprehensive financial KG dataset and a robust, reflection-based extraction framework that enhances accuracy and reliability over existing methods.

Findings

Reflection-agent mode achieves 64.8% compliance score.

Outperforms baseline methods in precision, comprehensiveness, and relevance.

Supports flexible extraction modes for different efficiency and accuracy needs.

Abstract

The financial domain poses unique challenges for knowledge graph (KG) construction at scale due to the complexity and regulatory nature of financial documents. Despite the critical importance of structured financial knowledge, the field lacks large-scale, open-source datasets capturing rich semantic relationships from corporate disclosures. We introduce an open-source, large-scale financial knowledge graph dataset built from the latest annual SEC 10-K filings of all S and P 100 companies - a comprehensive resource designed to catalyze research in financial AI. We propose a robust and generalizable knowledge graph (KG) construction framework that integrates intelligent document parsing, table-aware chunking, and schema-guided iterative extraction with a reflection-driven feedback loop. Our system incorporates a comprehensive evaluation pipeline, combining rule-based checks, statistical validation, and LLM-as-a-Judge assessments to holistically measure extraction quality. We support three extraction modes - single-pass, multi-pass, and reflection-agent-based - allowing flexible trade-offs between efficiency, accuracy, and reliability based on user requirements. Empirical evaluations demonstrate that the reflection-agent-based mode consistently achieves the best balance, attaining a 64.8 percent compliance score against all rule-based policies (CheckRules) and outperforming baseline methods (single-pass and multi-pass) across key metrics such as precision, comprehensiveness, and relevance in LLM-guided evaluations.