ChartEye: A Deep Learning Framework for Chart Information Extraction

TL;DR

This paper introduces ChartEye, a deep learning framework that automates chart information extraction by combining vision transformers, YOLOv7, and super-resolution techniques, achieving high accuracy on benchmark datasets.

Contribution

The study presents a novel end-to-end deep learning framework integrating multiple models for comprehensive chart information extraction, addressing style variation challenges.

Findings

F1-score of 0.97 for chart-type classification

F1-score of 0.91 for text-role classification

Mean Average Precision of 0.95 for text detection

Abstract

The widespread use of charts and infographics as a means of data visualization in various domains has inspired recent research in automated chart understanding. However, information extraction from chart images is a complex multitasked process due to style variations and, as a consequence, it is challenging to design an end-to-end system. In this study, we propose a deep learning-based framework that provides a solution for key steps in the chart information extraction pipeline. The proposed framework utilizes hierarchal vision transformers for the tasks of chart-type and text-role classification, while YOLOv7 for text detection. The detected text is then enhanced using Super Resolution Generative Adversarial Networks to improve the recognition output of the OCR. Experimental results on a benchmark dataset show that our proposed framework achieves excellent performance at every stage with F1-scores of 0.97 for chart-type classification, 0.91 for text-role classification, and a mean Average Precision of 0.95 for text detection.