Visual Exploration and Knowledge Discovery from Biomedical Dark Data

TL;DR

This paper presents a comprehensive approach using visualization and NLP techniques to explore and extract insights from the vast, unstructured biomedical dark data in PubMed, aiding knowledge discovery.

Contribution

It introduces a pipeline combining visualization and lexical analysis to analyze large biomedical datasets, addressing challenges of unstructured dark data.

Findings

Identified key research topics and trends in biomedical literature

Mapped relationships between authors, journals, and keywords

Demonstrated effective visualization techniques for large-scale dark data

Abstract

Data visualization techniques proffer efficient means to organize and present data in graphically appealing formats, which not only speeds up the process of decision making and pattern recognition but also enables decision-makers to fully understand data insights and make informed decisions. Over time, with the rise in technological and computational resources, there has been an exponential increase in the world's scientific knowledge. However, most of it lacks structure and cannot be easily categorized and imported into regular databases. This type of data is often termed as Dark Data. Data visualization techniques provide a promising solution to explore such data by allowing quick comprehension of information, the discovery of emerging trends, identification of relationships and patterns, etc. In this empirical research study, we use the rich corpus of PubMed comprising of more than 30 million citations from biomedical literature to visually explore and understand the underlying key-insights using various information visualization techniques. We employ a natural language processing based pipeline to discover knowledge out of the biomedical dark data. The pipeline comprises of different lexical analysis techniques like Topic Modeling to extract inherent topics and major focus areas, Network Graphs to study the relationships between various entities like scientific documents and journals, researchers, and, keywords and terms, etc. With this analytical research, we aim to proffer a potential solution to overcome the problem of analyzing overwhelming amounts of information and diminish the limitation of human cognition and perception in handling and examining such large volumes of data.