A Proposed Architecture for Continuous Web Monitoring Through Online Crawling of Blogs

TL;DR

This paper presents an architecture for continuous online crawling of blogs to monitor web content in real-time, aiding analysts in timely decision-making by focusing on relevant blog data through a weighted graph-based focused crawler.

Contribution

It introduces a novel architecture utilizing a focused crawler and weighted graph to efficiently monitor and analyze blogs in real-time, addressing the challenge of vast web data.

Findings

Effective continuous blog monitoring achieved

Weighted graph improves relevance of fetched data

Real-time analysis supports timely decision-making

Abstract

Getting informed of what is registered in the Web space on time, can greatly help the psychologists, marketers and political analysts to familiarize, analyse, make decision and act correctly based on the society`s different needs. The great volume of information in the Web space hinders us to continuously online investigate the whole space of the Web. Focusing on the considered blogs limits our working domain and makes the online crawling in the Web space possible. In this article, an architecture is offered which continuously online crawls the related blogs, using focused crawler, and investigates and analyses the obtained data. The online fetching is done based on the latest announcements of the ping server machines. A weighted graph is formed based on targeting the important key phrases, so that a focused crawler can do the fetching of the complete texts of the related Web pages, based on the weighted graph.