iCrawl: Improving the Freshness of Web Collections by Integrating Social Web and Focused Web Crawling

TL;DR

This paper presents iCrawl, a novel integrated focused crawler that combines Web and Social Media streams to enhance the freshness and relevance of online content collections for current event monitoring.

Contribution

It introduces a unified system that leverages social media streams to improve the freshness and topical relevance of focused web crawling.

Findings

Enhanced content freshness in Web collections

Effective integration of Social Media and Web crawling

Improved relevance for current event monitoring

Abstract

Researchers in the Digital Humanities and journalists need to monitor, collect and analyze fresh online content regarding current events such as the Ebola outbreak or the Ukraine crisis on demand. However, existing focused crawling approaches only consider topical aspects while ignoring temporal aspects and therefore cannot achieve thematically coherent and fresh Web collections. Especially Social Media provide a rich source of fresh content, which is not used by state-of-the-art focused crawlers. In this paper we address the issues of enabling the collection of fresh and relevant Web and Social Web content for a topic of interest through seamless integration of Web and Social Media in a novel integrated focused crawler. The crawler collects Web and Social Media content in a single system and exploits the stream of fresh Social Media content for guiding the crawler.