The iCrawl Wizard -- Supporting Interactive Focused Crawl Specification

TL;DR

The paper introduces the iCrawl Wizard, a tool that helps users, including non-experts, efficiently define focused web crawls by semi-automatically generating seed URLs and semantic descriptions using search engines, social media, and information extraction.

Contribution

It presents a novel interactive tool that simplifies and automates the process of specifying focused crawls, reducing the need for expert knowledge.

Findings

Enables non-experts to create focused crawls interactively.

Uses search engines and social media APIs for seed URL generation.

Provides semantic descriptions of crawl intent.

Abstract

Collections of Web documents about specific topics are needed for many areas of current research. Focused crawling enables the creation of such collections on demand. Current focused crawlers require the user to manually specify starting points for the crawl (seed URLs). These are also used to describe the expected topic of the collection. The choice of seed URLs influences the quality of the resulting collection and requires a lot of expertise. In this demonstration we present the iCrawl Wizard, a tool that assists users in defining focused crawls efficiently and semi-automatically. Our tool uses major search engines and Social Media APIs as well as information extraction techniques to find seed URLs and a semantic description of the crawl intent. Using the iCrawl Wizard even non-expert users can create semantic specifications for focused crawlers interactively and efficiently.