A Focused Crawler Combinatory Link and Content Model Based on T-Graph Principles

TL;DR

This paper introduces Treasure-Crawler, a focused Web crawler that combines link and content analysis using a T-Graph to accurately identify and prioritize topic-specific web pages for efficient crawling.

Contribution

It presents a novel combined link-content approach and a T-Graph based scoring method for improved focused crawling accuracy and prioritization.

Findings

High accuracy in predicting topical relevance of unvisited pages

Effective prioritization of URLs using T-Graph scoring

Successful architectural validation through test results

Abstract

The two significant tasks of a focused Web crawler are finding relevant topic-specific documents on the Web and analytically prioritizing them for later effective and reliable download. For the first task, we propose a sophisticated custom algorithm to fetch and analyze the most effective HTML structural elements of the page as well as the topical boundary and anchor text of each unvisited link, based on which the topical focus of an unvisited page can be predicted and elicited with a high accuracy. Thus, our novel method uniquely combines both link-based and content-based approaches. For the second task, we propose a scoring function of the relevant URLs through the use of T-Graph (Treasure Graph) to assist in prioritizing the unvisited links that will later be put into the fetching queue. Our Web search system is called the Treasure-Crawler. This research paper embodies the architectural design of the Treasure-Crawler system which satisfies the principle requirements of a focused Web crawler, and asserts the correctness of the system structure including all its modules through illustrations and by the test results.