Effective Focused Crawling Based on Content and Link Structure Analysis

TL;DR

This paper presents an improved focused crawling technique that enhances web navigation quality by using similarity functions, link prioritization based on metadata, and traversing irrelevant pages to increase topic coverage.

Contribution

It introduces a novel focused crawling method that combines content similarity, link prioritization, and irrelevant page traversal to improve relevance and coverage.

Findings

Enhanced relevance of crawled pages.

Increased coverage of target topics.

Improved navigation efficiency.

Abstract

A focused crawler traverses the web selecting out relevant pages to a predefined topic and neglecting those out of concern. While surfing the internet it is difficult to deal with irrelevant pages and to predict which links lead to quality pages. In this paper a technique of effective focused crawling is implemented to improve the quality of web navigation. To check the similarity of web pages w.r.t. topic keywords a similarity function is used and the priorities of extracted out links are also calculated based on meta data and resultant pages generated from focused crawler. The proposed work also uses a method for traversing the irrelevant pages that met during crawling to improve the coverage of a specific topic.