PDD Crawler: A focused web crawler using link and content analysis for relevance prediction

TL;DR

This paper introduces PDD Crawler, a focused web crawler that combines link analysis and content analysis to predict page relevance more effectively, aiming to improve search efficiency.

Contribution

It presents a novel crawling strategy that integrates HTML tag content analysis with link-based methods to assess page relevance.

Findings

Enhanced relevance prediction accuracy

Effective content and link analysis integration

Potential for improved search engine performance

Abstract

Majority of the computer or mobile phone enthusiasts make use of the web for searching activity. Web search engines are used for the searching; The results that the search engines get are provided to it by a software module known as the Web Crawler. The size of this web is increasing round-the-clock. The principal problem is to search this huge database for specific information. To state whether a web page is relevant to a search topic is a dilemma. This paper proposes a crawler called as PDD crawler which will follow both a link based as well as a content based approach. This crawler follows a completely new crawling strategy to compute the relevance of the page. It analyses the content of the page based on the information contained in various tags within the HTML source code and then computes the total weight of the page. The page with the highest weight, thus has the maximum content and highest relevance.