Using Exclusive Web Crawlers to Store Better Results in Search Engines' Database

TL;DR

This paper introduces an exclusive web crawler approach that improves search engine database accuracy and update efficiency by storing site-specific data tables, reducing costs and preventing outdated results.

Contribution

It proposes a novel crawling method that stores site-specific data in dedicated tables, enhancing data accuracy and update speed in search engine databases.

Findings

Improved data accuracy in search engine results.

Reduced crawling and updating costs.

Elimination of 404 errors in search results.

Abstract

Crawler-based search engines are the mostly used search engines among web and Internet users, involve web crawling, storing in database, ranking, indexing and displaying to the user. But it is noteworthy that because of increasing changes in web sites search engines suffer high time and transfers costs which are consumed to investigate the existence of each page in database while crawling, updating database and even investigating its existence in any crawling operations. "Exclusive Web Crawler" proposes guidelines for crawling features, links, media and other elements and to store crawling results in a certain table in its database on the web. With doing this, search engines store each site's tables in their databases and implement their ranking results on them. Thus, accuracy of data in every table (and its being up-to-date) is ensured and no 404 result is shown in search results since, in fact, this data crawler crawls data entered by webmaster and the database stores whatever he wants to display.