A Comparative Study of Hidden Web Crawlers

TL;DR

This paper reviews various hidden web crawlers that access data behind search forms, highlighting their techniques, advantages, and limitations, to improve web data accessibility.

Contribution

It provides a comparative analysis of existing hidden web crawlers, detailing their methodologies and identifying gaps for future research.

Findings

Different hidden web crawlers use diverse form learning techniques.

Most crawlers face challenges with form complexity and data freshness.

The paper highlights the need for more adaptive and efficient crawling strategies.

Abstract

A large amount of data on the WWW remains inaccessible to crawlers of Web search engines because it can only be exposed on demand as users fill out and submit forms. The Hidden web refers to the collection of Web data which can be accessed by the crawler only through an interaction with the Web-based search form and not simply by traversing hyperlinks. Research on Hidden Web has emerged almost a decade ago with the main line being exploring ways to access the content in online databases that are usually hidden behind search forms. The efforts in the area mainly focus on designing hidden Web crawlers that focus on learning forms and filling them with meaningful values. The paper gives an insight into the various Hidden Web crawlers developed for the purpose giving a mention to the advantages and shortcoming of the techniques employed in each.