A Simple Mechanism for Focused Web-harvesting

TL;DR

This paper presents a straightforward, customizable web-harvesting mechanism that enhances automated indexing by specifying targeted URLs and human-edited parameters, demonstrated through implementation in Indonesia's scientific index.

Contribution

The paper introduces a simple, flexible web-harvesting system with customizable parameters for focused data collection, improving efficiency and accuracy over traditional methods.

Findings

Implemented in Indonesia's Scientific Index

Parameters improve harvesting precision

System is accessible via web interface

Abstract

The focused web-harvesting is deployed to realize an automated and comprehensive index databases as an alternative way for virtual topical data integration. The web-harvesting has been implemented and extended by not only specifying the targeted URLs, but also predefining human-edited harvesting parameters to improve the speed and accuracy. The harvesting parameter set comprises three main components. First, the depth-scale of being harvested final pages containing desired information counted from the first page at the targeted URLs. Secondly, the focus-point number to determine the exact box containing relevant information. Lastly, the combination of keywords to recognize encountered hyperlinks of relevant images or full-texts embedded in those final pages. All parameters are accessible and fully customizable for each target by the administrators of participating institutions over an integrated web interface. A real implementation to the Indonesian Scientific Index which covers all scientific information across Indonesia is also briefly introduced.