AutoS$^2$earch: Unlocking the Reasoning Potential of Large Models for Web-based Source Search

TL;DR

AutoS$^2$earch is a novel framework that uses large models for zero-shot source search in web-based systems, mimicking human reasoning to improve safety and efficiency without relying on crowdsourcing.

Contribution

This work introduces AutoS$^2$earch, a new approach leveraging large models and visual-to-language conversion for autonomous source search in web applications.

Findings

Achieves near-human performance in source search tasks.

Eliminates need for crowdsourcing in hazard detection.

Demonstrates effectiveness in industrial safety scenarios.

Abstract

Web-based management systems have been widely used in risk control and industrial safety. However, effectively integrating source search capabilities into these systems, to enable decision-makers to locate and address the hazard (e.g., gas leak detection) remains a challenge. While prior efforts have explored using web crowdsourcing and AI algorithms for source search decision support, these approaches suffer from overheads in recruiting human participants and slow response times in time-sensitive situations. To address this, we introduce AutoS$^2$earch, a novel framework leveraging large models for zero-shot source search in web applications. AutoS$^2$earch operates on a simplified visual environment projected through a web-based display, utilizing a chain-of-thought prompt designed to emulate human reasoning. The multi-modal large language model (MLLMs) dynamically converts visual observations into language descriptions, enabling the LLM to perform linguistic reasoning on four directional choices. Extensive experiments demonstrate that AutoS$^2$earch achieves performance nearly equivalent to human-AI collaborative source search while eliminating dependency on crowdsourced labor. Our work offers valuable insights in using web engineering to design such autonomous systems in other industrial applications.