Beyond Pipelines: A Fundamental Study on the Rise of Generative-Retrieval Architectures in Web Research

TL;DR

This paper surveys how large language models, especially retrieval-augmented generation, are transforming web research by replacing traditional pipelines with generative approaches for various tasks, highlighting recent advances and future challenges.

Contribution

It provides a comprehensive overview of the rise of generative-retrieval architectures in web research, emphasizing recent developments, challenges, and future directions.

Findings

LLMs significantly influence web research practices.

Retrieval-augmented generation enhances information retrieval.

Open challenges include scalability and accuracy.

Abstract

Web research and practices have evolved significantly over time, offering users diverse and accessible solutions across a wide range of tasks. While advanced concepts such as Web 4.0 have emerged from mature technologies, the introduction of large language models (LLMs) has profoundly influenced both the field and its applications. This wave of LLMs has permeated science and technology so deeply that no area remains untouched. Consequently, LLMs are reshaping web research and development, transforming traditional pipelines into generative solutions for tasks like information retrieval, question answering, recommendation systems, and web analytics. They have also enabled new applications such as web-based summarization and educational tools. This survey explores recent advances in the impact of LLMs-particularly through the use of retrieval-augmented generation (RAG)-on web research and industry. It discusses key developments, open challenges, and future directions for enhancing web solutions with LLMs.