Evolving Paradigms in Task-Based Search and Learning: A Comparative Analysis of Traditional Search Engine with LLM-Enhanced Conversational Search System

TL;DR

This paper compares traditional search engines with LLM-enhanced conversational systems to understand how AI influences search behavior, learning outcomes, and user interaction in information retrieval and knowledge acquisition.

Contribution

It provides an empirical analysis of user search strategies and learning effects in traditional versus LLM-powered search environments, highlighting new insights into human-AI interaction.

Findings

LLM systems alter query formulation and search strategies.

LLM use improves comprehension and knowledge integration.

Traditional search remains effective for straightforward information retrieval.

Abstract

Large Language Models (LLMs) are rapidly reshaping information retrieval by enabling interactive, generative, and inference-driven search. While traditional keyword-based search remains central to web and academic information access, it often struggles to support multi-step reasoning and exploratory learning tasks. LLM-powered search interfaces, such as ChatGPT and Claude, introduce new capabilities that may influence how users formulate queries, navigate information, and construct knowledge. However, empirical understanding of these effects is still limited. This study compares search behavior and learning outcomes in two environments: a standard search engine and an LLM-powered search system. We investigate (1) how search strategies, query formulation, and evaluation behaviors differ across systems, and (2) how LLM use affects comprehension, knowledge integration, and critical thinking during search-based learning tasks. Findings offer insight into how generative AI shapes information-seeking processes and contribute to ongoing discussions in information retrieval, human-AI interaction, and technology-supported learning.