To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes

TL;DR

This study compares how CS2 students learn programming using ChatGPT versus traditional search engines, revealing differences in ease of learning and query strategies across methods.

Contribution

It provides empirical insights into students' learning strategies and effectiveness when using LLMs compared to traditional online resources.

Findings

Students find traditional methods easier for difficult concepts.

Students ask fewer follow-up questions with search engines.

Students use more keyword-based queries with search engines.

Abstract

LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students' experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.