TEC: A Collection of Human Trial-and-error Trajectories for Problem Solving

TL;DR

This paper introduces TEC, a dataset capturing human trial-and-error problem-solving trajectories, to improve AI's ability to learn from human strategies in complex tasks.

Contribution

The paper presents a new data annotation platform and dataset capturing detailed human trial-and-error trajectories, filling a gap in AI training data.

Findings

Humans outperform LLMs in trial-and-error accuracy.

The dataset includes 5,370 trajectories and 41,229 webpage reflections.

TEC platform and dataset are publicly available.

Abstract

Trial-and-error is a fundamental strategy for humans to solve complex problems and a necessary capability for Artificial Intelligence (AI) systems operating in real-world environments. Although several trial-and-error AI techniques have recently been proposed, most of them rely on simple heuristics designed by researchers and achieve limited performance gains. The core issue is the absence of appropriate data: current models cannot learn from detailed records of how humans actually conduct trial-and-error in practice. To address this gap, we introduce a data annotation platform and a corresponding dataset, termed Trial-and-Error Collection (TEC). The platform records users' complete trajectories across multiple trials and collects their reflections after receiving error feedback. Using this platform, we record the problem-solving processes of 46 participants on 58 tasks, resulting in 5,370 trial trajectories along with error reflections across 41,229 webpages. With this dataset, we observe that humans achieve substantially higher accuracy compared to LLMs, which demonstrates that humans are more effective in trial-and-error than LLMs. We believe that the TEC platform and dataset provide a valuable foundation for understanding human trial-and-error behavior and for developing more capable AI systems. Platform and dataset are publicly available.