When Experimental Economics Meets Large Language Models: Evidence-based Tactics

TL;DR

This paper explores how principles from experimental economics can guide the design of experiments with large language models, providing practical tactics to improve their reliability and scope.

Contribution

It introduces seven practical tactics for designing and implementing LLM experiments, bridging experimental economics and AI research.

Findings

Key considerations significantly affect LLM responses

Seven practical tactics improve experiment reliability

Enhanced guidelines for replicability and generalizability

Abstract

Advancements in large language models (LLMs) have sparked a growing interest in measuring and understanding their behavior through experimental economics. However, there is still a lack of established guidelines for designing economic experiments for LLMs. Inspired by principles from experimental economics with insights from LLM research in artificial intelligence, we outline key considerations in the experimental design and implementation stage, and perform two sets of experiments to assess the impact of these considerations on LLMs' responses. Based on our findings, we discuss seven practical tactics for conducting experiments with LLMs. Our study enhances the design, replicability, and generalizability of LLM experiments, and broadens the scope of experimental economics in the digital age.