MacBehaviour: An R package for behavioural experimentation on large language models

TL;DR

MacBehaviour is an R package designed to facilitate behavioral experiments on large language models, enabling researchers to systematically study LLMs' responses and behaviors across multiple models with streamlined tools.

Contribution

The paper introduces MacBehaviour, a comprehensive R package that simplifies and standardizes the process of conducting behavioral experiments on over 60 large language models.

Findings

LLMs exhibit human-like gender inference from names.

The package successfully replicates previous findings on LLM behavior.

Demonstrated utility across multiple LLMs in validation experiments.

Abstract

There has been increasing interest in investigating the behaviours of large language models (LLMs) and LLM-powered chatbots by treating an LLM as a participant in a psychological experiment. We therefore developed an R package called "MacBehaviour" that aims to interact with more than 60 language models in one package (e.g., OpenAI's GPT family, the Claude family, Gemini, Llama family, and open-source models) and streamline the experimental process of LLMs behaviour experiments. The package offers a comprehensive set of functions designed for LLM experiments, covering experiment design, stimuli presentation, model behaviour manipulation, logging response and token probability. To demonstrate the utility and effectiveness of "MacBehaviour," we conducted three validation experiments on three LLMs (GPT-3.5, Llama-2 7B, and Vicuna-1.5 13B) to replicate sound-gender association in LLMs. The results consistently showed that they exhibit human-like tendencies to infer gender from novel personal names based on their phonology, as previously demonstrated (Cai et al., 2023). In summary, "MacBehaviour" is an R package for machine behaviour studies which offers a user-friendly interface and comprehensive features to simplify and standardize the experimental process.