Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems

TL;DR

This paper evaluates the ability of large language models to reverse-engineer black-box systems, showing that active intervention improves understanding and offers insights into overcoming common failure modes.

Contribution

It demonstrates that prompting LLMs to actively intervene enhances their reverse-engineering capabilities and provides practical strategies for AI-driven scientific discovery.

Findings

Active prompting improves LLM reverse-engineering performance.

Interventions help LLMs avoid overcomplication and overlooking failures.

Sharing intervention data among LLMs further enhances understanding.

Abstract

Using AI to create autonomous researchers has the potential to accelerate scientific discovery. A prerequisite for this vision is understanding how well an AI model can identify the underlying structure of a black-box system from its behavior. In this paper, we explore how well a large language model (LLM) learns to identify a black-box function from passively observed versus actively collected data. We investigate the reverse-engineering capabilities of LLMs across three distinct types of black-box systems, each chosen to represent different problem domains where future autonomous AI researchers may have considerable impact: Program, Formal Language, and Math Equation. Through extensive experiments, we show that LLMs fail to extract information from observations, reaching a performance plateau that falls short of the ideal of Bayesian inference. However, we demonstrate that prompting LLMs to not only observe but also intervene -- actively querying the black-box with specific inputs to observe the resulting output -- improves performance by allowing LLMs to test edge cases and refine their beliefs. By providing the intervention data from one LLM to another, we show that this improvement is partly a result of engaging in the process of generating effective interventions, paralleling results in the literature on human learning. Further analysis reveals that engaging in intervention can help LLMs escape from two common failure modes: overcomplication, where the LLM falsely assumes prior knowledge about the black-box, and overlooking, where the LLM fails to incorporate observations. These insights provide practical guidance for helping LLMs more effectively reverse-engineer black-box systems, supporting their use in making new discoveries.