Evaluating the role of `Constitutions' for learning from AI feedback

TL;DR

This study examines how different written guidelines, called constitutions, influence the quality of AI-generated feedback in medical interview communication, revealing strengths in emotional aspects but limitations in practical skills.

Contribution

It provides empirical evidence on the impact of constitution detail level on feedback quality in AI training for medical communication.

Findings

Detailed constitutions improve emotive feedback quality.

Constitutions do not outperform baseline in practical skill improvement.

AI feedback may have limitations in certain skill areas.

Abstract

The growing capabilities of large language models (LLMs) have led to their use as substitutes for human feedback for training and assessing other LLMs. These methods often rely on `constitutions', written guidelines which a critic model uses to provide feedback and improve generations. We investigate how the choice of constitution affects feedback quality by using four different constitutions to improve patient-centered communication in medical interviews. In pairwise comparisons conducted by 215 human raters, we found that detailed constitutions led to better results regarding emotive qualities. However, none of the constitutions outperformed the baseline in learning more practically-oriented skills related to information gathering and provision. Our findings indicate that while detailed constitutions should be prioritised, there are possible limitations to the effectiveness of AI feedback as a reward signal in certain areas.