Facilitating Human Feedback for GenAI Prompt Optimization

TL;DR

This paper explores how different human feedback methods can improve the optimization of Generative AI prompts, emphasizing the potential of comparative feedback to enhance evaluation quality.

Contribution

It introduces a human-AI training loop and compares feedback strategies, highlighting the benefits of comparative feedback for prompt optimization.

Findings

Comparative feedback encourages more nuanced evaluations

Preliminary results suggest improved human-AI collaboration

Further research needed with larger samples

Abstract

This study investigates the optimization of Generative AI (GenAI) systems through human feedback, focusing on how varying feedback mechanisms influence the quality of GenAI outputs. We devised a Human-AI training loop where 32 students, divided into two groups, evaluated AI-generated responses based on a single prompt. One group assessed a single output, while the other compared two outputs. Preliminary results from this small-scale experiment suggest that comparative feedback might encourage more nuanced evaluations, highlighting the potential for improved human-AI collaboration in prompt optimization. Future research with larger samples is recommended to validate these findings and further explore effective feedback strategies for GenAI systems.