Dialogue with the Machine and Dialogue with the Art World: Evaluating Generative AI for Culturally-Situated Creativity

TL;DR

This paper introduces a dialogue-based evaluation method for generative AI in culturally-situated art practices, emphasizing social context and expert engagement, demonstrated through a case study in the Persian Gulf art scene.

Contribution

It proposes a novel dialogue approach involving artists and art world experts to assess AI's cultural relevance, expanding beyond traditional evaluation methods.

Findings

Enhanced understanding of AI's cultural impact in non-western art contexts

Demonstrated value of dialogue-based evaluation in capturing socio-cultural nuances

Facilitated artist adaptation of AI tools to their cultural practices

Abstract

This paper proposes dialogue as a method for evaluating generative AI tools for culturally-situated creative practice, that recognizes the socially situated nature of art. Drawing on sociologist Howard Becker's concept of Art Worlds, this method expands the scope of traditional AI and creativity evaluations beyond benchmarks, user studies with crowd-workers, or focus groups conducted with artists. Our method involves two mutually informed dialogues: 1) 'dialogues with art worlds' placing artists in conversation with experts such as art historians, curators, and archivists, and 2)'dialogues with the machine,' facilitated through structured artist- and critic-led experimentation with state-of-the-art generative AI tools. We demonstrate the value of this method through a case study with artists and experts steeped in non-western art worlds, specifically the Persian Gulf. We trace how these dialogues help create culturally rich and situated forms of evaluation for representational possibilities of generative AI that mimic the reception of generative artwork in the broader art ecosystem. Putting artists in conversation with commentators also allow artists to shift their use of the tools to respond to their cultural and creative context. Our study can provide generative AI researchers an understanding of the complex dynamics of technology, human creativity and the socio-politics of art worlds, to build more inclusive machines for diverse art worlds.