Fluid Transformers and Creative Analogies: Exploring Large Language Models' Capacity for Augmenting Cross-Domain Analogical Creativity

TL;DR

This study systematically evaluates large language models' ability to generate cross-domain analogies, revealing their potential to aid creative problem solving while also highlighting associated risks like harmful content.

Contribution

It provides empirical evidence on LLMs' usefulness and risks in augmenting cross-domain analogical reasoning through three comprehensive studies.

Findings

LLMs' analogies are often helpful in problem reformulation.

Approximately 80% of analogies lead to observable problem changes.

Up to 25% of outputs may be potentially harmful, mainly due to upsetting content.

Abstract

Cross-domain analogical reasoning is a core creative ability that can be challenging for humans. Recent work has shown some proofs-of concept of Large language Models' (LLMs) ability to generate cross-domain analogies. However, the reliability and potential usefulness of this capacity for augmenting human creative work has received little systematic exploration. In this paper, we systematically explore LLMs capacity to augment cross-domain analogical reasoning. Across three studies, we found: 1) LLM-generated cross-domain analogies were frequently judged as helpful in the context of a problem reformulation task (median 4 out of 5 helpfulness rating), and frequently (~80% of cases) led to observable changes in problem formulations, and 2) there was an upper bound of 25% of outputs bring rated as potentially harmful, with a majority due to potentially upsetting content, rather than biased or toxic content. These results demonstrate the potential utility -- and risks -- of LLMs for augmenting cross-domain analogical creativity.

Tables

Table 1. Table 1 . Results of prompt engineering explorations: Average semantic similarities between original problems and GPT-3 generated analogical problems and human judgments of potential usefulness and duplication. In these runs, the one-shot learning paradigm runs produced the best combination (with an extra round with another example for verification).

	Avg. semantic	Potential	Duplicate	Unique & potential
	sim	usefulness rate	rate	usefulness rate
Zero-shot	0.72	0.68	0.24	0.43
One-shot (manager)	0.48	0.82	0.12	0.70
One-shot (Duncker) [for verification]	0.43	0.8	0.03	0.77
Few-shots	0.31	0.67	0.02	0.65

Table 2. Table 2 . Systematic coding results (adding / rejecting / adding + rejecting) of analogical reformulations in Study 1.

	Count	Median of helpfulness rating
Adding	185 (84.09%)	4
Rejecting	23 (10.45%)	4
Adding + Rejecting	22 (10%)	4
Total	220	4

Full text

Large Language Models, Analogy, Creativity Support Tools

††journalyear: 2023††copyright: rightsretained††conference: Creativity and Cognition; June 19–21, 2023; Virtual Event, USA††booktitle: Creativity and Cognition (C&C23), June 19–21, 2023, Virtual Event, USA††doi: 10.1145/3591196.3593516††isbn: 979-8-4007-0180-1/23/06††ccs: Human-centered computing Empirical studies in HCI

1. Introduction

Large language models (LLMs) such as GPT-3 are attracting attention in the creativity research community for their potential to augment creative work by generating tailored design materials and prototypes. For example, researchers have explored the potential of LLMs to replicate or assist creative writing tasks such as metaphor generation (Chakrabarty et al., 2021; Gero and Chilton, 2019), science writing (Gero et al., 2022) and storytelling (Yuan et al., 2022; Roemmele, 2016; Singh et al., 2022; Berkovsky et al., 2018; Freiknecht and Effelsberg, 2020; Osone et al., 2021; Clark et al., 2018; Cao, 2023). This research thread parallels explorations in industry, such as the “AI Dungeon” startup’s use of GPT-3 to assist with world-building in Dungeons and Dragons (Team, 2020), prototype systems for AI-assisted content creation111https://www.regie.ai/, https://www.copy.ai/, https://writer.com/ and writing assistance222https://www.compose.ai/, as well as other applications outside of creative writing, such as Q&A conversation333https://chat.openai.com/chat and code generation444https://github.com/features/copilot. Here, we are particularly interested in probing the potential of LLMs to augment cross-domain analogical creativity.

Cross-domain analogical reasoning is the core cognitive ability to perceive and reason about deep structural similarities between situations that may differ on many surface details (Holyoak and Thagard, 1996; Gentner, 1983; Hofstadter and Fluid Analogies Research Group, 1995); for example, using analogy, people can recognize a strong similarity between the solar system and the atom in terms of relational similarity (both involve a central mass — sun, nucleus — orbited by smaller bodies — planets, electrons), ignoring other surface dissimilarities such as their relative size and color. Cross-domain analogy is a frequent source of creative breakthroughs, enabling creators to develop powerful new concepts (Gentner, 2002; Holyoak and Thagard, 1996; Hargadon and Sutton, 1997; Dahl and Moreau, 2002; Chan et al., 2011) or reformulations of their creative problems (Dorst, 2015; Helms et al., 2008; Okada et al., 2009; Bhatta et al., 1994; Grace et al., 2015). As an illustrative example, Dorst (Dorst, 2015) reported a case study of how designers used an analogy to a music festival to (re)frame the problem of night violence in King’s Cross from a crime prevention / reduction problem with a new entertainment perspective.

We are motivated by the potential of LLMs to augment creative analogy-making because cross-domain analogies can be hard for humans to retrieve in the creative process. Human retrieval is highly sensitive to surface similarities, favoring “near”, or within-domain, analogs that share attributes of an object over “far”, or structurally similar analogs from different domains that primarily share relations to the object (Holyoak and Thagard, 1996; Gentner and Landers, 1985; Gick and Holyoak, 1983; Gentner et al., 1993). This can lead to creators failing to retrieve relevant analogs from other domains because they are fixated on surface features of their source problem (Linsey et al., 2008b; Linsey et al., 2006). For example, people trying to solve Duncker’s (Duncker, 1945) radiation problem —- how to remove a cancerous tumor using radiation without damaging surrounding healthy tissue — are much more likely to retrieve analogs involving cancer or radiation than an analog of an army splitting up to attack a target. The ability of LLMs to dynamically generate elaborate text tuned to specific settings presents an opportunity to complement existing structured methods of supporting creative analogy-making (Gordon, 1961), such as the WordTree method (Linsey et al., 2008a) or the TRIZ method (Savransky, 2000), which are manual and effortful, and search-based systems that may require expensive, specialized pre-processing of databases of potential analogs (Fu et al., 2013; Kang et al., 2022; Hope et al., 2017; Palani et al., 2021a; Chan et al., 2018; Vattam et al., 2011; Palani et al., 2021b).

Past work suggests that LLMs may be able to generate analogies that resemble human-generated analogies. For example, there are substantial systematic investigations of LLM performance on standard analogy completion word problems such as the SAT four-term analogy problems; an example of this task is to generate a term that answers a question like “man is to woman as king is to …?” (where a correct answer is “queen”) (Brown et al., 2020; Webb et al., 2022; Bhavya et al., 2022). More recently, some researchers have shared proof-of-concept demonstrations of LLMs’ ability to generate longer natural-language analogies (Team, 2020), explanations of analogical mappings (Webb et al., 2022; Bhavya et al., 2022), or analogy-inspired concepts for creative problems (Zhu and Luo, 2022; Zhu et al., 2023; Webb et al., 2022).

However, with some exceptions (Zhu et al., 2023), prior work lacks systematic and direct investigation of how these analogies might be useful in the creative process. We want to answer questions like, can people use LLM-generated analogies, even if they may be “incorrect” (e.g., missing key relational mappings between a source problem and target analogy), to inspire problem reformulation or ideation? If so, to what extent does this happen? Can we predict in advance which LLM-generated analogies might turn out to be useful? Given concerns about potential bias/harm in LLMs (Bender et al., 2021; Bommasani et al., 2021), how might LLM-generated analogies’ potential usefulness for augmented creativity trade off with potential harmfulness or toxicity in LLM outputs?

These questions cannot be satisfactorily answered in a traditional NLP-oriented benchmark paradigm. For example it is common to compare outputs with a “correct” reference output, as in the analogy completion word problems; this is insufficient to capture the potential usefulness of analogical inspirations that may only emerge from usage, or run through the generating and iterating on ”bad” ideas (Dix et al., 2006; Gruber and Barrett, 1974; Chan and Schunn, 2015; Kneeland et al., 2020). Crowdsourced human evaluations of LLM outputs may also index only surface-level linguistic coherence vs. more substantive dimensions of quality without more specific (task-specific) instructions: for example, Clark et al (Clark et al., 2021) reported that crowd workers mostly relied on form vs. content heuristics to make their judgments about human-likeness of LLM-generated text; more specific instructions or task framings — such as contextualizing ratings within a creative task — may be necessary to move beyond surface judgments.

In this paper, we directly investigate the potential use of LLM-generated analogies in the creative task of design problem reformulation. First, following a prompt-based learning paradigm (Liu et al., 2023) (also called in-context learning in machine learning research (Brown et al., 2020)), we crafted a structured natural-language instruction prompt for generating cross-domain analogies (including, in some cases, one or more examples of cross-domain analogies). We used these prompts to generate 480 cross-domain analogies across six design problems, and systematically explored human judgments of predicted usefulness of LLM-generated analogies (i.e., not in the context of using analogies in a creative task), along with duplicate rate and semantic distance of analogies from the design problems (prompt engineering explorations; Section 3). Second, we collected user ratings of helpfulness of a subset of these analogies in the context of a problem reformulation task, along with content analysis of user reformulation behaviors with the analogies (Study 1; Section 4). Third and finally, we manually analyzed the nature and rates of potentially harmful outputs in the analogy generations (Study 2; Section 5).

Our primary findings were as follows:

1. (1)

Prompt engineering explorations: We were able to construct one- or few-shot prompts that yielded cross-domain analogies where  70% of outputs were both unique and judged by researcher to be potentially useful for creative problem reformulation. 2. (2)

Study 1: A majority of LLM-generated analogies were rated as helpful for individuals during a creative problem reformulation task, primarily spurring new considerations for the design problems, but also encouraging shifts in problem perspectives and redefining key elements in the original problem statements. Notably, there was no correlation between a priori judgments of potential usefulness from prompt engineering explorations and the use-time ratings of analogy helpfulness here. 3. (3)

Study 2: There was some evidence of harmful or biased/toxic outputs in generated analogies (upper bound of 25% of outputs as screened by human raters), though the clear majority of potentially harmful outputs ($\sim$80%) were describing situations that could be conservatively considered upsetting under some circumstances (e.g., describing situations of poverty or difficult childbirth), rather than biased or toxic.

Overall, our findings suggest that LLM-generated analogies hold potential as a creativity support tool for cross-domain analogical problem reformulation, and extend previous demonstrations of LLMs’ capabilities with analogical reasoning for creative settings. To facilitate further analyses by the community, we also share the code used to generate the datasets in prompt engineering explorations (along with the human judgment data), and raw and coded participant responses from Study 1. We hope that the rich descriptive data provided in this paper can help researchers of creativity support tools understand how to effectively leverage LLMs to augment cross-domain analogical creativity.

2. Related Work

The field of natural language processing (NLP) has studied analogies due to their common use in language and their importance for understanding semantic relationships between words and phrases. For example, classic work in NLP focuses on the concept of word embeddings where words like king and queen are statistically likely to be used in the same contexts (Mikolov et al., 2013). Careful analyses of pre-transformer architectures, such as word2vec (Mikolov et al., 2013), demonstrated that the surprisingly high accuracy (on the order of 60%) on analogy word problems (e.g., “man is to woman as king is to ¡ans:queen¿”) hid large variations in accuracy across subtypes of analogy word problems: for example, (Chen et al., 2017) observed consistently higher accuracy on syntactic analogies (e.g., based on morphological transformations like “big is bigger as small is to ¡ans:smaller¿”) compared to semantic analogies based on causal relations (Chen et al., 2017). Some researchers were nevertheless able to leverage these models as base layers or inputs to NLP pipelines that were able to do analogical retrieval in complex natural language datasets such as crowdsourced ideas (Hope et al., 2017), and research papers (Chan et al., 2018). More recent transformer-based models, such as GPT-3, have shown improved performance on this task. For instance, the largest 175B parameter version of GPT-3 (codenamed davinci) achieved comparable performance to humans ($\sim$65% accuracy) on a set of SAT four-term analogy problems, which are considered more challenging than the analogy word problems studied in previous work (Brown et al., 2020). There is also some evidence that more recent models that add alignment training procedures (Ouyang et al., 2022), such as GPT-3’s text-davinci-003, can match or exceed human benchmark performance on these tasks (Webb et al., 2022; Bhavya et al., 2022; Ding and Chan, 2023).

More importantly for our current purposes, researchers and practitioners have begun to produce proof-of-concept demonstrations of LLMs’ ability to generate more complex natural language analogies. For instance, Zhu et al (Zhu and Luo, 2022) showed examples of using GPT-3’s earlier davinci models to generate analogous design concepts when prompted with analogies between a source problem and real-world design. Similarly, Webb et al (Webb et al., 2022) replicated the classic analogical problem solving paradigm from Gick and Holyoak (Gick and Holyoak, 1980) with the text-davinci-003 version of GPT-3, showing that the model was able to generate a convergence solution for Duncker’s radiation problem when prompted with the analogous generals story, as well as describing the analogical mapping between the problem and the analogy (though the model failed to generate a plausible analogous solution for a different physics-based problem, despite successfully describing the analogical mapping between the problem and its analogy). In a slightly different paradigm, but also with an aligned version of GPT, Bhavya et al (Bhavya et al., 2022) showed that $\sim$60% of InstructGPT (Ouyang et al., 2022)-generated analogical explanations for scientific concepts were rated by crowd workers as containing a meaningful analogy, a rate comparable to a dataset of human-generated analogical explanations. On the industry/practitioner side, the prominent AI Dungeon startup published a blog post that described its experiments using GPT-3 to generate descriptions of fantasy worlds by analogy (Team, 2020). In contrast to the simpler analogy word problems, these more complex analogy-generation capabilities have received little systematic evaluation in the context of creative tasks. One exception is a recent study by Zhu et al (Zhu et al., 2023), who followed up on their previous proof-of-concept (Zhu and Luo, 2022) by integrating 10 analogically generated design concepts into a design team’s brainstorming process, and obtained ratings of the novelty and feasibility of the concepts: the analogical concepts generally received high novelty scores but low feasibility scores.

Overall, existing research suggests that LLM-generated analogies might frequently include sufficient cross-domain analogical mappings to inspire creative problem reformulation and ideation. In this study, we seek to extend the predominantly informal, proof-of-concept demonstrations in prior work with systematic, direct evaluation of LLM-generated analogies in the context of a creative task.

3. Prompt Engineering Explorations: Analogy Generation Quality as a Function of Prompt-Based Learning Design

To develop analogies for this study, we followed a prompt-based learning paradigm (Liu et al., 2023) (also called in-context learning in machine learning research (Brown et al., 2020)): rather than fine-tuning GPT-3 for analogy generation (e.g., in a supervised learning paradigm), we “prompted” an LLM by providing a set of natural language instructions for a task of generating cross-domain analogies, and took the LLM’s generated text completion as the output. We iteratively improved our prompt through prompt engineering (Reynolds and McDonell, 2021), which involves crafting a prompt for a LLM, issuing the prompt, and evaluating the response from an LLM.

We supplemented this iterative process with a systematic evaluation of the best-performing prompt from the prompt engineering process, exploring zero-, one-, and few-shot learning variants of the prompt. The results from this systematic evaluation are reported in this section.

3.1. Task setup and prompt-based learning design

We conducted our experiments with OpenAI’s GPT-3 API, with the text-davinci-002 model, with temperature = 1 for the largest output variety and token count = 400 to accommodate the output length. Because we are interested in exploring how LLM-generated analogies could be integrated into problem formulation, we experimented with a prompt design where we provided a structure of the input problem and the output analogy by dividing a design problem into four components—stakeholder, context, goal and obstacle led to higher quality outputs. This was inspired by previous work on problem formulation in design (MacNeil et al., 2021b), and in our initial prompt engineering iterations, we informally observed that this enabled us to balance the distance (stakeholder and context are different) and usefulness (goal and obstacle are similar) of the generated analogy much better than an earlier design that prompted GPT-3 to generate an abstracted schema for a given problem statement, taking inspiration from previous work on schema-based analogical transfer (Gick and Holyoak, 1983; Yu et al., 2014). We also explored a range of zero-shot to few-shot prompt-based learning paradigms. The structure of the prompt given to GPT-3 in the experiments in this paper is shown in Figure 1. We constructed three example problem-analogy pairs to illustrate the core idea of a cross-domain analogy. Figure 2 shows the three examples we constructed.

We wrote six design problems across various topics (food insecurity, job security, entertainment, etc) as shown in Figure 3. Figure 3 also shows examples of prompt and GPT-3 generated output pairs with zero-shot, one-shot, and few-shot paradigms. We used GPT-3 to generate 20 analogies each for input problems, for each of three prompt programming paradigms: zero-shot - no example of input and output; one-shot - one example; and few-shot - three examples. To reduce the potential for overfitting on a specific example in the one-shot case, we conducted two runs of the one-shot paradigm, each with a different example (the manager and Duncker analogy examples). This process resulted in a total of 480 problem-analogy pairs across four runs (one zero-shot, two one-shot, and one few-shot). The code for these prompts, along with the 480 generated analogies, is included in the Supplementary Material.

3.2. Measures

All 480 analogies generated with different methods were rated by a single PhD-level researcher with expertise in creativity support tools, example-based creativity, and problem reformulation, who was blind to condition. Analogies were rated for potential usefulness for creative reformulation. The primary criteria were 1) whether there was an analogical mapping between the generated analogy and the design problem, 2) whether the analogy was unique or a duplicate/repeat of any other generated analogy for the same problem, and 3) a “best guess” of whether the potential inferences from the analogy might be useful for inspiring new creative formulations of the problem. Note that the “potential usefulness” criterion would be more systematically and robustly investigated in Study 2; here, we use these heuristic judgments to inform our initial prompt engineering efforts, before systematically testing the outputs from the best-performing prompt design in the context of a creative task. The researcher also measured the degree of duplication manually during screening, judging whether analogies were so similar to previously encountered analogies that it was redundant. To summarize, the rater screened the analogies and categorized them using three codes: 1) not potentially useful, 2) potentially useful and new, and 3) potentially useful but duplicated with a previous one. Examples of duplicate analogies are shown in Figure 4.

Because we are interested in the ability of LLMs to help with cross-domain analogical reformulation, we also measured the similarity of the generated analogies to the input problems, using semantic similarity calculated with SentenceTransformers555https://www.sbert.net/.

3.3. Exploration results

As shown in Table 1, our key result was that one-shot outputs were judged to be consistently more potentially useful ($\sim$80% compared to $\sim$40%) and dissimilar (¡0.5 compared to ¿0.7) from the input problems compared to zero-shot outputs. Few-shot outputs were also substantially less similar to input problems ($\sim$0.3 compared to $\sim$0.7) compared to zero-shot outputs, but the judged potential usefulness was slightly lower than one-shot outputs ($\sim$67% compared to $\sim$80%.

While we found an advantage of the one-shot paradigm over the zero-shot paradigm in terms of potential usefulness and semantic distance, we are not confident that the one-shot paradigm would consistently produce better results than few-shot paradigms in general, given the limited number of examples tested and the strong prior from the literature that in-context learning performance improves with the number of examples (Brown et al., 2020; Winata et al., 2021). We are comfortable concluding for our study that few-shot learning (with k of at least 1) is likely to yield better results than zero-shot learning; practically, because the one-shot learning paradigm required fewer examples, reducing the effort and cost associated with prompt size, we systematically evaluated the outputs of the one-shot paradigm in the context of a creative task in Study 1.

4. Study 1: Analogy Generation Quality in Usage in a Creative Problem Reformulation Task

Having established an initial estimate of the rate at which GPT-3 is able to produce potentially useful cross-domain analogies (i.e.,  70% unique and potentially useful cross-domain analogies in the best prompt design), we now directly study the degree to which these analogies could actually be useful in a creative task. To do this, we provided the analogy outputs from the best-performing one-shot learning paradigm (i.e., the prompt that included the Duncker example analogy) to people doing a creative ideation task, and asked them to try to use it to inform/inspire their creative ideation and evaluate the analogies’ usefulness for that purpose.

4.1. Methods

4.1.1. Design and Materials

To account for potential variation across people in their reactions to the analogies, we sought to obtain two judgments of usefulness per analogy. We also aimed to minimize the anchoring of our results to any particular design problem (since there was some variation in the duplicate and potential judged usefulness rate across the design problems). Finally, we wanted to balance the time requirements per rater against minimizing the potential overhead of learning the task, since using analogies for problem reformulation may be different from a more typical NLP evaluation task like judging whether a given text is human-like (though still well within the range of everyday ability). Given these requirements, we designed our overall rating task to have two analogies for each rater for the same design problem, and approximately equal representation of analogies across the six design problems. The details of the rating task are explained in the next section.

To construct sets of two analogies for each rater, we first took the 92 of the 120 total analogies from the Duncker run of the one-shot learning paradigm in prompt engineering explorations that were judged to be both nonredundant and potentially useful, as well as the 17 analogy outputs that were judged to be nonredundant cross-domain analogies but unlikely to be useful. We ignored the remaining 11 analogy outputs that were potentially useful but duplicates (N=4), highly similar to the original problem (N=4), and off-task (e.g., repeating the original problem, incorrect format; N=3). This resulted in 109 unique analogies across the six problems. To enable construction of pairs of analogies, we then randomly selected one of the analogies from a design problem that had an uneven number of usable analogies to drop from the sample. This resulted in a final set of 108 analogies. From these 108 analogies, we then constructed all possible pairwise combinations of analogies within each design problem, and sampled iteratively from these combinations to select pairs, such that each of the 108 analogies showed up in exactly 2 pairs. This resulted in 108 analogy pairs that were given to raters.

4.1.2. Task and Procedure

The overall rating task was designed to follow substeps of analogical problem reformulation, beginning with initial formulation of an incompletely specified problem, initial ideation, processing of an analogical stimulus in relation to the problem, and then reformulation and ideation over the reformulated problem. As an example, a designer given a problem of designing ways for people who want to find interesting places to visit and have fun but lack information and resources about nearby places might (re)formulate the problem by adding information about the stakeholder(s) (e.g., thinking about friends and family who might have good recommendations) or rejecting/shifting a potentially implicit assumption that they need to discover new places, and instead explore new goals around re-experiencing familiar places. This reformulation might be spurred by comparison to analogous problems, such as a parent trying to find ways to entertain their toddler during summer vacations (e.g., by considering how children can often repeatedly enjoy similar experiences or toys in a variety of ways).

Thus, the overall task was divided into three phases: an initial formulation phase, and two analogical reformulation and ideation phases, one for each analogy in the pair assigned to the participant. In the initial formulation phase, participants were first given a design problem and asked to construct an initial formulation of the problem through adding details to the problem components (stakeholder/context/goal/obstacle), in response to the question, “What characteristics of the stakeholder/context or other problem components do you think would be important to consider when trying to solve the problem?”. Then, participants were asked to generate at least one solution to that problem with details added. In each of the two analogical reformulation and ideation phases, participants were presented with an analogy, and then answered two questions 1) “What interesting commonalities and differences do you notice between the problems (the original and analogical problems)?” 2) “Based on your comparison of the original problem with the new problem, can you think of ways to change your initial understanding of the original problem that might lead to new solutions? What might you add/change to your understanding of the problem components?” The first question was meant to facilitate analogical/case comparison, a known effective strategy for inducing analogical schemas for ideation (Gick and Holyoak, 1983; Loewenstein, 2010; Kurtz et al., 2013). The second question was meant to probe for potential reformulation of the problem based on the analogical mapping. After this, participants were asked to come up with a new solution to the original problem. To get a more realistic sense of the potential value of the analogies for ideation, we did not force participants to generate ideas in this phase from the analogies: participants were allowed to use Google to search for inspiration to generate ideas. After each task, participants were asked to rate how helpful they found it to compare their problem situation with the analogous one from 1 - “Not helpful at all” to 5 - “Very helpful” (no rating was 0). This process was then repeated for the second analogy in the set. After the second analogical reformulation and ideation phase, participants had a chance to give open-ended comments on the study. Appendix A shows screenshots of the task interface for the initial formulation and analogical reformulation and ideation phases of Study 1. We obtained institutional IRB approval for the project prior to Study 1.

4.1.3. Participants

We recruited participants from Prolific666https://www.prolific.co/ as Prolific participants typically exhibit greater care in completing tasks compared to Amazon MTurk participants (Oppenlaender et al., 2020). We limited all participants to U.S. residents. Each participant was paid $5.25 ($10.5 per hour) for their participation of 30 mins. A total of 123 participants enrolled the task, but 13 of them only finished the first round of analogical reformulation and ideation and did not complete the second round. After removing those participants, our final sample included 110 participants who finished the study. Due to our randomized procedure for assigning participants to analogy sets, removing incomplete responses also resulted in 22 analogies being used and evaluated by only one participant.

4.1.4. Systematic analysis of reformulation behaviors

To understand how participants used the analogies to reformulate the original problems, we systematically coded participants’ responses to the reformulation question in the analogical reformulation and ideation phases:

“Based on your comparison of the original problem with the new problem, can you think of ways to change your initial understanding of the original problem that might lead to new solutions? What might you add/change to your understanding of the problem components?”

We wanted our coding scheme to capture a core distinction in the creativity literature between formulation and reformulation. The former — formulation — entails adding details to to transform an ill-structured problem into a more structured problem that can make the search for solutions more tractable. This process is necessary because creative problems are not static “givens”; rather, these problems are ill-structured and malleable. In contrast to well-structured problems (e.g., the Tower of Hanoi, algebra), where the initial description of the problem sufficiently delineates the goal state and allowable transformations, in creative problem domains, such as innovative design or interdisciplinary research, the “problem statement” typically underspecifies what the desired solution is. In information-processing terms, “the start state is incompletely specified, the goal state is specified to an even lesser extent, and the transformation function from the start to goal states is completely unspecified” (Goel and Pirolli, 1992). Analogical transfer can be instrumental in this elaboration process, adding functional decompositions or subproblems to solve that might address the high-level problem (Helms et al., 2008). The latter — reformulation — entails reshaping an existing structured problem, often by rejecting or relaxing constraints and assumptions, or even reformulating the goal itself. An innovative designer can choose to see the design problem differently, to relax certain constraints or redefine certain customer needs in other terms, thereby arriving at innovative solutions. Such problem-reframing has been characterized as key element of creative expertise in design (Cross and Cross, 1998; Schön, 1983). For example, Bucciarelli (Bucciarelli, 1994) observed in his ethnographic studies of engineering designers that a common strategy was to re-evaluate and sometimes alter formulations of the design problem. Goel and Pirolli (Goel and Pirolli, 1992) call this a “reversal of the transformation function”, where, instead of reducing the difference between the current and goal states by means of a design “move” (e.g., a particular product configuration), a designer moves closer to the goal state by changing what the goal state is. In insight problem solving, too, a key mechanism for moving past design impasses, is to relax constraints and assumptions (Knoblich et al., 1999), or even change what details of the problem are attended to, which can be characterized as a search not for solutions, but for problem spaces, where solutions are more readily found (Kaplan and Simon, 1990). Here, too, analogy can be used to support reformulation; For example, (Dorst, 2015) described a case where Designers used an analogy to a music festival to (re)frame the problem of night violence in King’s Cross from a crime prevention/reduction problem to an entertainment frame.

Thus, we initially employed two codes - ”adding” and ”shifting” - to capture notions of ”formulation” (elaborating/adding), and ”reformulation” (changing/rejecting/shifting). However, we discovered that the ”shifting” code also involved instances of ”adding,” and disagreements arose regarding whether a reformulation constituted ”adding,” ”shifting,” or ”adding” plus ”shifting.” This resulted in low inter-rater reliability. To address this, we decomposed the ”shifting” code into two more concrete and reliable codes: ”adding” and ”rejecting”, which were not mutually exclusive, and (if both are present) could still capture the notion of shifting. Our final coding scheme thus consisted of two independent codes: 1) adding (0/1), indicating any presence of new concepts or descriptions in the answer compared to the initial problem statement or formulation; and 2) rejecting (0/1), indicating direct statements from the participant that they are rejecting a goal/obstacle/context/ stakeholder from the problem or their initial formulation. A problem reformulation could be seen as a combination of adding and rejecting. Examples of adding and rejecting are provided below:

Example of adding

Original problem

Stakeholder: people who stay at home for a long time and want to travel

Context: international travel is restricted under the pandemic

Goal: find interesting places to visit, eat and have fun

Obstacle: lack of information and resources about nearby places

Analogy

Stakeholder: a software company

Context: developing a new mobile application

Goal: find out user requirements

Obstacle: most of the users are not familiar with the application’s purpose

Participant’s response to reformulation question

I would add specific locations to the problem because that would narrow down the scope of how to get information to them.

Example of rejecting

Original problem

Stakeholder: owners of travel agency

Context: the restriction of pandemic has been mitigated and people are willing to travel again

Goal: reopen their traveling business

Obstacle: cannot find enough employees because people have left the travel industry during the pandemic

Analogy

Stakeholder: a farmer

Context: the restriction of the use of pesticides has been mitigated

Goal: use pesticides to increase crop yield

Obstacle: the farmer cannot afford to buy pesticides

Participant’s response to reformulation question

It’s not that the travel agency can’t find employees, it’s that they can’t afford to pay employees to work for them after being closed for so long, thus causing a feedback loop of: not enough employees -¿ less money -¿cant afford to hire employees -¿ not enough employees.

Two graduate-level researchers with expertise in creativity support tools, example-based creativity, and problem reformulation, applied the coding scheme to the participants’ reformulation responses. After five rounds of coding, discussing (and resolving) disagreements, and refining the coding scheme between two expert coders, the inter-rater reliability reached Cohen’s Kappa=1 (perfect agreement) for adding codes and Cohen’s Kappa=0.847 (near perfect agreement) for rejecting codes respectively for 50 problem-analogy pairs (Landis and Koch, 1977). After developing the substantial inter-rater reliability, one coder coded the remaining 170 reformulation responses.

4.2. Quantitative Results

All 220 analogy-response pairs from all completed participants (N=110) are included in the Supplementary Material. We first report a range of quantitative analyses of LLM-generated analogies’ quality at a whole sample level, before describing some illustrative case studies of how the analogies were processed and used to contextualize the high level results.

4.2.1. Median 4 out of 5 helpfulness rating for analogy, with increased rating on the second analogy

Since analogies received unequal numbers of ratings, we report the median as an estimate of the central tendency of ratings across analogies. The median rated helpfulness of analogies was 4 out of 5 (see Table 2 and Figure 5). Interestingly, we noticed that ratings were higher for the analogies that were rated second in the analogical reformulation phase ($median=4$) compared to the analogies that were rated first ($median=3$). A Wilcoxon test (within subjects, two groups) estimated that this increase in median ratings from the first to second analogy was statistically significant ($statistic=561.5,p-value=0.002$). We also compared the median of helpfulness ratings for analogies judged by two or more participants and the median of helpfulness ratings for all analogies to check whether there was a difference between only one judgment vs. multiple judgments: both medians were 4, suggesting that we obtained similarly reliable estimates from single vs. multiple-judgments for each analogy

4.2.2. Frequent analogical reformulations, primarily by adding information, positively correlated with helpfulness ratings

The results of our systematic coding of the reformulation responses are reported in Table 2, showing 185/220 (84.09%) instances of adding, and 23/220 (10.45%) instances of rejecting. There was high overlap between instances of adding and rejecting: specifically, there were 22/220 (10%) instances of both adding and rejecting (shifting).

Since the add/reject/any change codings were binary and the helpfulness ratings were continuous, we calculated point-biserial correlations between them. We observed a statistically significant positive correlation between reformulation responses coded as having an addition and helpfulness ratings for the corresponding problem-analogy pair ($point-biserial=0.2559,p=0.0001$) and between any change codings (add or reject) and helpfulness ratings ($point-biserial=0.2669,p=0.00006$). However, we observed no statistically significant correlation between reject codings and helpfulness ratings ($point-biserial=0.0094,p=0.8891$).

4.2.3. No correlation between a priori usable ratings and in-use helpfulness ratings

To explore the correspondence between the a priori judgments of potential usefulness from prompt engineering explorations (outside the context of a creative task) with in-use ratings of usefulness of analogies, we calculated the correlation between the potential usefulness ratings from prompt engineering explorations and the helpfulness ratings here. Since the potential usefulness ratings were binary and the helpfulness ratings were continuous, we calculated a point-biserial correlations between them. We observed no statistically significant correlation between a priori usable ratings and helpfulness ratings: $point-biserial=-0.0266,p=0.6939$. However, since we only utilized one researcher’s ratings to predict the usefulness ratings provided by another set of participants, we cannot discount the possibility that the lack of correlation might be due to alternative explanations. For instance, it could be that the specific rater was unable to predict usefulness without actually using the analogies, or that the difficulty of predicting usefulness for someone else contributed to the lack of correlation.

4.3. Qualitative descriptions of how people used LLM-generated analogies to reformulate problems

To complement our systematic quantitative analyses of participants’ reformulation behaviors in 4.1.4, we illustrate how participants used analogies with two more cases shown in Figure 6. In the first example of the “local tourism” problem, P48 wanted to gain more information of the context such as external resources and constrains (”resources or loans”, “willingness of people”, etc). P48 described additional contexts that were not included in the original design problem, such as “resources or loans that were available” and “willingness of people… to tolerate health risks.” After seeing the analogical problem that a “child is resistant to taking medicine and going to bed”, P48 highlighted the role of persuasion to solve the original problem and incorporated personal persuasion into the solution. P48 did not explicitly reformulate the original problem, but instead began ideating a solution, which is a common behavior for novice (MacNeil et al., 2019; MacNeil et al., 2021a). But the solution suggested that P48 noticed a potential core obstacle “a lack of personal persuasion” within the original problem that employers “cannot find enough employees because people have left the travel industry during the pandemic”. This is interesting because “the child is resistant to taking medicine and going to bed” can be seen as an interesting and plausible analogy to “previous employees of the travel industry who are resistant to return”. In the second example, the original problem formulation of P109 was around the stakeholder. After seeing the analogy, P109 noticed the high-level similarity between “lack of entertainment information for people who stay at home” and “noisy environment of a student’s house” and then explicitly reformulate the root problem as “they’re not getting out of their homes enough”. Those two examples demonstrated that the generated analogies were able to facilitate participants to deepen their understanding of the original problem and reformulate the root issues, which could inform better solutions.

Complementing these example cases are some notable open-ended comments from participants about how the analogies impacted their formulation and ideation processes. For example, P40, P29, and P109 commented on how the analogies opened their mind:

Comparing the two problems opened new mental doors, only one of which led to a practical, workable solution. But the fact that those mental doors were completely invisible to me before seeing the analogous problem suggests that looking at related but not identical problems can be extremely helpful. (P40)

I think reading from analogies can definitely open your mind up to new ideas and improve critical thinking skills. I didn’t even realize at first that it was “people” in Philadelphia, until the next slide. I totally overlooked it, but it’s interesting how so many things correlate, even if they’re very different. (P29)

This was extremely interesting for me! I really like that I was able to compare problem situations with the analogies here and expand my original thinking. This allowed me to identify new, creative solutions to the original problem that I would not have thought of without the expanded analogies! (P109)

Similarly, P79 noted:

These juxtapositions create very higher order thinking involving public policy expertise as well as scientific and community knowledge. Very interesting! These creative solutions are necessary to improve social welfare and environmental problem solving. Thank you! (P79)

Another participant’s comment also illuminated the lack of correlation between pre-judged potential usefulness and helpfulness in the context of the creative task:

Overall, I found the analogies to be a bit random and not the best fit, but they did make me think of the problem from a slightly different angle, which was interesting.” (P68).

Overall, these qualitative descriptions and open-ended comments complement the quantitative results above by adding more concrete details to illustrate how the LLM-generated analogies were useful in the creative problem formulation task.

5. Study 2: Potential Harmfulness and Toxicity of Generated Analogies

In this final section, we systematically investigate the degree to which GPT-generated analogies might be harmful or toxic. We do this because we are interested in real-world usage of a method like this, and want to facilitate weighing of cost-benefit considerations and design of mitigations for potential harms, especially given concerns about potential harms from large language model usage (Bender et al., 2021).

Our analysis here is anchored on the potential use case of giving LLM-generated analogies to designers as an aid to problem formulation. Inspired by previous analyses of the ethics of LLMs based not just on direct harm to the user but also participation in harmful and oppressive systems of power (Blodgett et al., 2020; Bender et al., 2021), we consider the following primary categories of harm: 1) perpetuating hegemonic systems of oppression via uncritical incorporation of harmful stereotypes and biases in LLM outputs, 2) unknowingly framing problems in a way that leads to solutions that violate laws, and 3) experiencing psychological distress from upsetting or abusive content.

Analyzing the bias and stereotype-based categories of harm typically involves consideration of specific social categories at risk of harm from biases, such as gender (Sheng et al., 2019), or disability (Hutchinson et al., 2020). As Bender et al (Bender et al., 2021) note, which particular social categories are salient is culture-bound. Thus, we note here that we anchored our conceptualization of protected social categories from a US-centric perspective due to our positionality as US-based researchers, and considered the dimensions of disability, race/ethnicity, gender, sexual orientation and age. To analyze potential illegal suggestions, we considered the context of US laws, again due to our positionality. Finally, we intentionally drew on raters from a variety of demographic backgrounds of race/ethnicity and gender, to allow for more expansive detection of content that may be considered psychologically distressing. We note all these to emphasize the situatedness of our results, and make no claims as to their generality to other contexts, and share our raw outputs to assist with extensions of this work that consider this question from other positionalities.

From a measurement perspective, harmfulness audits have often been done using automated measures such as the Perspective API777https://www.perspectiveapi.com/ toxicity classifier (Bender et al., 2021). However, the sensitivity and accuracy of these measures is uncertain, and there is evidence that they may produce false positives with mere mentions of terms associated with marginalized identities, such as race or disability (Jigsaw, 2018; Hutchinson et al., 2020). Thus, we chose to manually review all outputs, and estimate ranges in rates of harmfulness or toxicity, as well as qualitatively describe patterns of outputs on these dimensions.

5.1. Methods

Three members of the research team (1 White male junior faculty, 1 Asian immigrant male junior faculty, and 1 Asian immigrant male PhD student), and two additional research assistants (1 Asian female undergraduate student, 1 North African female PhD student) reviewed all 108 analogies used in Study 1. We report the general demographics of the screeners to facilitate understanding of how our results here on the potential harmfulness of the outputs may be shaped by our positionality, recognizing that other researchers or users in different positionalities may reach different conclusions about the LLM outputs.

Using the three-fold conceptualization of harm described above (bias, illegality, and potential psychological distress), each screener independently went through each analogy, and flagged it as potentially harmful or not. Screeners also added descriptive comments about potential harm, where appropriate. We use the harmful/ not harmful flags to estimate rates of harmfulness, and the descriptive comments to qualitatively describe the nature of potential harms.

5.2. Results

There was wide variation in the number of flagged analogies across screeners, ranging from 1 to 27 out of 108 total analogies: specifically, the number of flagged analogies across the five screeners was, in ascending order: 1, 2, 2, 10, and 27. Using the most expansive approach of counting any flag by any screener increased the upper bound rate to 28/108 (26%).

Qualitatively, the majority of the analogies that were flagged across screeners were for potentially upsetting content, rather than biased/abusive language or illegal behavior. Figure 7 shows the 10 analogies that were flagged by at least 2 of the 5 raters as potentially harmful, along with the associated screener notes. There was one clear instance of biased depiction of people in a small village in Africa with multiple stereotypes of Africa as poverty-ridden and a ”wild safari”. There was also one instance of illegal behavior described (smuggling liquor in the 1920’s). The remaining 8 analogies were flagged for potentially upsetting content, such as difficult birthing/pregnancy, refugees fleeing their country, trying to escape an abusive relationship, substance addiction, and war. In the subset of screening results from the screener who flagged the most analogies, we also observed a similar pattern where predominantly potentially upsetting content accounted for most of the potentially harmful analogies.

6. Discussion

6.1. Summary and interpretation of findings

In this paper, we directly investigated the potential of LLMs to generate analogies that might be useful for creativity support, specifically in the task context of augmenting cross-domain analogical reformulation. Through carefully crafted prompts, we were able to use prompt-based learning to generate cross-domain analogies with GPT-3 that were semantically distant from source problems and judged to be potentially useful at a high rate ($\sim$70-80% of outputs; prompt engineering explorations). We complemented this prompt engineering exploration with a systematic analysis of how people used LLM-generated analogies in the context of a creative problem reformulation task, and found that they were able to leverage the analogies to reformulate design problems ($\sim$80% of cases), rated the analogies overall as helpful for reformulating the problems (average $\sim$3-4 Likert rating out of 5), and described the analogies as augmenting/broadening their thinking process (Study 1). Investigating potential tradeoffs against toxic/bias/harmful outputs, we found an upper bound of $\sim$25% of outputs being potentially harmful in some way, with a clear majority ($\sim$80%) of such cases being judged as such due to descriptions of potentially upsetting situations, rather than biased or toxic descriptions of specific social categories.

We infer from these analyses that LLMs, with carefully crafted prompts, can frequently generate analogies that can augment cross-domain analogical reformulation. The cost/benefit analysis of deploying these analogies in creativity support tools may vary by situation given the potential for upsetting content. However, note that our screeners were emphasizing minimizing false negatives, and many problematic situations may inherently contain content that may be upsetting to some (since they, by definition, describe situations that people want to transform to “better” states (Newell and Simon, 1972; Rittel and Webber, 1973; Dorst, 2015)).

More broadly, our results add to prior work on the analogical reasoning capacities of LLMs, extending from simpler analogy word problems (Brown et al., 2020; Webb et al., 2022; Bhavya et al., 2022) to more complex cross-domain analogical outputs, and extending prior proofs-of-concept (Webb et al., 2022; Bhavya et al., 2022; Zhu and Luo, 2022) with systematic testing of analogical outputs in the context of creative tasks. Our prompting methodology with structured inputs also has the potential to be extended to other domains with repeating structural components, such as scientific reasoning (e.g., theory, evidence, method) or legal reasoning (e.g., precedent, defendant, issue). Additionally, our observed lack of correlation between a priori judgments of potential usefulness from prompt engineering explorations and helpfulness ratings and reformulation behaviors in the context of an actual task underscore the importance of going beyond simple benchmark-based measures of output “quality” (Lee et al., 2022), or even human judgments of quality separate from usage in a creative task (Clark et al., 2021) (although, as we note in the results section for Section 4, we cannot rule out alternative explanations such as lack of ability for this specific rater, or the possibility of successfully predicting usefulness for oneself, but not others). The significant increase in helpfulness ratings from the first to second analogy in the analogical reformulation phases in Study 1 also suggest that some degree of “settling into” a creative analogy-making task is needed to get more accurate measures of the quality of LLM-generated analogies; this effect is reminiscent of prior results showing the induction of a “relational mindset” from initial processing of cross-domain analogies that improved subsequent recognition and processing of cross-domain analogies (Vendetti et al., 2014; Goldwater and Jamrozik, 2019).

6.2. Limitations and next steps

6.2.1. Beyond the one-shot prompt-based learning paradigm

In this paper, we only sampled from one run of the one-shot paradigm with one example for Study 1’s in-use evaluation due to the cost of human evaluation. This made sense given that prompt engineering explorations showed an advantage of the one-shot paradigm over the zero-shot paradigm in terms of uniqueness and judgments of potential usefulness, and comparable uniqueness and potential usefulness to the few-shot paradigm. However, while we are fairly confident that the one-shot paradigm produces better results than the zero-shot paradigm, we are not confident that one-shot paradigms would be consistently better than few-shot paradigms, given the small number of examples we tested, and the substantial literature on how in-context learning improves with the number of examples (Brown et al., 2020; Winata et al., 2021). A potential direction for future work is to expand on different instantiations of few-shot learning paradigms. For example, future work might explore the effect of providing uniform or diverse examples on the usefulness of generated analogies. It may also be fruitful to more systematically explore variations of prompt programming, such as more specific, varied, or longer / more complex prompts, as well as investigating the extent to which a more traditional fine-tuning approach may yield substantially better results. That said, from a practical standpoint, one-shot learning paradigms have the benefit of requiring less examples, which reduces prompt engineering effort and costs associated with prompt size.

6.2.2. Improving the depth of insight in LLM-generated analogies

While the rate of generating cross-domain analogies was relatively high, it is unclear to what extent these analogies could spark deep insights or highly creative conceptual leaps. Note that the rate of rejecting/shifting elements of the problem in participants’ reformulation (10.45%) was fairly low. This relative lack of rejecting/shifting may reflect the particulars of the task and participants in Study 1. But we also wonder if, as others have suggested (Webb et al., 2022), the insightfulness of LLM-generated analogies may be bounded by the degree to which the model has access to rich domain knowledge, such as commonsense or physical knowledge. In this sense, LLMs’ capacity for analogy generation observed here may be more a function of what Mahowald et al (Mahowald et al., 2023) call “formal linguistic competence” (knowledge of rules and patterns of a given language), as opposed to deeper “functional linguistic competence” (being able to understand and use language in real-world tasks, such as formal reasoning, situation modeling, and social reasoning. This general idea is consistent with the informal observations of the screener for prompt engineering explorations, who noticed that many of the analogies seemed to be quite diverse, but didn’t describe deep details about any particular domain. We wonder what the LLM-generated analogies might inspire if they were generated from a combination of LLMs and more structured/specialized knowledge bases (e.g., modeling common sense, domain ontologies, or physical world modeling); a recent proof-of-concept of something like this is a prototype using ChatGPT as a natural language interface, drawing from the much more structured and curated computational knowledgebase of Wolfram Alpha (Wolfram, 2023).

6.2.3. From static to iterative LLM-assisted cross-domain analogical reasoning

Our experiments were conducted with a static set of pre-generated examples for the zero- and few-shot prompts. We wonder how LLM-powered analogy generation might be integrated into the more iterative nature of design and creative cognition (Dorst and Cross, 2001). For instance, rather than using a fixed design problem as a source for LLM-generated analogies, could we integrate users’ reformulations into subsequent prompts? Might the information of current user path enable LLMs to provide more personalized creative support for taking a deep dive or a “creative leap” (Chan et al., 2017; Siangliulue et al., 2015; Noy et al., 2012)? We are curious how future work might fruitfully build on explorations by (Zhu et al., 2023), who used LLMs not just to generate analogies, but also generate explanations for potential mappings in an analogy, and generate potential concept ideas. We can imagine design patterns where an initial round of LLM-generated analogies helps to stimulate memory retrieval of diverse domains — to assist with fixation (Linsey et al., 2008b; Linsey et al., 2006) and surface similarity bias in analogical retrieval (Holyoak and Thagard, 1996; Gentner and Landers, 1985; Gick and Holyoak, 1983; Gentner et al., 1993) — and then subsequent integration of LLM-assisted “deep dives” via explanations of potential mappings, generating variations of ideas within a theme, and so on.

Another interesting direction may be to leverage LLM-generated analogies to improve the process of iteration from feedback (Chan et al., 2016; Dow et al., 2010; Chan et al., 2015). Studies of analogies in creative work show that they are useful for more than concept generation and problem reformulation: analogies are frequently used to assist with explaining unfamiliar concepts (Dunbar, 1997; Bearman et al., 2002; Christensen and Schunn, 2007). For example, the deployable space array technology is more explainable with an origami-folding analogy (Bolanos et al., 2022). We wonder how AI-generated analogies could be used by creative workers not only to inspire themselves, but also to assist them in conveying design concepts to others by connecting to their domain knowledge.

Acknowledgements.

This research was supported in part by ONR N000142012506.

Appendix A Screenshots of User Interface for Study 2