A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians

The document explores the use of large language models (LLMs) as creativity support tools for comedy writing, evaluating their effectiveness and ethical considerations through surveys, focus group discussions, and quantitative analysis with professional comedians and performers.

Abstract

We interviewed twenty professional comedians who perform live shows in front of audiences and who use artificial intelligence in their artistic process as part of 3-hour workshops on ``AI x Comedy'' conducted at the Edinburgh Festival Fringe in August 2023 and online. The workshop consisted of a comedy writing session with large language models (LLMs), a human-computer interaction questionnaire to assess the Creativity Support Index of AI as a writing tool, and a focus group interrogating the comedians' motivations for and processes of using AI, as well as their ethical concerns about bias, censorship and copyright. Participants noted that existing moderation strategies used in safety filtering and instruction-tuned LLMs reinforced hegemonic viewpoints by erasing minority groups and their perspectives, and qualified this as a form of censorship. At the same time, most participants felt the LLMs did not succeed as a creativity support tool, by producing bland and biased comedy tropes, akin to ``cruise ship comedy material from the 1950s, but a bit less racist''. Our work extends scholarship about the subtle difference between, one the one hand, harmful speech, and on the other hand, ``offensive'' language as a practice of resistance, satire and ``punching up''. We also interrogate the global value alignment behind such language models, and discuss the importance of community-based value alignment and data ownership to build AI tools that better suit artists' needs.

Method

The authors employed a mixed-methods approach, combining quantitative and qualitative research methods to evaluate the use of large language models (LLMs) as creativity support tools for comedy writing. The methodology included a writing exercise using LLMs, followed by surveys to assess participants' experiences with the AI system and calculate the Creativity Support Index (CSI) of the writing tool. Additionally, focus group discussions were conducted to explore the usefulness of LLM-generated outputs, the comedy writing process, and various biases and stereotypes associated with LLMs. The study also involved the analysis of transcripts from the focus groups to identify major themes and insights. The authors used a combination of Likert scale surveys, psychometric surveys, and open-ended questions to gather data and insights from the participants. Additionally, the study considered ethical considerations, such as participant anonymity and the potential adverse impacts of the research.

Main Finding

The authors' discoveries encompassed several key findings related to the use of large language models (LLMs) as creativity support tools for comedy writing. These findings included concerns expressed by participants regarding copyright and data ownership, particularly in light of recent litigation against technology companies training LLMs on copyrighted data. The study also highlighted the economic and artistic concerns related to the potential devaluation of human-authored works and the displacement of labor by LLMs. Additionally, the research emphasized the need for ethical considerations, such as the disclosure of AI origin in text or images, and the importance of integrating relational context and empowering artists to contribute to the development of LLMs aligned with specific audiences. The study also revealed the challenges associated with AI's ability to understand and generate humor, as well as the potential biases and stereotypes of LLMs. Furthermore, the research provided insights into the participants' experiences with LLMs, including their enjoyment of writing with AI, concerns about ownership, and the mixed nature of their responses regarding collaboration and the uniqueness of AI-generated content. The study's quantitative results indicated a mediocre Creativity Support Index (CSI) score, reflecting the participants' varied experiences with LLMs for creative writing. Overall, the authors' discoveries encompassed a range of ethical, economic, and creative considerations related to the use of LLMs in comedy writing.

Conclusion

The conclusion of the paper "A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs’ Humour Alignment with Comedians" highlights the need for ethical considerations and the importance of integrating relational context when using large language models (LLMs) for creative writing, particularly in the field of comedy. The study emphasizes the challenges associated with LLMs' ability to understand and generate humor, the concerns over copyright and data ownership, and the potential biases and limitations of LLMs. It also suggests avenues for improving LLMs to better serve the needs of artists, such as conceptualizing LLMs aligned with specific audiences, integrating necessary relational context, and empowering comedians to reclaim ownership of the tools and the processes for gathering and curating training data for these models. The paper does not seek to endorse any specific perspective but rather presents an exploration of external viewpoints, emphasizing the importance of considering diverse opinions in creative communities and the potential for future research to explore the diversity of opinions across a broader range of familiarity with AI tools and openness to using them in creative practices.

Keywords

Large Language Models, Comedy, Creativity, Offensive speech, Censorship, Value Alignment