Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts | Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

2.1 The Promises of Non-Expert Prompt Design

Today’s chatbot design practice—i.e., designing multi-turn conversational interactions—follows a well-established workflow [12, 13, 24, 25, 39, 44]. Designers first (i) identify the chatbot’s functionality or persona and draft ideal user-bot conversations, for example, through Wizard-of-Oz or having experts drafting scripts; (ii) create a dialogue flow template (e.g., “(1) greeting message; (2) questions to collect user intention; (3)...”); (iii) fill the template with supervised NLP models (e.g., user intent classifier, response generator, etc.); and finally (iv) iterate on these components to achieve a desired conversational experience. In this “supervised learning” paradigm, designers make NLP models generate their desired interactions by improving its training data and feature design; tasks that require substantial machine learning and programming knowledge [50].

The emergent “pre-train, prompt, predict¹” paradigm in NLP promises to lower the entry barrier for non-experts innovating on conversational interactions [30]. In this paradigm, designers can create conversational agents with little to no training data, programming skills, or even NLP knowledge. Leveraging pre-trained large language models such as ChatGPT [1], designers can create a general-purpose conversational system with passable, though sometimes problematic, performance. Next, they can improve the LLM outputs using natural language prompts (see Table 1 for an example) and/or model fine-tuning. In this paradigm, people without programming skills or NLP knowledge can nonetheless make NLP models generate desired interactions by crafting effective prompt strategies (e.g., natural language instructions, examples, and templates) [3, 15].

2.2 Known Challenges in Prompt Design

While prompting can appear as easy as instructing a human, crafting effective and generalizable prompt strategies is a challenging task. How a prompt or a prompt strategy directly impacts model outputs, and how prompts modify LLMs’ billions of parameters during re-training, are both active areas of NLP research [30, 42]. Moreover, established prompt design workflows do not yet exist. Even for NLP experts, prompt engineering requires extensive trial and error, iteratively experimenting and assessing the effects of various prompt strategies on concrete input-output pairs, before assessing them more systematically on large datasets.

That said, ongoing NLP research does offer some hints toward effective prompt design strategies. Notably, these prompt strategies are effective in improving LLMs’ performance over a range of NLP tasks and conversational contexts; it remains unclear to what extent these strategies can improve any particular conversational interactions or contexts.

Little research has investigated how non-experts conduct prompt engineering intuitively or the challenges they encounter. One rare exception is a CHI’22 workshop paper “How to prompt?” [14], where Dang et al. conducted a focus group with HCI researchers and identified an initial set of challenges they encountered in prompting. These challenges include “the lack of guidance in trial and error,” “poor representation of tasks and effects,” and “computational costs and ethical concerns.” In prompting text-to-image models such as DALL · E, an adjacent area to prompting LLMs, Liu et al. describe a study with artists, identifying emergent strategies for prototyping prompts, such as “try multiple generations to get a representative idea of what prompts return” and “focus on keywords rather than the phrasings of the prompt” [31]. This paper builds upon this emergent line of research and aims to deepen these understandings.

2.3 Non-Expert Prompt Design Tools

Arguably, the most widely used prompt design tool among non-experts remains the OpenAI Playground, a “plain text” input box that triggers GPT-3 to predict text completions. The Playground interface includes over 50 example prompts and enables anyone to experiment with different prompts for accomplishing various tasks. Though useful, Playground lacks any support for systematic evaluation of prompts.

Propelled by the potential of non-expert prompt design, a rapidly growing set of end-user-facing LLM-based applications is emerging from HCI research, many with features that support prompt engineering. Such applications span across LLM-enhanced story writing and programming [23, 38, 45, 48]. NLP researchers have also started creating tools to enable anyone to contribute prompts to help train LLMs [3, 15].

Interestingly, these tools offer little to no support for non-experts in generating or evaluating prompts, often assuming that looking at individual input-output pairs in isolation is sufficient. One telling example is AI Chains [49], a tool for exploring human-LLM collaborative writing interactions. AI Chains allows designers to construct a chain of LLMs where the output of one LLM becomes the input for the next, and to test the resulting interactions themselves. The tool successfully enables designers to explore prompt and chaining strategies more efficiently and strategically [48]. However, it is unclear whether the resulting strategies are effective or robust beyond the few interaction contexts that the designers experimented with.

2.4 Known Challenges in Non-Expert Programming and Machine Learning

Early HCI research tells us that program-authoring interactions that do not require programming are not necessarily accessible to non-programmers; drag-and-drop and interactive machine learning (iML) tools cannot necessarily enable non-ML experts to build models [50], and prompting can be viewed as a programming or iML task. In this context, one might ask: does prompt engineering similarly involve tacit knowledge that non-AI experts do not have? What mental models might hinder non-AI experts’ ability to devise robust prompt strategies and create desired conversational interactions? Research on prompt design has not yet asked these questions.

Here, we briefly overview known barriers and challenges in end-user programming and iML (list below [27]) as well as how experts and non-experts approach these tasks differently (Table 2), to motivate and contextualize our investigation into end-users’ intuitive approaches to prompt design.

In this work, we use these previously-identified challenges to better understand why we observe some of the struggles with end-user prompt engineering that we do.

3.1 Designing a No-Code Prompt Design Tool as Probe

Design Goals. We have two goals in designing BotDesigner: First, to enable end-users, without any prompt design experience or even programming or ML experience, to (i) create a chatbot using prompts, without code, and (ii) systematically evaluate their prompts. Second, to allow end-users to engage in this process in a flexible, open-ended way, allowing us to observe their intuitive behaviors and thought processes.

Supporting a full chatbot design workflow, while adhering to a prompt-only, no-code format, BotDesigner has to support two separate, complementary activities:

Figure 1:

Related work in tool design. . Two challenges stand out on the path to achieving these goals: (i) How can BotDesigner allow users to observe a prompt’s impact on a single conversation? (ii) How can BotDesigner enable users to inspect a prompt’s impact on a full set of conversations, while each of these conversations unfolds differently depending on the user’s utterances and the LLM’s probabilistic outputs?

To address the challenge in tracking prompt’s effects, BotDesigner builds on tool design in prior interactive ML research (see [16] for a review). Explainability and model manipulation tools for text and code generation have used a variety of interactions for exploring underlying reasons for specific behavior, such as the “questioning” approaches by Liao et al. [29] and iterative dialog approaches by Lakkaraju et al. [28].

To address the challenge of visualizing and analyzing multiple divergent conversations, BotDesigner draws inspiration from prior work that tackles analogous challenges in computer-generated images by supporting side-by-side comparisons. One early interactive system for exploring the design space of computer-generated images, Marks et al.’s Design Galleries [33] enables users to explore a space of images generated by a specific set of parameters specifying lighting and other image features on a rendering tool. Modern takes on this work have traded CGI parameters for deep model hyperparameters, such as Carter et al.’s Activation Atlas [10] for inspecting image classifiers, Evirgen and Chen’s GANzilla [18] for searching a GAN-based image generator’s latent space using scatter/gather interactions [21], and Zhang and Banovic’s interactive image galleries for sampling from a GAN. These tools, and the concepts they embody, directly informed the design of BotDesigner.

Figure 2:

BotDesignerDesign. . BotDesigner has two interfaces: a Conversation view and an Error Browser. Using the Conversation view, end-user chatbot designers can create a chatbot by authoring a set of natural language prompts, referred to as prompt template (see Appendix A.1 for a concrete example of a prompt template and its resulting chatbot). Designers can explore how any given prompt template behaves as a chatbot by engaging in a conversation with the chatbot defined by that template (Figure 1). Specifically, the prompt template consists of several text fields: (i) A preamble that consists of direct natural language instruction to the bot; (ii) A fixed set of first turns of conversation, causing all conversations with this bot to start with this specific set of messages; and (iii) A reminder that is included in the LLM prompt immediately before LLM is queried for its next conversational turn.

Using BotDesigner’s Error Browser, designers can run a new or modified prompt template against a full set of prior conversations. The system then displays a list of how the LLM-produced chat messages change given the new bot template (Figure 2).

BotDesignerImplementation. BotDesigner is implemented as a React-based web application with a node.js-based backend, relying on OpenAI’s GPT-3 API and its text-davinci-002 model as the underlying LLM. Much of the implementation of the application consists of standard CRUD-style techniques; we offer a detailed description of BotDesigner’s implementation and pilot usability study in Appendix A.

Of note, we selected text-davinci-002 because it was the most capable model available at the time of our study, in particular, the most capable at taking instruction in natural language with fewer in-line examples. Because we expect the costs of large model API calls to decrease over time, we did not consider performance/cost tradeoffs, reasoning that future models will likely have different tradeoffs and that today’s “most capable” models will be next year’s intermediate models in any case. Indeed, as of this writing, OpenAI has already released text-davinci-003 and ChatGPT—models that will certainly have different capabilities.

3.2 BotDesigner at Work: an Example Prompt-Based Chatbot Design Process

To demonstrate the full potential of BotDesigner, consider the following example: an end-user/designer, Alex, wishes to create a chatbot that walks the user through cooking a recipe (Mixed Veggie Tempura), answers any questions they might have, and engages in social chit-chat if needed. Alex also plans to use the chatbot via voice while cooking.

This workflow (reproduced visually in the Appendix as Figure 5) mirrors HCI’s traditional iterative prototyping process in many ways; however, it differs from almost all previous non-expert prompt design tools, which focused solely on ad-hoc experimentation (steps 1-2).

It is worth noting that although BotDesigner enables this full workflow, we do not necessarily expect all end-users to engage in all its steps. However, we did validate BotDesigner’s functionality and usability throughout this workflow via a set of pilot user studies (see Appendix A.3 for details) to ensure that these pieces are available should users choose to systematically assess their prompt designs.

3.3 User Study Design

To understand how end users intuitively approach prompt design, we invited 10 non-expert prompt designers to use BotDesigner and think-aloud. We invited the participants to improve upon a baseline chatbot template towards given goals (see Tasks below), while we observed and took note of the various prompt design approaches and testing strategies they engaged in. We chose the task of improving a chatbot, instead of creating it from scratch, in order to avoid the Blank Page Syndrome, the effect where users staring at a blank page do not know where to begin.

Participants. . All participants (N = 10) had little to no experience with prompt design; none had substantial prior experience working with LLMs. We recruited participants with varying levels of programming and chatbot design experience. Recall that our focus here is understanding end-users in the domain of prompt design; experience in chatbot design was thus not considered grounds for exclusion. Our sample size was chosen in line with prior work around formative testing for usability [19, 43]: our goal is to explore the first, “easy-to-find” problems users encounter when engaged in our task, and our experience with a pilot user suggested that problem discoverability was likely to be high.

Our participant pool, skewed towards professionals and graduate students in STEM-related fields, is not representative of the population at large. As a result, we cannot (and will not) make claims about the relative prevalence of specific behavior or beliefs among the general population; instead, we identify behaviors and intuitions among a population that we expect is disproportionately likely to be early adopters of LLM-based tools—and some of our findings are especially surprising given the high level of technical literacy among our participant pool.

Task. . Participants were asked to recreate, in chatbot form, a professional chef that walks an amateur (the chatbot’s user) through the various steps of cooking a recipe. This task was inspired by the web series Back to Back Chef, in which Bon Appétit Test Kitchen chef Carla Lalli walks celebrities through cooking a specific recipe “through verbal instructions only.” [2] Carla can see neither the celebrities that she is instructing nor their actions; all information is communicated by voice, making this web series an ideal source material for a purely-language-medium chatbot. Participants’ ultimate goal is to modify the chatbot’s utterances and behavior to better match Carla Lalli’s performance in the series: engaging in humor and social conversation, making analogies, simplifying complicated concepts, asking for confirmation that specific steps have been completed, and performing a few other types of actions found in the web series.

We operationalized the decision to chose a chatbot improvement task rather than a chatbot creation task by providing the baseline bot template pre-loaded in the BotDesigner interface, instead of asking participants to generate one. This baseline bot can successfully list out ingredients and steps, but in general does not engage in the behaviors described above.

In selecting this particular chatbot improvement task, we considered (and even piloted) a few other chatbot design tasks, including an open-ended “design a chatbot” task, without any particular target or baseline chatbot goals. We offered a few baseline chatbot templates and asked users to come up with a chatbot idea and implement it, but ultimately, our pilot users’ behaviors were dominated by open-ended exploration of the baseline bot’s capabilities, and they did not engage very much in iterative prompt design. Understanding GPT-3’s capabilities was a non-goal of our study, and we found too few instances of prompt design for the amount of effort our pilot participants were engaged in. We ultimately selected the Carla chatbot (“CarlaBot”) design task because it captures a number of behaviors that are (i) hard to do with traditional NLP, that (ii) GPT-3 and other LLMs are nominally capable of, making them motivating and persuasive for our participants, and that are (iii) not present in the baseline bot, requiring prompt iteration to achieve.

Interview Protocol. . We showed users a specific episode of Back to Back Chef in which Carla walks actress Elizabeth Olsen through the preparation of Mixed Veggie Tempura [2]. After watching the first few minutes of the episode, we showed participants a transcript of that video, containing within it a set of color-coded highlights that we drew participants’ attention to verbally and then described. These highlights identified several specific instances of Carla engaging in humor and social conversation, making analogies, simplifying complicated concepts, asking for confirmation that specific steps have been completed, and performing a few other types of actions that are atypical of cookbook recipes—and unlike what one might expect a typical voice assistant to perform. The first few turns of this transcript appear in Figure 3, and the full transcript is provided in Appendix B.

Figure 3:

We asked participants to use BotDesigner to improve on a baseline chatbot we provided (see appendix; Figure 4b). We began by describing the components of the interface, showing where the designer could have a conversation with the chatbot under development, as well as identifying the preamble, first turns, and reminder components of the bot template and explaining their connection to the chatbot, as well as identifying the “start a new conversation” button.

We encouraged participants to first evaluate what the baseline bot template is (and is not) capable of, and then asked participants to try to prompt the bot to perform the additional types of conversational tasks that Carla engages in, but that are absent from the baseline. Participants were allowed up to 1 hour to get as far as they could on this task, and we communicated that we were most interested in observing how they set about this task, and not how well they managed to do so; we also set expectations that completing this task may not be possible at all, let alone in the allotted time. Participants were allowed to explore the tool in an open-ended manner and attempt multiple different strategies to complete the task.

In a few cases, described here, we informed or reminded participants of specific functionality of BotDesigner later in the interview, at a certain stage of the participant’s progress. First, when participants identified an error, we would point out the labeling functionality, and show how to use the systematic error testing functionality. Second, when participants modified a bot template and expressed a wish to evaluate it, with a question like “how do I see what this change does?”, we would additionally point out the “edit” and “replay” local testing functionality.

The interview proceeded in two phases. For the first 10-15 minutes of the prompt design task, we did not offer participants suggestions for prompt design unless we observed zero prompt design attempts in a span of several minutes; at those points where participants got stuck for several minutes, in the interest of observing subsequent behavior, we would make a suggestion for a particular prompt to try, and noted which participants were unable to attempt prompt design without assistance (RQ1).

After this initial phase, again in the interest of observing subsequent behavior, we would offer suggestions for prompts to try that were in line with the research around prompt design. Examples of this advice include “try reframing that ‘do not’ statement into a positive statement, say what it is you want the bot to say, not what you don’t want it to say” or “have you considered copying some of the Carla-Lizzie transcript directly into the preamble?” We did not intentionally vary this advice across participants; however, due to the unpredictable nature of which errors participants chose to focus on, not all participants received the same advice or advice in the same order, a limitation of our approach.

Analysis & Evaluation. . We asked participants to think aloud, then engaged in exploratory data analysis, transcribing all videos and observing what prompt design approaches participants attempted, and noting when participants appeared to struggle and where interviewers intervened. Two of the authors then compared these approaches across participants and categorized their struggles using affinity diagrams [34] and by creating a service blueprint [5] that documents the specific approaches individual participants attempt. We chose these methods from HCI and the field of service design because our focus is on discovering opportunities created by a new technology, rather than understanding how an existing set of users engages in established work practices—a context in which a grounded theory approach would be more conventional.

4.1 Overview: Non-Experts’ Approach to Prompt Design

In our open-ended prompt design task, participants almost exclusively took an ad hoc, opportunistic approach to prompt exploration.

Before engaging in the following flow, participants were asked to watch a short segment of the Back to Back Chef episode described in §3.3, shown a transcript of that episode, led through a description of a few types of desirable human behaviors Carla engages in (e.g., analogies, concept simplification, confirmation of recipe steps), and asked to modify the baseline chatbot’s instructions in order to replicate some of those behaviors in CarlaBot.

From that starting point, the typical flow is:

About half of participants used the reminder field, which starts empty. One participant just wanted the bot to “Confirm that the user has completed the step before starting the next.”

Participants did not make much use of the error labeling functionality, and as a result also did not find the systematic testing functionality useful. Instead, participants engaged primarily in repeated local testing, without consideration for whether changes made later in the prompt design process reintroduced earlier, previously-solved errors. This local testing was typically conducted by using the “retry” button on the specific problematic utterance the participant was trying to resolve, or by starting new conversations.

4.2 Challenges in End-User Prompt Design

Though we report on a wide spectrum of our participants’ struggles iterating on prompts, we found that many of these behaviors arose from two fundamental sources: over-generalization from limited experience, and a social lens that filtered participants’ prompts (and the system’s responses to those prompts) through expectations originating in human-human interactions.First, participants over-generalized from single data points, whether positive or negative. For example, when writing a prompt to generate specific behavior, participants often stopped iterating once the behavior was observed in a single conversational context, without considering other conversations or contexts—or gave up too early if the behavior was not observed on a first or second attempt, and assumed the system was incapable of generating that behavior. Second, participants filtered their prompts and observations through a lens based on behavioral expectations drawn from human-human interactions, in some cases so strongly that they avoided effective prompt designs even after their interviewer encouraged their use and demonstrated their effectiveness. For example, all participants held a strong bias for direct instruction over providing examples in-line, and when encouraged to give examples, most participants did, and noted their effectiveness, but none subsequently made substantial use of examples in their prompt designs. Other effects of the human-human interaction expectations included some participants’ beliefs that the chatbot would “understand” the instructions provided, not considering the instruction merely as priming a language model—resulting in surprise that a prompt like “Do not say ABC” leads to the chatbot saying ABC verbatim. In the following sections we illustrate these behaviors with example prompts and quotes drawn from our interviews, explicitly connect behaviors with the causes described above, and draw parallels where these behaviors echo challenges described in the end-user programming systems and interactive machine learning literature.

4.3 Impacts: Challenges’ Effects on Prompt Design

In this section, we describe and illustrate the specific behaviors and effects we observed in our participant interviews, organized by subtask: generating prompts, evaluating prompt effectiveness, and explaning system behavior.

5.1 What Do People Currently Do? Implications for Training and Education

One major opportunity for training and education revealed by our study is that end-users should collect more data than they are naturally inclined to. Systematic testing and robust prompt design—designing prompts that work across many users and many conversations—are by definition impossible if the user only ever engages in a single conversation, even continuously updated. That said, for some contexts non-robust prompt design can be appropriate: chatbots intended for a single user who is also the designer do not necessarily need to be robust.

A second major opportunity stems from the fact that most of the prompt design approaches participants tried before receiving any guidance (see §3.3 for details regarding this guidance) had some effect—enough for participants to feel that they could, in fact, affect chatbot output with prompt changes—but participants hit dead ends that incurred interviewer intervention more frequently than we expected.

A searchable repository of examples, such as The DALL·E 2 Prompt Book [37], could help users overcome the “design”, “selection”, and “use” barriers that often led to the aforementioned dead ends. In particular, these examples should include specific prompts that work, and the contexts in which they work, so users can set their own expectations around when to try what kind of instruction, or what kind of sample dialog (or other input-output pair) to include, how much to repeat themselves, how much emotion to give, etc. Examples would also help users develop reasonable expectations about whether a specific type of prompt design should work in a given context—and if not, whether rephrasing or repositioning that prompt would help.

Third, in our related work, we identified a few prompt design approaches that recent literature supports as effective, including the use of example input/output pairs (such as sample conversational turns, in the context of chatbots) and the use of repetition within prompts. We found, however, that some users avoided these known-effective strategies, sometimes even when encouraged by the interviewer to try them. This may be a result of what communications researcher Clifford Nass identifies as a “computers are social actors” (CASA) [35] effect; that line of work shows that humans are cautious about giving feedback to a computer system they are actively interacting with, despite recognizing the computer as incapable of feeling judged. One finding of Nass’s work is that if the human’s instructions to the computer are perceived by the human as applying not to the computer the human is interacting with, but rather to a third party that is not present, then humans can avoid social pitfalls—that is, avoid behaving as though the chatbot is a social actor they’re instructing. Achieving this may be a matter of combining training with specific tool design; we offer some possible direction in the next section.

Fourth, some users, especially those with limited programming experience, struggled to recognize the ways in which the instructions they gave in the preamble, in the reminder, or in conversation with the chatbot had different time horizons. Instructions in conversation, in particular, had no bearing on any future conversations; the prompt sent to GPT-3 consists only of the preamble, the single conversation in progress, and the reminder—prior conversations are not “remembered” in any sense. Though this problem is likely especially pronounced for chatbots that present a social, interactive interface, making sure that users understand the specific lifecycle over which instructions have an effect is important to avoid frustrations arising from the feeling that the user has to repeat themselves or that they “told you this already”.

Finally, we conclude this section on training by drawing on our finding that participants in general chose not to engage in systematic testing: for users to engage in systematic testing of prompts, we must train and encourage users to do so.

5.2 Opportunities for Design

How might future tools be designed to help users avoid some of the learning barriers and pitfalls?

To avoid struggles around having enough data for systematic testing, future tools might benefit from explicitly supporting robustness testing and demonstrating its value—a set of heuristics here will likely be very useful. These heuristics would ideally help the user understand:

To help users avoid EUPS-style struggles, developers can take a few steps. First, tool designers should consider how to make the successful examples described earlier discoverable and even salient in the moments when users most need them.

Second, tool support for explicit explorations of cause-and-effect, such as through highlighting prompt changes and the resulting conversational changes, would help users overcome the “understanding” and “information” barriers. Indeed, a number of participants with programming experience specifically requested just such an interface for BotDesigner.

Though examples and instructions to the designer will hopefully help, application designers can also try to hide the direct nature of the instructions-to-the-bot experience to mitigate the impacts of the social expectations we associated above with the CASA paradigm. Some methods of achieving this might include:

Importantly, we note that several of these design opportunities explicitly surface the non-humanness of the app to the user or suggest ways for users to consider general. While visions of new technologies often imagine “seamless” interactions where the technology behind interactions is made invisible and fades into the background [47], “seamful” design interactions (originating from ubiquitous computing) can make the technology and its imperfections visible to the user [11, 22]. In this case, we note that rather than making the app interface try to act more human, there may be useful design opportunities by making the app interface act explicitly in non-human ways.

5.3 Limitations

We do not claim here that BotDesigner or even LLMs in general should replace contemporary chatbot design techniques, but we do observe that the simplicity and accessibility of prompt-instructed chatbots suggests that we may be on the cusp of an explosion of end user-created chatbots and other similar systems that rely solely on human natural language prompting for their behavior. In this paper, we use chatbots as an example domain to explore the interactions between end user designers and prompt-based systems backed by LLMs (as BotDesigner is).

A few additional limitations of our study bear exploration, and point to future studies. First, the reliance on a relatively small pool of likely “early adopters” drawn from academia may skew the observed phenomena towards a specific set of analogies and preconceived notions around technology and its capabilities. We claim neither that our observations are universal of all end users, nor that they are a complete description of all common misconceptions or behaviors of end-users working with LLM-backed systems. Additionally, the time we gave participants to engage in prompt design was relatively limited, and it is likely that, given enough time and motivation, end users would advance quite quickly in developing better mental models and more effective strategies. Ultimately, we hope that our observations offer fruitful and interesting directions for future work, including to what extent our findings generalize to other domains (e.g., non-chatbots, where users’ social expectations may be less pronounced) and a broader population, as we describe in §5.4 Open Questions.

While this paper explores the positive potential for LLM-backed systems to be more widely accessible to a broader range of users, we acknowledge that prior research also highlights the potential for negative or complex social effects related to LLMs (such as implications for intellectual property rights and authorship, online harassment, or the loss of jobs due to automation). While a deeper discussion of these issues is beyond the scope of this paper’s use of an LLM system as a probe, we surface these as a reminder of additional considerations that may be necessary when developing LLM-backed systems that are meant to be deployed “in the wild.”

5.4 Open Questions & Future Work

In addition to the implications for training and design given above, our work reveals a few questions we feel warrant future study:

First, to what extent do the challenges and behaviors we describe here also appear in other populations that might use these tools? A larger study, looking at a more diverse population than we have here, will likely uncover new challenges and can speak to the degree to which the behaviors we observe are universal (or not!) in the general public. Further: which challenges persist, and what new challenges emerge, when users engage in natural language tasks in other domains, including non-chatbot domains? Chatbot “programming” involves giving instructions to an agent, a context in which social expectations might be much more pronounced than when, say, prompting a poetry generator or a search tool—while the EUPS and iML struggles, like overgeneralization, seem more likely to persist.

Relatedly, to the extent the challenges we describe here—or new ones uncovered by future work—can be addressed through the mechanisms we offer in §5.2, which approaches are most effective, and why?

Second, can we quantify the extent to which users’ prompts and interactions are actually impacted by social expectations? Nass’ CASA work suggests the effect could be substantial, and if that proves true, then we should further explore: do examples suffice to overcome this bias, or is more explicit tool support required? Does the magnitude of the intervention depend on individual societal context, and do tools need to be aware of their users’ societal contexts as a result? Or, do these impacts reduce in impact over time, as users habituate to models’ responses and behaviors? Third, are there broader effects of end user culture on some of the hypothesized social interaction inhibitions? Do members of high-context cultures struggle in different ways than what we observed in this work?

Fourth, how can tools appropriately set capability expectations for end users? LLMs are not an artificial general intelligence, sometimes known by the initialism “AGI,” but will end-users hold those expectations by default—and if so, what are effective ways of dispelling those notions?

A.1 Mapping Template to Prompt

Figure 4:

To generate a given bot’s chat messages, BotDesigner concatenates the preamble, the entirety of the conversation so far (using Bot: and User: prefixes for each turn as appropriate), the reminder (if any), and the final prompt Bot:. (Figure 4b in this appendix shows this relationship.) This text string is then sent as a single prompt to GPT-3. For consistency across tests, BotDesigner always uses GPT-3’s text-davinci-002 model with temperature³ set to 0.

A.2 Evaluating Effects of Prompt Changes

To evaluate whether a particular template change affects any of the identified problematic chatbot responses, BotDesigner replays conversations containing errors and displays any modified responses. Two implementation approaches are possible for this task: (a) a system could either perform an “single replay” by assuming all conversational turns prior to the error will occur as in the original conversation, and test only whether the error utterance is changed; or, (b) it could perform a “total replay” in which every conversational turn is replayed and any changed utterances are flagged for user review. Both approaches have merit; the “total replay” approach is more consistent with the “regression testing” concept—certainly, a designer would not want to inadvertently introduce problematic utterances where none previously existed—but providing clear feedback requires identifying which conversational turns have changed in trivial ways, itself a nontrivial task.

For BotDesigner, we default to the “single replay” in an attempt to reduce noise, and accept the resulting short-term trade-off in accuracy that allows more rapid iteration—but leaving designers with the need to perform more extensive testing before deployment.

A.3 Pilot Evaluation of BotDesigner’s Use

To validate that BotDesigner does enable systematic prompt evaluation, we recruited N = 3 academic researchers with a range of interest levels and experience with conversational agent design⁴ and evaluated how effectively they could identify common or particularly severe bugs or errors in a baseline chatbot, and how effectively they could evaluate a new bot template for improvements over the baseline bot.

In advance of the interview, we collected conversations with the baseline bot from AMT workers conversing with a baseline chatbot. We then asked our pilot users to (1) browse the collected conversations to find errors and label them with categorization tags; (2) evaluate a “new” template with updated prompts, provided by us, and decide whether this new template resolved any of the errors participants had previously identified.

Our pilot users had no problems detecting most of a set of 5 error categories we had previously identified in the set of conversations between AMT workers and the baseline chatbot: (1) skipped steps, (2) ignored user expressions of negative emotion, (3) ignored user requests to wait until the user had completed some task, (4) factually incorrect responses to questions, and (5) otherwise unhelpful responses.

We learned a few things from this small pilot that ultimately informed our probe. First, we learned that BotDesigner functioned well enough for a small set of researchers to use it successfully. Second, critically, we observed that looking over a set of prerecorded conversations is quite cognitively demanding, involving lots of context switching, because these were not conversations pilot users themselves had with the bot. As a result of these observations, we (1) retained the “Error Browser”, but (2) dropped a prototype “Conversation Browser” view (focused on tracking common conversational threads across conversations) from the version of BotDesigner we used in our probe. We also (3) designed a much more open-ended task for our probe requiring less knowledge of task minutiae to identify errors, and encouraging users to have their own conversations with the bot, while skipping the initial collection of conversations from AMT workers and thus avoiding the need for participants to understand the task well enough to identify subtle errors in instruction.

Figure 5:

B.1 Behavioral categories

Observed behavioral categories for Carla:

These categories were highlighted in the following transcript.

B.2 Transcript

Today we have 20 minutes to make mixed veggie tempura and we’re gonna see if Lizzie can follow along with me through verbal instructions only.

Expert: So the first thing we’re gonna do is take this very brain-looking giant mushroom – so this is a hen-of-the-woods mushroom, also called a maitake. I got my squash and my lemons out of the way.

So the first thing I want you to do is just turn it over so the stem side is up got it and then just use your hands to kind of break big clusters off of the bottom and like kind of large pieces.

And then we’ll go back in and we can make them smaller and I like doing this with my hands because then you get -

Amateur: oh you’re not using your paring knife

Expert: No, I’m only using the paring knife if it was dry or spongy at the bottom. We’re probably used a quarter of these (mushrooms.) So then just take the tray with the mushrooms on it and move that over to your right-hand side and we’re gonna make the batter.

So next thing – you got your board cleared?

Amateur: Yes, there’s mushroom juice on it but it’s fine.

Expert: I think that’s the line from a Woody Allen movie. So now you’ve got two bowls. One is flour and one is cornstarch. So put those into the larger bowl together. We’re gonna make the tempura batter now.

Once you have your flour and your cornstarch. You have a little dish that has salt and baking soda and those get combined too.

Amateur: Wait… Wha.?

Expert: They should be in the same little bowl.

Amateur: There’s another little bowl that has salt and baking powder - it was to my left. Is it to your left?

Expert: It’s in my bowl now but yeah there was a little bowl with white stuff that isn’t salt then that’s your…

Amateur: Okay.

Expert: And so let’s just whisk the dry ingredients first. So once the dry ingredients are whisked together, take your measuring cup that’s over to the right and those two little bottles of seltzer and we’re gonna measure 2 cups, which I want to add gradually. So take your whisk in one hand, your measuring cup in the other and then gradually whisk in. Do about four-fifths of this and then we’ll check the texture …

Amateur: It’s like… you know what “Oobleck” is?

Expert: What did you say? “Blueblack?”?

Amateur: “Oobleck”. It’s like a thing you play with as a kid with, like baking soda and water and that’s what it’s reminding me of.

Expert: Cool, and it should be about the consistency of heavy cream.

Amateur: Yeah okay. I’m gonna try and get a little more bumps out but I think I’m almost there.

Expert: All right so I’m gonna put my whisk into the garbage bowl just to get it out of the way and do you feel like you’re in a lump-free place? Because we’re gonna start frying.

Amateur: Em oh no... I’m like a perfectionist, so I’m gonna... I think I’m just gonna say yes…

Expert: Okay, great. Take one big handful of mushrooms.

Amateur: Uh huh.

Expert: And, put them right into the batter. And, there should also be a fork, and a spoon.

Amateur: Yeah.

Expert: And, I’m gonna use my hands and the fork to really gently, ’cause I don’t wanna break these clusters up, but I wanna coat all of the mushrooms with batter.

Amateur: Yup.

Expert: So, we should be at around 350. So, maybe while we’re waiting we can start prepping our acorn squash.

Amateur: Great.

Expert: So, now you need your cleaver.

Amateur: Yup, oh God.

Expert: You got your acorn squash.

Amateur: Yes.

Expert: So put it on one side so that the stem end is like closest to your dominant hand.

Amateur: Yes, got it.

Expert: And then, just cut straight down, kinda close to the stem. And, just shear off that front part.

Amateur: How do you use a cleaver? Do you just like... hit it?

Expert: You can hit it. You can use it like a slicing knife and just go straight down.

Amateur: I make really fun faces when I’m frustrated, or like straining, and it’s not pleasant to have a camera in front of me.

Expert: All you need to do right now is just create like a flat edge so that we can stand–

Amateur: Did it. Okay.

Expert: Great. And then, we’re gonna cut it into two lobes. Straight down.

Amateur: Okay.

Expert: And, if you’re off to one side it’s fine, ’cause we only need a quarter of the squash.

Amateur: Just a quarter.

Expert: Just a quarter. So, get–

Amateur: So, cut the halves into quarters as well.

Expert: Yeah, so once you have it in half.

Amateur: Got it.

Expert: Take your spoon, and scoop the seeds into that little trash bowl that we made.

Amateur: Yes.

Expert: And then, we’re gonna leave the skin on, and cut this into thin slices.

Amateur: Okay.

Expert: Going crosswise, so they’re kind of like, they look like little clouds to me.

Amateur: Okay. Do you find it easier to use a cleaver than like a chef’s knife?

Expert: Not necessarily.

Amateur: Yeah, I’m not finding it the easiest.

Expert: You could switch to your paring knife if you want.

Amateur: It’s okay. I’m just kind of scared of big silvery things.

Expert: Yeah, it’s a scary tool for sure. All right, so by now, do you have a quarter sliced? A quarter acorn sliced?

Amateur: Yup, just about.

Expert: All right. So, go back to the mushrooms.

Amateur: Yup.

Expert: And, then using your hand and the fork again.

Amateur: Yes.

Expert: Lift them out and let that excess batter drip off. And then, you wanna just go right into the oil.

Amateur: Okay, I’m using my hands ’cause I–

Expert: Same. But, we’re not afraid of fry oil.

Amateur: Well, it’s not splattering the way my, the way I’ve been frying, ’cause I don’t have a thermometer...so clearly I’ve been–

Expert: You might be going really high.

Amateur: I think it was way too hot. Wonder if I should’ve added more of my club soda.

Expert: If it feels thick, yeah, add more.

Amateur: ’Cause I do, I feel like mine’s–

Expert: A little too thick?

Amateur: It’s a thick tempura.