We are so excited. We can pull up a broad set of information on a given topic – without even thinking. We can create engaging, concise and well-structured writing – one we would never write ourselves. And we can express our ideas verbally to generate visuals, from photorealistic images to sketchy illustrations or diagrams. But something is not quite working. The end result seems to be just one more step away:
One more iteration. One more choice. But wait, I lost that one and I need to go back. Oh. And I need more depth here. And I want a better synthesis altogether…Ah. I should stop here…It is not getting any better.
After investing quite a bit of time (and yes AI is saving, but also taking away time as we struggle for perfection), we take the best output and continue with manual edits. Getting our intent, our ideas out of our head and describing them in a way that AI can find, synthesise, expand and visualise, is far from straightforward. The more natural the conversation, the more frustrating it feels when the generated content is not right. This online article by Jacob Nielsen describes the “articulation barrier”, highlighting how “prompt engineering” is becoming more and more of a specialised profession:
“That “prompt engineering” can be a job suggests that many business professionals can’t articulate their needs sufficiently well to use current AI user interfaces successfully for anything beyond the most straightforward problems.”
The usability issue with prompt-driven AI is evident. Every time an AI user expresses frustration with the results, the answer they get usually get from peers follows along: “Wrong prompt”, “you need a different prompt”, “you need to change the order” etc. It reminds me of the early days of the web: if something is not quite working, it is the user’s fault. In reality, natural language prompts are still like code. You don’t need to memorise specific commands or put any </> to get something back. However, the exact words that you use and how you order your questioning – matters. So “prompts” are actually “scripts”. You copy and paste someone else’s script with your data and aha – you suddenly get a better output.
The barrier of articulating prompts does not only relate to literacy, although literacy is indeed a huge barrier for AI adoption in low resource settings. I have a PhD in Human Computer Interaction (so I consider myself quite literate), yet I am often frustrated with the process of iterative prompting – especially when creating visual content. Although, one can achieve a lot through experimentation, continuous trial and error – natural language interactions are not enough. Prompt engineering is creating another kind of digital divide. It is not AI who’s threatening to take over your job, but rather other professionals, who are highly skilled at … prompting AIs.
The challenge is being able to think like an “engineer”. We need a mental model of how Gen AI works in order to be able to adjust our prompts accordingly. AI interactions lack “discoverability”: being able to identify the available functionality of a system. We rely on trial and refinement, because we are not able to understand the workflow that an LLM follows and the parameters that define the text, images, sounds or video that it produces in response to our prompt. As much as we might understand the general principles and available use cases for a given Gen AI tool, we do not have a clear “conceptual model” for AI. As Don Norman says in “The Design of Everyday Things” in 1990:
“A good conceptual model allows us to predict the effects of our actions. Without a good model we operate by rote, blindly; we do operations as we were told to do them; we can’t fully appreciate why, what effects to expect, or what to do if things go wrong.”
Since last year, AI tools have improved to enable more user control and to increase transparency. This is evidenced in the use of interactive elements, like sliders, buttons, or pre-defined prompts, in addition to text or voice user input. These are steps towards the right direction – a good start.
But let’s think beyond widgets, drop-down menus, buttons and sliders. Every transformative technology has had its interaction design challenges at start. Remember the first internet-enabled phones? I do. In 2007 I worked at Deutsche Telekom and designed interaction models for some of the first touchscreen phones, creating experimental prototypes, mock-ups and learning through iterative user testing. At that time, there were no “design systems”. No design guidelines either, enabling users to efficiently explore internet content on a phone or to interact with a touchscreen. Those were exciting times for us, interaction designers – because we had to figure out these guidelines. We are now living a similar moment with Gen AI. However, many haven’t realised yet that solving this design challenge requires a more holistic, innovation-driven approach – rather than relying on existing patterns.
My vision for our near future interactions with AI lies in co-creating dynamic, real-time workflows, navigating intuitively back and forth between versions, selecting and extracting specific elements to refine or to create a new path.
It would take a design innovation process to bring this vision a step closer, through close collaborations with AI engineers, prototyping and user research to define the best performing interaction models. To create new guidelines, patterns and eventually design systems for interaction design with AI that tackle the problem at its core. Trying to bring more user control by adding a few contextual menus here and there, is just scratching the surface.
If we want to save our time, attention and sanity, our interactions with AI need to be more efficient, more purposeful. Less is more.
*** If you are a tech start-up or a research team working on interaction design for AI and this article resonated with you, please comment or message me. And if you know anyone who would be interested, please tag them. Great things happen by connecting the dots… and people. ***
One last thing – just for fun.
I had a chat with ChatGPT to determine what users are missing when interacting with AI. As with most of our “conversations”, the response felt incomplete and verbose. So I ended it with a creative twist, resulting in this:
“Researchers and practitioners around the globe have noted that we require richer interfaces—vivid controls that appeal to our visual instincts, modular tools that let us tend to each part of our creation, and living feedback loops that reveal, in real time, the subtle dance between our intent and the machine’s response.”
(This article was originally posted on LinkedIn on 4/05/2025)
Leave a Reply