When AI makes mistakes, it’s called hallucinating. There’s a better, if ruder, word for it. That means this post contains what the dictionary calls vulgar slang. You have been warned.
We all know that generative AI gets things wrong.
Even the people who make these tools know they get things wrong. They say so on the tin, albeit in very tiny lettering, and often only after you have actually asked for something. Right at the bottom of the page, OpenAI tells us that “ChatGPT can make mistakes. Check important info.” And Google’s Gemini says: “Gemini can make mistakes, so double-check it”. DeepSeek says, mysteriously, “AI-generated, for reference only”.
This getting things wrong is often called hallucinating. But I don’t think that’s the right word, at all. Hold on to your hats for a deep dive into what’s actually going on.
What is AI hallucination?
When people hallucinate, the common understanding is that they are seeing or hearing things that aren’t there. The Oxford Dictionary definition of “to hallucinate” goes like this: “experience an apparent sensory perception of something that is not actually present”.
On that basis, it’s clear that an AI tool can’t hallucinate, because it doesn’t have sensory perceptions, real or imagined. In the digital context, the definitions of the term therefore have to say that it is a metaphor, like this from Cloudflare:
Artificial intelligence (AI) hallucinations are falsehoods or inaccuracies in the output of a generative AI model. Often these errors are hidden within content that appears logical or is otherwise correct… The term “hallucination” is metaphorical – AI models do not actually suffer from delusions… Instead they produce unexpected outputs that do not correspond to reality in response to prompts. They may misidentify patterns, misunderstand context, or draw from limited or biased data to get those unexpected outputs.
They make these mistakes because of how they are made. LLMs are prediction engines. As Nielsen Norman Group user experience specialist Caleb Sponheim puts it:
They work by looking at the words that came before and selecting the most likely word to come next, based on the data they’ve seen before… Since LLMs are exposed to training data from all corners of the internet (copyrighted or otherwise), they learn associations between syllables from extensive examples of sentences people have written across human history, leading to their impressive ability to respond flexibly to requests from users. However, these models have no built-in understanding of truth, of the world, or even of the meanings of the words they generate; instead, they are built to produce smooth collections of words, with little incentive to communicate or verify the accuracy of their prediction. (my emphasis)
Scientist Gary Marcus gives a practical example, using chess: “An LLM may purport to play chess, but despite training that likely encompasses millions of games, not to mention the wiki page The rules of chess,… it never fully abstracts the game. Crucially, the sentences ChatGPT can create by pastisching together bits of language in its training set never translate into an integrated whole.” (his emphasis)
In other words, ChatGPT and it’s ilk can’t tell their metaphorical arses from their metaphorical elbows. They aren’t hallucinating, they’re just making patterns.
Why do we call it hallucination then, when it plainly isn’t?
The term’s been around since the 1990s, when it was used in fairly arcane texts about artificial neural networks. Over time, it’s come to mean the generation of incorrect outputs by AI systems, and became more widely used as tools like ChatGPT grew in popularity.
I think that the word is used simply because there’s a widespread tendency to anthropomorphise AI tools. They talk to us like people, so we think of them as people.
But there are problems with that.
To quote Usama Fayyad, executive director for the Institute for Experiential Artificial Intelligence at Northeastern University:
When you say hallucinations, you’re attributing too much to the model. You’re attributing intent; you’re attributing consciousness; you’re attributing a default mode of operating rationally; and you’re attributing some form of understanding on the part of the machine… Demystifying the technology and the behaviors exhibited by algorithms, good or bad, establishes real progress and creates valuable outcomes on all fronts: theoretical, academic, commercial and practical.
As we’ve seen, chatbots don’t have intent or understanding. But if we think and behave as if they do, we make no progress in understanding how to use their outputs.
How to think about it instead: the bullshit approach
I’m still waiting to find a better way of thinking about this than that given by astrophysicist Katie Mack on Bluesky (quoted here in slightly edited form):
I expect that consumer-facing AI programs will continue to improve and they may become much more useful tools for everyday life in the future. But I think it was a disastrous mistake that today’s models were taught to be convincing before they were taught to be right… most humans don’t speak confidently & coherently about something unless they actually know it. The ones who do… well, we have words for them. If a human told you things that were correct 80% of the time but claimed, flat out, with absolute confidence, that they were correct 100% of the time, you would dislike them & never trust a word they say. All I’m really suggesting is for people to treat chatbots with that same distrust & antagonism.
What she’s talking about is, of course, bullshitting, defined by the dictionary like this: “talk nonsense to (someone), especially in an attempt to deceive them”.
The term has been defined academically too, by American philosopher Harry G. Frankfurt, who said it was “speech intended to persuade without regard for truth”. It’s different from lying in that people who tell lies know the truth and understand that they are lying; a bullshitter is indifferent to truth, Frankfurt says in this video: On Bullshit – Harry Frankfurt (2005)
How does this apply to LLMs?
One of the pillars of bullshitting is the intent to persuade. If LLMs can’t have intentions, can they be said to be true bullshitters? Perhaps not, but it’s also true that they are designed to be conversational, approachable, easy-to-use. So it may be that LLMs are not bullshitters any more than they are hallucinators or liars (because they don’t know what truth is). But they are also not reliable all of the time.
As Arvind Narayanan and Sayash Kapoor write in ChatGPT is a bullshit generator. But it can still be amazingly useful:
Large Language Models (LLMs) are trained to produce plausible text, not true statements. ChatGPT is shockingly good at sounding convincing on any conceivable topic. But OpenAI is clear that there is no source of truth during training.
So where does that leave us?
Does this mean we shouldn’t use AI at all? Or that we should approach it with distrust and antagonism, as Katie Mack thinks?
I don’t think so – there are many ways in which it can be wonderful: time-saving, helpful to humanity and just plain fun (see Five frivolous things to do with AI and From babies to bees: five ways in which AI is good for the world). I’d argue instead for an attitude of sceptical curiosity, a spirit of experimentation.
(An aside: In the course of researching this article I came across another possible use of AI. I It might be supremely good at helping to do David Graeber’s bullshit jobs, pointless jobs invented just for the sake of keeping us all working. As pointed out by Alberto Romero: “No one wants to be a bullshitter all day long yet many people have to. ChatGPT can’t help but be one. The match couldn’t be more perfect. ChatGPT isn’t emptying their professional lives of meaning. No, it’s emptying them of the modern illness of meaninglessness.”)
Whether you are deploying AI to find a recipe, or do a meaningless task in your boring job so that you can do something more interesting, or to summarise the contents of research papers for your ground-breaking research into TB, there are some ground rules worth remembering. Based on Narayanan and Kapoor’s advice, use AI when:
- you know enough about a subject to know if it is bullshitting you,
- you don’t need factual accuracy,
- you know something about the data it has been trained on (particularly in detail-driven areas), and
- you know enough about how a particular tool works to know that it will give you output that is useful to you (a custom GPT set up as a coach might be better for giving you an alternative view on a complex problem than a general chatbot, for instance).
Main picture: Doug Beckers, Flickr, (Deed – Attribution-ShareAlike 2.0 Generic – Creative Commons)
Other things I have written
Which AI tool to use? Start by playing – Knowing which AI tool to use can be hard to figure out – there are apparently 10,500 tools out there. I suggest a place to start…
Renee’s four golden rules of artificial intelligence – So much hype, so much uncertainty, so much information about artificial intelligence. Here are some guiding principles…
Five frivolous things to do with AI as the year turns – The AI hype that floods our screens is all about productivity and efficiency. But there are some small fun and useful ways to interact with AI.
From babies to bees: five ways in which AI is good for the world – The flood of information about artificial intelligence never seems to end, and so much of it is focused on the workplace. Here’s a list of ways in which AI is good for the world.
How can I help you make order from chaos?
Join the Safe Hands AI Explorer learning circle!
Sign up for my Sensible Woman’s Guide to AI and Content Creation, which comes out fortnightly.
Or sign up for my personal fortnightly email newsletter In Safe Hands (it’s a place to pause and think about things).
Book a free half hour of my time here. Find out how I can help with writing, editing, project management, general clear thinking, technical hand holding, or an introduction to AI.
Contact me via email.