We live in a world where articles and pictures and videos can be generated by a machine. Do we want to go down the detective road?
Chapter One of the Constitution of the Republic of South Africa was written in 1996 – by generative AI.
At least, that’s what an AI detection tool has just told me.
To be fair, this particular AI detection tool (ZeroGPT) has been rated by The University of Chicago’s Academic Technology Solutions unit as “deeply problematic”.
Other tools that I tried were confident that our constitution was in fact written by humans (although as my husband points out, it was a committee of humans, which may have bearing on the matter.)
It’s also quite possible that if I ask the same tool the same question in a week’s time it will give a different answer. As one person in my freelancing association’s email group said: “There are many AI detectors… when I have used them it seems sometimes the detector works well, and sometimes not. Also have run the same text through more than one and got different results.”
What is an AI detection tool anyway?
The very short answer: a tool into which you paste text to find out if it was written by a human.
The slightly longer answer? I fed two links from reputable sources (How Do AI Detectors Work? Key Methods and Limitations | Grammarly and How Do AI Detectors Work? Understanding the Tech Behind AI Detection – AI in Education) into Google’s NotebookLM and asked it to give me a short text explaining what an AI detector is and how it works. I got 352 words back, which, of course, is no one’s definition of a short text. With many a deletion, here are the need-to-knows:
AI detectors use machine learning models trained on vast datasets containing both human-authored and AI-generated text. The detectors then compare new content to these datasets, looking at two key metrics:
Perplexity (the predictability of a word sequence): AI models choose the most likely next word, resulting in “low perplexity”. Human writing is often more unpredictable and has “higher perplexity”.
Burstiness (variation in sentence length and structure): Human writers mix short, punchy sentences with long, complex ones. AI-generated text tends to be more uniform and consistent in its flow.
Some detectors also look for metadata traces or compare text against a database of known AI-generated outputs to find matches.
What are the problems with AI detectors?
There are many! In short – they get it wrong (see note about the Constitution above). That’s because AI detectors look at probabilities, and can’t provide definitive proof. This leads to two common errors:
False positives: When human-written content is mistakenly flagged as AI-generated. (Note that this often happens with writers who have a very formal style or second-language English speakers whose writing may follow more predictable patterns. Studies indicate that neurodivergent students and students for whom English is a second language are flagged by AI detection tools at higher rates than mother-tongue English speakers.)
False negatives: When AI-generated content goes undetected.
AI detectors in action
I did a small and completely unscientific study of AI detector tools. The tools I chose to use were based on search results and recommendations in two different sources. The texts I used for testing were written by me – two of them well before the advent of ChatGPT, and one afterwards. (I then went a little rogue and tried the SA constitution in two tools*, inspired by this 2023 post in which both the Bible and the US constitution had been declared to be written by AI).
The TL;DR on the results: Three tools correctly identified my older articles as human-written and flagged some of the content in the 2026 article as AI-written: exactly as I know the texts to have been created. One tool (ZeroGPT) got almost everything wrong.
TEXTS I USED
How to fail at being middle class – written in 2015, quirky and personal
How news organisations establish trust – written in 2018, opinion
Both of those contained no quotations from other sources, as I thought that would muddy the results.
A parent’s guide to Gen AI: 11 ways to help your offspring stay safe and learn stuff – published in 2026, based on research done using NotebookLM and containing text generated by the tool, and edited by me (all of which was declared in the post, as per my AI policy)
Test one: GPTZero
I tested this one because two different sources declared it the best free AI detector (Business Insider and Chicago university). Results:
Being middle class – We are highly confident this text is entirely human (100%)
News organisations – We are highly confident this text is entirely human (100%)
Parenting and AI – We are highly confident this text is entirely human – but the rating was 86% which the tool said meant: “This result isn’t the percentage of human text, it’s the model’s confidence in the classification of the entire text as human.”
SA constitution – We are highly confident this text is entirely human (100%)
Test two: Originality AI
Being middle class – 100% Confident That’s Original
News organisations – 100% Confident That’s Original
Parenting and AI – 74% Confident That’s Original
Test three: Grammarly
Being middle class – 0% of this text appears to be AI-generated
News organisations – 0% of this text appears to be AI-generated
Parenting and AI (it scanned 1400 words, as the full length was 1518 words, over the tool’s limit) – 17% resembles AI text
Test four: ZeroGPT
Being middle class – Human written but 3.9% of text likely to be written by AI.
News organisations – Human written but flagged 2.88% as AI generated
Parenting and AI – Your Text is Most Likely Human written – 19.48% AI written
SA Constitution – Your Text is AI/GPT Generated
Why do people use them then?
My reading consistently brings up two big areas of friction:
1. When people have been hired to do paid writing work, the people doing the paying want to know that said writer did not spend five minutes getting ChatGPT to do the work and then bill them for six hours. There are three underlying things possibly going on there. The employer genuinely wants human judgement and creativity to be in play – or they are suspicious and untrusting. Or the writer is actually taking their client for a ride.
2. The world of education: here, there are two main strands. One, the thinking that a learner or student won’t learn anything if they use AI, and two, that using AI is in fact cheating: the student did not do the work themselves.
In both cases, there are legitimate concerns – but I’m not sure that using an AI detector is the solution.
The “prove you are human” paradigm leads us down a road where there are now websites like TextPolish where you can “transform your AI-generated text into natural, human-like content that bypasses all detection tools”.
Also, people who are genuinely doing the writing are now changing their natural style so as not to sound like AI, as described by Bo Hu at the School of Social and Behavioural Sciences, Nanjing University: “Sometimes I deliberately replace words and phrases, hoping that the revised expression will convincingly show that I am human and that my writing was not produced by artificial intelligence (AI).”
What to do?
I looked at a wide range of sources and opinions (all available in this NotebookLM) and the consensus was to stay away from AI detectors as much as possible: they are not necessarily reliable, they are biased against second-language English speakers, they might be unfairly declaring people to have written an academic essay using AI.
My own thinking is that these tools are a waste of time. People will use AI openly or in secret, and setting up ways of trying to “catch them out” just introduces a mess of mistrust and back-and-forth into a relationship.
Instead, the recipient of a text, whether a teacher or the commissioner of a blog post, is going to have to do some work (the irony is not lost on me: time-saving tools like AI causing more work down the line). Here are some of the things you might do:
1. If you insist on using an AI detector, treat it as an indicator. Run the text through several detectors. Heck – two people I talked to said ChatGPT itself is good at detecting AI-generated text. Then talk to the writer, or compare the flagged text against the author’s established style, voice, and previous work. See if they have a revision history showing their work on the document.
2. If you are an institution or employer, get yourself a policy that clearly communicates how AI detection fits into your evaluation standards (and tell people about it).
Generally, don’t use these tools:
· As sole indicators of misconduct
· To evaluate second-language English speakers or neurodivergent writers
· To assess short passages (AI detectors are apparently worse at scanning short pieces of text)
· As the final arbiter in “professional” contexts
· Without human judgement
For businesses, focus on quality and results
In some business contexts, the method of creation may be less important than the final outcome. Judge work by quality, audience engagement, conversion rates, or search ranking: if the content provides high value and substance, the level of AI assistance is a secondary concern.
It’s also important to remember that when you have hired a writer or editor, you have entered into the territory of credence goods: products of services with qualities that cannot be observed by the consumer after purchase, making it difficult to assess its utility (like when a mechanic fixes your car). And in that scenario, you can get yourself out of the entire morass by developing relationships with your service providers, by establishing mutual trust.
READ: Editors – how to tell the good from the bad and Writers – how to tell the good from the bad
For writers and editors, do this:
Provide evidence of your process: Show how the work was created, rather than just the final output. Share the final version of the Google Doc, allowing your client to see the version history (in the file menu). Or provide screenshots or screencasts of earlier drafts and time-stamped edits to prove the document evolved over several days.
Education: Many clients do not realise that AI detectors have high false-positive rates (often 5–15%). Share this knowledge!
Contractual clauses: Protect your business by including a clause in your contracts stating that AI detection scores are not absolute and that disputes will be resolved through a review of version history rather than automated tools.
* Note – the constitution text was not tested on all the tools, as I ran out free tests.
Main picture: Patrick Fore, Unsplash
Previous Sensible Guide articles to check out
Editors – how to tell the good from the bad – You need an editor. But you aren’t really sure what an editor does, or how to tell if they are any good. Some tips…
Writers – how to tell the good from the bad – You need a writer. But how do you tell if the person you are hiring knows that they are doing? Some tips…
Being a creator in the AI swamp – a way forward – If you work in the creative industries (writing, editing), artificial intelligence looms like a tsunami over you. My thoughts on a way forward…
If any of this is relevant to your work…
You might find one of these useful:
