Using an AI detector to catch a modern LLM is like sending a 1990s-era MS Office Paperclip Helper to fight a T1000 from the Terminator movies. The LLM is a trillion-parameter model trained on supercomputer clusters; the detector is a thin classifier guessing from shallow patterns.

This is not just some tech-bro trivia. These tools are often touted as digital border guards of academic and professional integrity, but their fundamental flaws are causing real-world harm in education and business: false accusations, rejected work, eroded trust. Yet many institutions continue treating them as reliable arbiters of authenticity.

Turnin’ Tides

After a tumultuous couple of years with AI detection trackers, the cracks are starting to show. UCT recently announced its lecturers and tutors will officially discontinue using Turnitin’s AI Score from 1 October 2025, citing risks to student trust and academic fairness.

This came as no surprise to AI industry insiders. In fact, back in 2023 OpenAI, arguably the most sophisticated AI software company in the world, quietly shut down its own AI-text classifier due to low accuracy.

But many corporations, agencies, and universities continue to rely on these tools, and disclaimer-laden proclamations like “Your text appears to be somewhat AI generated,” continue to wreak havoc. Hopefully the institutional level disavowal from the likes of UCT is a bellwether for a paradigm shift, or at least a paradigm pivot.

How We Got Here

The story of how we got here is one of systemic pressure, technological overreach, and a fundamental misunderstanding of the problem we are trying to solve. 

The demand for these tools is undergirded by larger economic and historical forces like rapid technological advance, globalisation, and the scaling of higher education, which gave rise to semi-autonomous modes of assessment. 

There is an ongoing debate in South Africa as to whether, and to what extent, higher education should be socialised or commoditised. Yet knowledge itself resists commodification in a fundamental way: unlike physical goods, it multiplies rather than diminishes when shared. Throughout history, from oral traditions to open-source software, societies have maintained channels where knowledge flows freely alongside – or even in defiance of – market logic. 

In any event, amidst the reality of ballooning class sizes, tools like Turnitin became a necessity for overworked lecturers to check against plagiarism. In the academic context these AI detectors have a legacy in a broader category of provenance tracking software, most notably plagiarism checkers like Turnitin. But the plagiarism and originality checking process is fundamentally different to AI detection. 

The former searches against massive databases of pre-existing text and, while not 100% accurate (with shorter passages there’s always a chance of arriving at the same sequence of words independently), is still considered much more reliable than the latter, which (in machine learning terms) is based on thin heuristic classifiers; a form of pattern recognition for text. 

Perplexing and bursty AF

These AI detectors mostly measure “perplexity” and “burstiness” – an algorithmic equivalent of what we humans simply think of as creativity with language, word choice and sentence structure. 

This is also why they’re brittle and easily duped by simple paraphrasing or a clever prompt. You’re often better off asking ChatGPT itself to determine if a passage was written by ChatGPT, than relying on these dedicated AI-flagging tools. 

Systemic pressure and Institutional panic

When ChatGPT burst onto the academic scene, to universities, adopting the solutions provided by their software vendor partners like Turnitin was a no-brainer: like buying a fire truck in a blaze. One that, on closer inspection, turned out to be an ordinary garden hose. 

For lecturers – and for managers in the corporate context – these tools offered a shield of plausible deniability, allowing them to outsource a difficult human judgment and say, “The tool flagged your work,” which is easier than levelling an outright accusation. 

Let’s also take a moment to consider the reasons students use AI in the first place. A massive wave of anti-AI sentiment (partly justified, partly misguided) created an assumption that any use of AI meant zero effort whatsoever. But this is a false binary, and the majority of use cases among students are likely legitimate. 

Some use AI to augment and amplify their efforts: a web-search and reasoning-mode enabled brainstorm can help immensely in making sense of complex strands of thought; a retrieval augmented generation (“project” or “custom GPT”) can help collate large bodies of textual research material and synthesising disparate information; In certain domains, a Perplexity-augmented deep research query may be more powerful than a traditional library or scholarly resource database search; A student from abroad may use it to overcome the inherent disadvantage of not being a native speaker of the language of instruction. 

From their side, the vendors of these tools were responding to a genuine market need, and, to their credit, they have been upfront about the limitations of their offerings from the start. They have never claimed that their solutions are fully accurate or comprehensive. As they would tell it, given the severe peril to authenticity in the era of cyber-content, a fairly accurate – if imperfect – detection technology serves an important purpose in today’s software ecosystem. And to be fair it probably does have a place. 

That said, the UCT reversal shows that the Overton window has shifted in SA. Decision-makers are realising that current detector technology is simply not fit for the high-stakes purposes for which it’s being used.

Turing it in

Allan Turing’s famous Turing test highlights the irony here: with GenAI, we’ve unleashed systems capable of producing human-indistinguishable text. No one is confident any longer in their ability to discern human-written from machine-generated content (heck, us humans are very capable of manufacturing generic, hackneyed-sounding prose).” 

We’ll get GTA 6 before we get a reliable AI text detector

The very aspect of LLMs that created the need for automated machine-human distinction is the very thing that makes it impossible to achieve. As GenAI tools are further entrenched into everyday tools and technologies – De Facto daily drivers like MS Word and Gmail – and integrated into ever more complex workflows at various touchpoints, the distinction becomes all the more blurred.

The detection technology of today is a crude solution to a profound and incredibly sophisticated challenge that resounds in our deepest-held notions of authenticity and what it means to be a human being. 

The biggest problem is that the detectors present a probabilistic guess as an actionable score. Vendors rarely publish error rates or calibration data. Users end up acting on the output of an unauditable black box. LLMs are also unauditable blackboxes, just way more advanced. 

As noted, companies with substantial computational resources do not seem too interested in pursuing the project to upgrade detector technology. Perhaps because they think it simply cannot be done, or perhaps because they see it as antithetical to their core mission; understanding, amplifying, and, ultimately, manufacturing human-like intelligence. 

For better or worse (probably worse), the likes of Google, MS, X and Meta are likely the only companies capable of comprehensively solving the AI text provenance problem. Creating reliable detection technology would entail fundamentally rewiring the way dataflows, computation and text encoding are handled at the kernel and machine-code levels across all devices on a staggering unprecedented global scale. It’s frankly unfeasible.

What about text ‘watermarking’?

Tech-savvy defenders often point to solutions like Google’s SynthID-Text as a fix. But this isn’t a silver bullet. Obviously any metadata-level solutions can easily be bypassed with a good old-fashioned plain text copy-and-paste. Some text watermarking technology also works by programming the AI to subtly prefer certain word choices. For example, picking “serendipitous” instead of “uncanny.”  It only works in a closed system. A detector from the same company can then use a given key to identify this statistical pattern and estimate likelihood of AI authorship. In the wild west of the open internet, the trail quickly goes cold.

But with this sort of method, it’s still just a probability, not a proof. A human author could also have chosen “serendipitous” for his own reasons. Just like in the current detector modality, this fragile input signal breaks if you simply paraphrase the text (or even have a chatbot paraphrase it for you).

Centaurs and Reverse Centaurs

There’s a useful framework from AI ethics: the distinction between “centaur” and “reverse centaur” systems. In a proper centaur configuration, the human ‘head’ remains in charge, and the equine AI steps in to take on the legwork, albeit constantly steered by the human in the loop. For example: the student using AI to research more broadly, or the writer using it to sharpen his own ideas. 

In a reverse centaur model, we hand the reins over to the AI, which then makes the decisions that the human labourers must carry out, or be beholden to. Uber’s algorithmic management of drivers is a classic example. 

Reverse centaurs are particularly problematic in high-stakes domains. When an institution relies on an AI detector to make pass/fail judgments, they’ve effectively ceded their authority to an algorithm that cannot bear the moral weight of that decision. 

Lifelong Learnings

Something my grade five teacher, Mr Marule, said ahead of the end-of-year exams that has stuck with me ever since. It is a simple piece of wisdom that I fear educators today often forget. He said, “When you study, don’t study for marks. Study for knowledge.” His words were so profound I’m still unpacking it to this day, some 30 years later. 

I now understand that the AI detection problem is a heuristics trap: when a measure (such as marks in academics) becomes the target, people will game the system to optimise for the measure, at times even at the expense of the original goal, value, or principle (in this case, knowledge; internalising it, synthesising it and creating it).

Naturally, some students will try to cheat by using AI in lieu of doing the mental work. But relying on these detectors to identify culprits has resulted in another heuristics trap: Students now optimise to “beat the detector” rather than how to write to express their complex thoughts with optimal clarity.

Likewise, in the corporate context, copy and prose can go pear-shaped when the copywriter is pulled in different directions by “stick closely to the brand voice” and “check your work against the detector we’ve chosen to use”. The writer is then optimising to navigate these stringent trade-offs, when they should be spending that mental energy on crafting resonant stories that connect with other humans in a way machines can’t measure.

True Solutions

The solutions we lean into may turn out to be non-technological. In education, it could mean reframing the question: not “did you use AI?” but “Do you truly understand the material?” Reformed assessment could include two-phase tasks; a take-home draft followed by a short in-class presentation where students defend their reasoning in real time. It means process portfolios that track version history and thinking, not just final polish. It means embracing AI as the “new pen” and teaching students to use it well, with disclosure but without punishment for honest use.

Likewise,  in business, the answer lies in trusting the people you hire to use the appropriate tools, and judging the quality of output, not the method. One of the most important skills in the AI age is honing our own human taste. 

We can’t outsource judgments on quality to the machine. Nor should we outsource our judgements about provenance and authenticity.