South Africa's national AI policy was pulled over citations the AI made up.

In full

On 10 April 2026 the Department of Communications and Digital Technologies published the draft National AI Policy in the Government Gazette as Notice 3880 of 2026, opening a 60-day window for public comment. It was meant to be the country's first structured attempt to govern artificial intelligence. Sixteen days later, on 26 April, Minister Solly Malatsi withdrew it. The reason was not a disagreement over substance. It was that the reference list contained sources which did not exist, generated by the same kind of AI the policy was written to govern.

The most quotable irony of the year, and a real warning underneath it

It is an easy story to enjoy at the government's expense, and the headlines duly enjoyed it. The minister was blunt that this was not a cosmetic slip: he said the inclusion of fictitious citations had compromised the integrity and credibility of the draft. Senior officials were reported to have been suspended while the matter was investigated. A document about trustworthy AI had been undone by an untrustworthy use of AI.

A model that will invent a citation will, with exactly the same confidence, invent a clause, a number, or a line of code.

Strip away the institution and the failure is mundane, which is precisely why it should worry anyone using these tools at work. A large language model produced text that read as authoritative. Somewhere in it were references that looked right, formatted correctly, attributed to real-sounding authors and journals. Nobody checked them against reality before the document went out. That is not a government problem. That is the default outcome whenever AI output is trusted by appearance rather than verified against a source.

Why this keeps happening

Language models do not retrieve facts; they predict plausible text. A fabricated citation is not a bug in that sense, it is the system doing exactly what it does, filling a gap with the most likely-looking string. The output is confident whether or not it is correct, and confidence is the one signal humans are worst at discounting. The more polished the draft, the less likely anyone is to interrogate the boring parts at the bottom.

• Fluency is not accuracy. The two come from different mechanisms, and the model only guarantees the first.
• Plausible is the dangerous kind of wrong. An obvious error gets caught; a believable one ships.
• Trust is what scales the risk. The more a team leans on AI without a check, the more invented detail slips into work unexamined.

The fix is process, not a better model

The instinct after an episode like this is to wait for a model that hallucinates less. That is the wrong lever. Even a model that is wrong one time in a hundred will eventually put a fabricated fact into something that matters, and you will not know which time it was. The reliable control is structural: anything a model asserts as fact, a citation, a statistic, a legal reference, a configuration value, is checked against a real source before it is used. In our own systems that means AI generates, and a verification step confirms against ground truth, with the unverifiable flagged rather than passed through. The model is treated as a fast first draft, never as the authority.

What it means for how you adopt AI

None of this is an argument against using these tools. It is an argument against the one habit that turns them into a liability: accepting output because it looks finished. The organisations that get value from AI without getting embarrassed by it are the ones that decided, up front, where a human or a deterministic check sits between the model and anything consequential. The South African policy did not fail because AI was used. It failed because nobody verified what the AI produced before it carried a government masthead.