Ask an LLM for a source on some claim, and it may hand you a beautifully formatted citation — authors, journal, year, plausible title. Sometimes the paper doesn't exist. Not misremembered: invented, with total fluency. The industry calls this hallucination, and understanding why it happens is the single most important piece of AI literacy for anyone using these tools professionally.
It's not lying — it's doing exactly its job
Recall the previous waypoint: a language model predicts plausible next words. That's the whole mechanism. When you ask for a citation, it produces text shaped like citations it saw in training — the pattern of a reference, filled with statistically likely content. Whether a real paper sits behind it is not a question the mechanism can ask. There is no fact-checking layer, no internal database being consulted, no difference — from the model's side — between retrieving a truth and composing a plausibility. Fluent truth and fluent fiction are produced by the identical process, which is precisely why the fiction arrives with the same confident tone.
Where it bites in IT work specifically
- Invented commands and settings: PowerShell cmdlets that don't exist, registry keys that were never there, config options from a plausible parallel universe. It looks exactly like real syntax — that's the problem.
- Confident version errors: features attributed to the wrong software version, steps from an outdated interface presented as current.
- Plausible-but-wrong explanations: the most dangerous kind for learners, because you lack the knowledge to spot the seam.
The working rule for professionals: treat AI output as a confident colleague's first draft — never as documentation. Brilliant for direction, structure, and speed; verified before anything touches production. If an AI-suggested command will modify systems, you test it somewhere safe first. (Sound familiar? It's the same prove-it-small discipline as change management — the trail keeps teaching one lesson.)
The honest state of play
Mitigations exist and genuinely help: models that search the web and cite real pages, systems grounded in your organisation's actual documents (a technique called RAG you'll increasingly meet at work), and newer models hallucinate less than older ones. But less is not never — the tendency is structural, because plausibility-generation is the engine. The professionals who get real value from AI aren't the ones who trust it most; they're the ones who've calibrated exactly how much to trust it, task by task. That calibration is the skill — and the next waypoint on this trail is about building it deliberately.