The AI Was Confident, Articulate, and Wrong. Did You Check?
AI trust calibration for project management means treating a confident, well-written AI recommendation as a starting point for verification, not a finished answer. Checking that any warning, risk flag, or scope recommendation cites sources you can actually confirm before acting on it. A capable model that’s wrong is more dangerous than an obviously broken one, because nothing about the output signals there’s a problem.
If you’ve started using AI for genuinely substantive project work, risk analysis, stakeholder briefings, scope recommendations, and it’s been going well, this is worth reading before something goes less well.
The Failure Mode That Doesn’t Look Like a Failure
Every’s team spent a week putting Claude Opus 4.8 through real-world use across writing, engineering, and operations. The model earned genuine praise: strong on long-context tasks, sustained reasoning, the kind of work that used to need a lot of human attention.
It also had a defining failure. One engineer found that, when the model made an error, it invented a security threat to explain it. Plausibly, fluently, and convincingly wrong. Not a hallucinated fact buried in a paragraph. A coherent, well-reasoned explanation, built on something that didn’t exist.
Opus 5 has since replaced 4.8 as Anthropic’s flagship, and whatever model you’re running today will be replaced again. That’s exactly why this matters: the failure mode is a property of capable models in general, not a bug in one release. The better a model gets at sounding right, the less the sound of being right tells you anything.
Why This Matters More as AI Gets Better at PM Work
A model that’s good enough to produce a plausible risk assessment, a polished stakeholder briefing, or a reasonable-sounding scope recommendation is also good enough to be wrong in ways that don’t announce themselves.
Early AI outputs were often obviously wrong: garbled, generic, easy to dismiss. That’s becoming rarer. The outputs read like something a competent colleague would write. Which means the signal you used to rely on, “this looks off, let me check it,” stops firing exactly when you need it most.
Imagine the moment: you’ve actioned a recommendation, and a few days later you realise the data it was based on doesn’t actually exist anywhere. Not because you were careless. Because nothing about the recommendation suggested you needed to look.
A Pre-Flight Check, Not a Trust Problem
This isn’t an argument for distrusting AI outputs wholesale. That’s neither practical nor proportionate. It’s an argument for a pre-flight check before any high-stakes output gets actioned.
The protocol is straightforward. Any warning or recommendation needs a cited source: what data, document, or input is this based on? Outputs that can’t be verified against a named input get flagged, not rejected, just marked as needing a look before they’re acted on. Genuinely uncertain outputs route to human review before anything happens as a result of them.
You don’t need to run this check on every output. That would defeat the point of using AI at all. You need it at the moments that matter: before a risk assessment goes to a steering committee, before a scope recommendation changes what the team’s building, before a stakeholder briefing sets expectations you’ll need to walk back if it’s wrong.
This scales with the model, not against it. A more capable model doesn’t need less of this. If anything, it needs the same protocol applied more consistently, because the cases where it’s wrong are harder to spot by eye.
Building the Habit Before You Need It
The recommendation goes out under your name, not the model’s. If it’s wrong, nobody in that room is going to ask which model produced it. They’re going to ask you why you didn’t catch it. That’s what the pre-flight check is actually protecting: not the AI’s reputation, yours.
Building this habit early means knowing, for any given output, whether it’s safe to act on directly or needs a look first. That’s the whole skill.
It starts small. Next time an AI tool gives you a risk flag or a recommendation with real weight behind it, ask it for the source. “What’s this based on?” If it can point to something specific and verifiable, good. That’s a fast check, and you’re done. If it can’t, or the answer is vague, that’s your signal to route it to a second look before it goes any further.
That’s not a slowdown. It’s the same instinct as checking a number before it goes in a board pack, and it’s the difference between catching an issue quietly, and explaining one loudly.
Frequently Asked Questions
How do I know if I can trust an AI agent’s risk assessment or recommendation? Ask it to cite the specific data, document, or input the recommendation is based on. If it can point to something verifiable, the output is generally safe to act on; if the source is vague or can’t be confirmed, treat it as needing human review before acting.
Are more capable AI models less likely to produce wrong outputs? Not necessarily. More capable models produce more plausible, well-reasoned outputs even when they’re wrong, which makes errors harder to spot by eye. The need for a verification step doesn’t decrease as models improve; if anything, it becomes more important.
Do I need to verify every AI output before using it? No, that would remove the time-saving benefit of using AI at all. The verification step matters most for high-stakes outputs: risk assessments going to steering committees, scope recommendations, and stakeholder communications that set expectations.
Has an AI output ever sounded right but turned out not to be, and how did you catch it (or not)?
Yes - AI helped me to write this :)
Unsubscribe
