Don't trust AI just because it sounds sure

Automation bias is the tendency to favor an automated system's output even when other information says it's wrong. A 2010 review by Raja Parasuraman and Dietrich Manzey found it affects both beginners and experts and isn't easily trained away, which makes confident-sounding AI tools especially risky to trust without checking.

Here's something worth knowing about every AI tool you use: it sounds exactly as confident when it's wrong as when it's right.

Same polished sentences. Same helpful tone. Same neat bullet points. There's no hesitation in its voice, no "um, I think maybe," no nervous glance. And our brains are wired to trust confidence.

That combination has a name, and researchers were studying it long before ChatGPT existed.

Automation bias

For decades, psychologists have studied what happens when people work alongside automated systems: pilots with autopilot, doctors with diagnostic software, operators with computer alerts.

In 2010, researchers Raja Parasuraman and Dietrich Manzey published a major review of this research. They described two related problems.

Automation complacency is when people monitor an automated system less carefully because it usually works.

Automation bias is when people favor the automated system's suggestion, even when other information says it's wrong. It shows up as two kinds of mistakes: missing a problem because the system didn't flag it, and doing the wrong thing because the system recommended it.

Here's what struck me most about their review. These problems showed up in both beginners and experts. And according to the research they reviewed, they couldn't simply be trained or instructed away.

Why AI makes this worse

Old-style automation was usually narrow. An autopilot flies a plane. A spell-checker checks spelling.

Today's AI tools answer almost anything, in fluent, confident language. They're right a lot of the time, which builds trust fast. And when they're wrong, they're wrong in a way that sounds exactly like being right. That's a perfect setup for automation bias.

The more often a tool is right, the less carefully we check it, and the more a single wrong answer can cost us.

How to use AI without falling for it

The goal isn't to stop using AI. It's to build habits that keep your judgment switched on.

1. Match your checking to the stakes. Not everything needs a deep review. A brainstormed list of blog titles? Use it freely. Anything involving money, contracts, legal issues, health, specific facts, or a client's name? Verify it.

2. Ask for sources, then check them. When AI gives you a fact, ask where it came from. Then actually look. AI can produce citations that look real but don't say what it claims, or don't exist at all.

3. Read the original for anything important. AI summaries are great for deciding what to read. They're not a substitute for reading the parts that matter, especially contracts, policies, and anything you're signing.

4. Ask it to argue against itself. Try: "What might be wrong with this answer?" or "What would someone who disagrees say?" It's a quick way to surface weak spots.

5. Notice when you stop checking. The research says complacency creeps in with trust. If you realize you haven't questioned an AI answer in weeks, that's your sign to slow down, not a sign the tool has become perfect.

6. Keep a human in the loop for client-facing work. In my own business, AI drafts a lot, but nothing goes to a client without my eyes on it first. That rule has saved me more than once.

Confidence isn't competence

We all know the person who says everything with total certainty and is wrong half the time. We learn to double-check them. AI can be a lot like that person, just much more helpful and much more polite. It's worth using. It's also worth double-checking.

So keep using AI. Let it save you time. Just remember that it doesn't know when it's wrong.

That's still your job.

Good habits stick when someone's checking in. If you want weekly accountability for the habits that matter most in your business, here's how I work.

Study to link: Parasuraman & Manzey (2010), "Complacency and Bias in Human Use of Automation," Human Factors

Frequently asked questions

What is automation bias?

Automation bias is over-relying on an automated system's suggestions. It shows up as two kinds of mistakes: missing a problem because the system didn't flag it, and doing the wrong thing because the system recommended it.

What is automation complacency?

Automation complacency is monitoring an automated system less carefully because it usually works. The more often a tool is right, the less carefully people tend to check it.

How can I avoid trusting AI too much?

Match your checking to the stakes, ask for sources and verify them, read the original for anything you're signing, ask the AI what might be wrong with its answer, and keep a human review step for client-facing work.