Where AI Chatbots Fall Short, and Why It Matters in a Cyber Emergency
More people are typing “I think I’ve been hacked” into an AI chatbot than into a search bar. Our firm often cleans up what happens next, and the pattern looks a lot like self-diagnosing a chest pain on the internet at 2 a.m.
Written by Paul Pioselli on June 24, 2026
10 min read
IN THIS ARTICLE
A woman called our firm last spring already three days into her own incident. She’d woken up to a fraud alert, asked a popular AI chatbot what to do, and dutifully followed the list it gave her. She reset her email password, from the same laptop the attacker was sitting on. She deleted the suspicious messages “to be safe.” She factory-reset her phone before we ever saw it. Every step felt productive. Every step also destroyed evidence we needed, tipped off the intruder, and left the actual hole, a compromised password manager, wide open.
She didn’t do anything foolish. She did exactly what the tool told her to do, in exactly the order it listed. That’s the problem I want to talk about, because I see it almost every week now: smart, capable people using ChatGPT, Claude, Gemini, and other AI chatbots as their personal cybersecurity expert, and unintentionally making a bad situation worse.
To be clear up front: I am not anti-AI. Our team uses these tools daily. They’re extraordinary for explaining concepts, drafting a checklist, or helping a nervous person understand a scary term at midnight. The danger isn’t the technology. It’s the moment it gets handed the steering wheel during an actual emergency.
The short version
- AI chatbots are excellent tutors and terrible first responders. Use them to learn, not to run an active incident.
- They sound certain even when they’re wrong. Independent testing in 2025 found most leading models were more likely to give a confident incorrect answer than a correct one on hard questions.
- Cyber emergencies are about sequence and triage what you do, and in what order. A generic checklist with no triage can destroy evidence, alert the attacker, or reinfect a clean device.
- Do your own research from authoritative .gov sources, and bring in a professional the moment money, identity documents, ongoing access, or legal exposure is involved.
Cyber self-diagnosis is the new WebMD
If you’ve ever typed your symptoms into a search engine and convinced yourself you had something terminal, you already understand this post. Researchers have a name for the spiral: cyberchondria, the escalating anxiety that comes from self-diagnosing online. And the accuracy problem behind it is well documented. A widely cited audit of online symptom checkers published in The BMJ found they listed the correct diagnosis first only about a third of the time and gave appropriate triage advice in roughly half of cases. Studies generally put experienced physicians far ahead of that.
The medical world also learned something that maps directly onto mine: large language models can amplify the self-diagnosis spiral. Because a chatbot tends to be agreeable and to personalize its answer to how you framed the question, it can confidently validate a conclusion you’d already half-decided on. In medicine that has led to documented harm, including a case study in Annals of Internal Medicine of a man who developed a rare poisoning after following dietary advice from ChatGPT.
A symptom checker doesn’t panic, but it also doesn’t examine you. A chatbot doesn’t lie on purpose, but it also can’t see your accounts, your network, or the attacker who’s still logged in.
Cybersecurity has its own version of every one of these failure modes. Swap “chest pain” for “my email is sending spam” and the pattern is identical: a frightened person, an authoritative-sounding answer, and no one actually examining the patient.
Why AI chatbots are so confidently wrong about security
The single most important thing to understand about these tools is that they are built to produce a plausible answer, not a verified one. When a model is uncertain, it has no built-in mechanism to say so — it generates the most statistically likely text and delivers it with the same calm confidence it uses for things it knows cold.
This isn’t a fringe concern. Consider the evidence:
>90%
Of 40 leading AI models tested against the 2025 AA-Omniscience benchmark, all but four were more likely to give a confident, incorrect answer than a correct one on difficult questions.
58–82%
Hallucination rate for general-purpose chatbots on legal research queries in a Stanford HAI–affiliated study; even specialized tools built to reduce errors still hallucinated more than 17% of the time.
~20–64%
Rate at which tested models invented non-existent software package names — a hallucination attackers now exploit by registering those fake names with malware (“slopsquatting”).
Now layer in the security-specific risk. As IBM has pointed out, in a live incident a hallucination doesn’t just waste time — it can hand you an inaccurate “next step” that prolongs the breach or actively makes recovery harder. The chatbot can correctly tell you something looks suspicious and then confidently recommend the wrong response to it.
And remember the agreeableness problem from medicine. If you open with “I’m pretty sure my account is fine, I just want to double-check,” you will get a very different answer than the situation may deserve. The tool is optimized to be helpful and affirming. An attacker doesn’t care how reassured you feel.
Four ways DIY AI advice quietly makes things worse
Inaccuracy is only half the danger. The other half is that even correct general advice, given without triage, can be harmful when applied to a real emergency. Here are the four I see most often.
- 1. It destroys the evidence you’ll need later. “Delete the suspicious emails. Reset the phone. Clear your history.” Reasonable-sounding, and exactly wrong. Those logs, headers, and timestamps are how a professional traces the breach — and what your bank, insurer, or the police may require. Once it’s wiped, it’s gone.
- 2. It tips off the attacker. Changing a password from the compromised device, or confronting a scammer, often just tells the intruder you’re onto them — prompting them to lock you out, drain accounts, or burn what they’ve stolen before you can contain it.
- 3. It gives you steps with no order of operations. A checklist is not a plan. Real incidents are about sequence: contain before you clean, preserve before you change, secure the entry point before you reset everything downstream. A flat list invites you to do step 7 before step 2.
- 4. It offers false reassurance. The most dangerous output isn’t a wrong step — it’s “you’re probably fine.” A confident all-clear from a tool that cannot actually see your systems sends people back to sleep on top of an active compromise.
What a professional actually does differently: triage first
Here’s the part the chatbots leave out, because they were never built to do it. Before a competent responder touches anything, they triage. The word comes from emergency medicine for a reason — it means assessing severity and deciding what gets handled, in what order, with what resources.
This isn’t improvised. The U.S. National Institute of Standards and Technology publishes the framework the whole industry leans on — NIST SP 800-61, updated in 2025 — and it organizes incident response into a disciplined lifecycle: prepare, detect and analyze, contain, eradicate, recover, and learn. The Cybersecurity and Infrastructure Security Agency (CISA) publishes companion playbooks that say the quiet part out loud: move through the first steps in sequence. Their ransomware guidance, for instance, explicitly warns that shutting a machine down the wrong way erases evidence stored in memory.
So when someone calls my firm, we don’t open with a checklist. We ask: Is this still happening right now? What can the attacker still reach? What absolutely cannot be lost — money, documents, evidence? What’s the one thing we contain before we do anything else? That ordering — not any single tip — is the expertise. A chatbot can recite the steps. It cannot weigh your situation and tell you which one is step zero.
The stakes are real, and rising fast
I’m not trying to scare you; fear is what gets people clicking the wrong link in the first place. But the scale of what’s out there is worth seeing plainly, because it’s why “close enough” advice matters so much.
$16.6B
Reported losses in the FBI’s 2024 Internet Crime Report — a 33% jump over the prior year — across nearly 860,000 complaints. The top categories: phishing/spoofing, extortion, and personal data breaches.
$12.5B
Consumer fraud losses reported to the FTC in 2024, up 25%, alongside more than 1.1 million identity-theft reports.
48%
Share of identity-fraud victims who sought help from the Identity Theft Resource Center yet were still unresolved up to a year later — a reminder that how you respond early shapes how long you suffer.
The FBI also notes that older adults are hit hardest, and that its own recovery teams manage to freeze only a fraction of stolen funds — because with wire and crypto fraud, the window to act is measured in hours. That’s the brutal math of a cyber emergency: the cost of a wrong first move isn’t embarrassment, it’s the difference between recovering your money and not.
A smarter way to use AI, and a simple test for if you need a human expert
So where does that leave the AI chatbot you already have open in another tab? Use it the way you’d use a good medical website: to understand, not to treat. Ask it to explain what a phishing kit is, what two-factor authentication actually does, what questions you should be asking. Then verify anything that matters against an authoritative source before you act on it.
To make the “when do I call someone” decision easier, here’s the rough triage I’d give a friend.
GREEN — USUALLY DIY-ABLE
Low stakes, no ongoing access
One spam email you didn’t click. A single weak password to upgrade. General “how do I stay safe” learning. A chatbot plus official guidance is genuinely fine here.
AMBER — RESEARCH CAREFULLY, GET A SECOND OPINION
One account compromised, contained, no money moved
A single hacked social or email account you’ve since locked down. Follow the provider’s official recovery flow, watch for spread to linked accounts, and don’t hesitate to ask a professional if anything feels off.
RED - CALL AN EXPERT
Money, identity, ongoing access, or legal exposure
Funds moved or at risk · identity or government documents involved · an attacker who may still be inside · multiple linked accounts · extortion, sextortion, or stalking · anything you may need evidence for with police, insurance, or a court. This is where correct sequencing and evidence handling pay for themselves.
How to do your own research without doing harm
“Do your own research” is good advice right up until the sources are bad. The fix is to anchor on authorities that have no incentive to upsell you and every incentive to be right:
- IdentityTheft.gov (FTC) — the official, step-by-step recovery plan for identity theft, tailored to your situation.
- IC3.gov (FBI) and ReportFraud.ftc.gov — where to report a cybercrime so funds can potentially be frozen and patterns tracked.
- CISA’s Secure Our World — plain-language, vendor-neutral security basics from the U.S. government.
- The official support pages of the actual service involved — your bank, email provider, or platform — not a third-party page that merely looks official.
If a chatbot’s answer disagrees with one of these, trust the authority. And if you’re in the red zone above, the single best research you can do is to stop researching and pick up the phone.
Frequently Asked Questions (FAQ)
Can ChatGPT or another AI chatbot fix a hacked account?
It can describe generic steps, but it can’t see your accounts, confirm whether an attacker still has access, or sequence the steps for your specific situation. For an active compromise, follow the official recovery page for the affected service and a trusted source such as the FTC’s IdentityTheft.gov — and bring in a professional if money, identity documents, or multiple linked accounts are involved.
Is it safe to ask an AI chatbot for cybersecurity advice?
For learning concepts and vocabulary, yes. As an incident-response plan, no. Chatbots are known to produce confident, incorrect answers and tend to agree with how you frame a question. Treat their output as a starting point to verify, never as instructions to follow blindly during an emergency.
What should I do first if I think I’ve been hacked?
Stop, don’t delete anything, and don’t act on the first instruction you see. Professionals triage first: contain the threat, preserve evidence, then prioritize what to fix and in what order. Disconnect a clearly compromised device, change passwords from a different trusted device, and contact the affected institutions before wiping or reformatting.
When should I hire a cybersecurity professional instead of doing it myself?
When money has moved or is at risk, when your identity or government documents are involved, when an attacker may still have access, when multiple linked accounts are affected, when you’re being extorted or stalked, or when you may need evidence for police, insurance, or court. Those situations reward the correct sequencing and evidence handling that DIY approaches usually get wrong.
Dealing with a cyber emergency right now?
Don't wait. Every minute matters.
