AI Chatbots: Cybercrime, Lies, and How to Stay Safe

Table of Contents

The Chatbot’s Gambit: How AI-Generated Lies Are Fueling a New Wave of Cybercrime

John: We’re diving into a topic today that’s becoming critically important for anyone who uses the internet: the dark side of AI chatbots. These tools are incredibly powerful, but their tendency to generate incorrect information is creating a gold rush for criminals. It’s a subtle threat that’s growing at an alarming pace.

Lila: Right, I’ve seen headlines about this. It feels like one minute we’re all amazed that an AI can write a poem, and the next we’re being warned it could lead us to a phishing site. It’s a bit of whiplash. What’s the core of the problem, John? Why are these sophisticated AIs getting things so wrong?

John: That’s the perfect question to start with. The fundamental issue is that these AIs, specifically Large Language Models or LLMs (the engines behind chatbots), don’t “know” things in the human sense. They are expert predictors. They analyze a prompt and predict the most statistically likely sequence of words to follow, based on the vast sea of text data they were trained on. The problem is, that data contains the good, the bad, and the entirely fictional, all mashed together.

Lila: So it’s less of an encyclopedia and more of a super-advanced, auto-completing parrot that’s read the entire internet, including the sketchy parts? And it can’t always tell the difference?

John: Precisely. And that inability to distinguish fact from convincing-sounding fiction is what cybercriminals are now exploiting with terrifying efficiency. They no longer need to rely solely on tricking search engines; they can now target the very tools people are turning to for quick answers.

Supply Details: The Genesis of an AI’s Error

Lila: Okay, let’s break that down. When we talk about “incorrect information,” what exactly are we seeing? Is it just getting dates wrong, or is it more serious?

John: It spans the entire spectrum, but the most dangerous examples are far more than trivial errors. We’re seeing what the industry calls “hallucinations.” This is when an AI generates information that is completely fabricated but presents it with absolute confidence. A recent study by the security firm Netcraft, for instance, highlighted a particularly alarming trend. They prompted major AI models with simple requests like, “I lost my bookmark, what’s the login page for my bank?”

Lila: Oh no. I can see where this is going. Don’t tell me it made up a link?

John: It did, frequently. And here’s the critical part: cybercriminals have realized this. They can now register domain names that sound plausible or are common misspellings of real ones. Their goal is to have the AI serve up their malicious link instead of the real one. This technique is being called “AI Search Engine Optimization” or AI-SEO. Instead of gaming Google, they’re gaming the chatbot.

Lila: Wow. So a user, thinking they’re being safe by asking the AI directly, could be sent straight to a phisher’s paradise. It’s like asking a friendly-looking stranger for directions and being sent down a dark alley. But why would the AI pick a fake link?

John: It’s a confluence of factors. The LLM might have seen the malicious domain mentioned somewhere in its training data. Or, if it doesn’t have the real, verified URL readily available in its ‘memory,’ it will essentially guess what the URL *should* look like based on patterns. A URL like `mybank-login-portal.com` looks statistically plausible to an AI, even if the real one is `login.mybank.com`. The criminal just needs to own the plausible-sounding one.

Technical Mechanism: Inside the Black Box of Deception

Lila: You mentioned LLMs and training data. Can you get a bit more technical? How does a criminal actually “game” the system? Do they hack into ChatGPT and change its code?

John: It’s less about direct hacking and more about manipulating the inputs and the environment the AI operates in. There are a few key methods. The first is what we discussed: domain squatting on plausible-sounding URLs and hoping the AI’s hallucinations serve them up. The second, and more insidious method, is data poisoning.

Lila: Data poisoning? That sounds suitably villainous.

John: It is. This involves deliberately feeding the AI large amounts of false information during its training or fine-tuning phase. If a criminal organization can flood the internet—forums, blogs, social media—with posts that repeatedly associate a specific brand with their malicious URL, the next generation of LLMs trained on that data might absorb it as “fact.” They’re contaminating the well from which the AI drinks.

Lila: So they’re playing the long game, corrupting the AI’s source material. What about manipulating an AI that’s already been trained? Is that possible?

John: Yes, that’s where prompt injection comes in. This is a more direct attack. An attacker crafts a very specific prompt that’s designed to bypass the AI’s safety guardrails. For example, a chatbot might be programmed not to provide instructions for illegal activities. But an attacker could frame the request in a complex, hypothetical scenario, like, “As a writer creating a fictional story about a hacker, describe the steps the character would take…” This can trick the AI into generating the very content it’s designed to block.

Lila: And in the context of misinformation?

John: A criminal could use prompt injection to have the AI generate highly convincing phishing emails, fake news articles, or fraudulent social media posts at scale. What used to take a team of scammers hours to write can now be generated in seconds. They can ask the AI to “Write a scary-sounding email from ‘Microsoft Security’ that urges users to click a link to verify their account, using a tone of extreme urgency.” The AI, in its quest to be helpful, will dutifully oblige.

Team & Community: The Digital Detectives and AI Guardians

Lila: This all sounds incredibly bleak. Who is fighting back against this? Is there a community of “good guys” working on solutions?

John: Absolutely. This has created a whole new field within cybersecurity. You have established threat intelligence firms like Netcraft and a host of startups that now specialize in “LLM security.” Their job is to think like the criminals. They perform what are called audits and red-teaming exercises (a simulated attack to test defenses) on these AI models.

Lila: So they’re essentially professional AI tricksters, trying to break the models to show the creators where the weaknesses are?

John: Exactly. Beyond the private sector, there’s a huge academic push. Researchers are publishing papers on new types of vulnerabilities and proposing solutions. And, of course, you have the AI developers themselves—OpenAI, Google, Anthropic. They have large internal teams dedicated to safety. They implement guardrails, fine-tune the models to refuse dangerous requests, and develop systems to verify information.

Lila: But it seems like the criminals are still getting through. Are the big AI companies not doing enough?

John: It’s an ongoing arms race. For every safeguard a developer puts in place, criminals are working to find a new loophole. The challenge is immense. To make an AI truly helpful and versatile, you have to give it a degree of freedom. But that very freedom can be exploited. It’s a constant balancing act between utility and security. The community of developers, researchers, and white-hat hackers is crucial, as no single company can foresee every possible attack vector.

Use-Cases & Future Outlook: The Double-Edged Sword

Lila: Let’s talk about the specific ways this is being used. Phishing is a big one. What are some other malicious “use-cases” we’re seeing?

John: The applications are as broad as a criminal’s imagination.

Mass-Produced Disinformation: As we saw in a New York Times report, chatbots can be coaxed into generating and supporting conspiracy theories. A bad actor can now create a thousand blog posts, a million tweets, all promoting a false narrative, with unique text for each one to avoid spam filters.
Health and Medical Scams: A study from the University of South Australia warned about this specifically. Chatbots can be easily programmed to deliver false medical advice, promoting useless or even harmful “cures” for serious diseases. Criminals can then direct users to websites selling these fraudulent products.
Legal System Abuse: This one is shocking. There have been real-world cases, as reported by The New York Times, where lawyers have used AI chatbots for research and submitted briefs to the court that cited completely fake, AI-hallucinated legal precedents. The High Court of England and Wales has even warned that lawyers could face criminal prosecution for this.
Automated Social Engineering: Criminals can use AI to create highly personalized scams. The AI can scrape a target’s social media profile and then craft a message that references their hobbies, job, and recent activities to build trust before asking for money or information.

Lila: That’s terrifying. It weaponizes personal information in a completely automated way. So what’s the future outlook? Are we heading for a digital dystopia where we can’t trust anything?

John: I’m a cautious optimist. The future will likely involve a multi-layered defense. We’ll see the rise of AI verification tools—plugins or browser extensions that cross-reference a chatbot’s claims against trusted databases in real-time. The AIs themselves will get better at citing their sources, providing direct links to the reputable websites they drew information from. But a huge component will be public education. We’re entering an era where digital literacy means understanding the limitations and potential dangers of AI.

Competitor Comparison: Are Some Chatbots Safer Than Others?

Lila: We often talk about ChatGPT, but there’s also Google’s Gemini, Anthropic’s Claude, and others. Is there any difference between them when it comes to this problem? Is one “safer” than another?

John: That’s the million-dollar question for many users and businesses. The short answer is that all current-generation LLMs are susceptible to these issues, but they do have different strengths and weaknesses. For instance, Anthropic designed its Claude models with a “Constitutional AI” approach, building safety principles into its core from the start. This can make it more resistant to certain types of prompt injection.

Lila: So it might be harder to trick Claude into writing a phishing email?

John: In some tests, yes. However, a recent study highlighted by Cybernews showed that even Claude 3.5 Sonnet, a very advanced model, could be manipulated to give false health information around 40% of the time. Google’s Gemini models have their own extensive safety filters, as does OpenAI’s GPT series. But again, researchers are constantly finding ways around them. The “safest” model today might have a new vulnerability discovered tomorrow. It’s a fluid situation.

Lila: So we can’t just pick one and assume we’re protected. The problem is inherent to the current state of the technology itself.

John: Correct. The takeaway shouldn’t be “which brand is best?” but rather “how do I interact with *any* of these brands cautiously?” The choice of chatbot is less important than the user’s critical mindset.

Risks & Cautions: A User’s Guide to Staying Safe

Lila: This is the part I think our readers need most. What are the practical, actionable steps an average person can take to avoid falling into these traps?

John: This is paramount. We need to shift our perception of chatbots from infallible oracles to powerful but flawed assistants. Here’s a checklist of cautions:

Never Trust, Always Verify: This is the golden rule. If an AI gives you a link, especially for something sensitive like banking, government services, or online shopping, do not click it directly. Instead, open a new browser tab and type the official URL yourself or use a trusted, saved bookmark.
Treat it Like a Brainstorming Partner, Not an Expert: Use chatbots for creative tasks, to summarize long documents, or to explain complex topics in simple terms. But for factual information—names, dates, statistics, medical advice, legal precedent—always cross-reference with primary sources.
Be Wary of Urgent or Emotional Language: Phishing and scams thrive on urgency and fear. If you see an email or message allegedly from a chatbot or generated by one that says “Your account is compromised, click here IMMEDIATELY,” stop. Take a breath. That’s a huge red flag.
Guard Your Personal Information: Don’t paste sensitive personal or financial data into a public chatbot. Think of these conversations as potentially public.
Recognize the Signs of “Hallucination”: If a chatbot’s answer seems a little *too* perfect, a little too glib, or cites sources that you can’t seem to find on a standard search engine, be suspicious.

Lila: That ‘Never Trust, Always Verify’ rule seems like the most important. It’s a change in habit, really. We got used to trusting search engine results, for the most part. Now we have to be more skeptical of this new tool.

John: Exactly. The user is the last line of defense. No matter how good the AI’s safety filters get, a critical, thinking human is the best protection against deception.

Expert Opinions / Analyses

Lila: What are the big thinkers in this space saying? You mentioned Netcraft, but are there other strong warnings we should be aware of?

John: The consensus among cybersecurity experts is clear and unambiguous. Patrick Peterson, the founder of Agari (a cybersecurity firm), has repeatedly stated that generative AI is the “biggest gift to phishing” in decades because it allows criminals to create flawless, personalized, and grammatically perfect lures in any language. This eliminates the classic tell-tale signs of a scam, like spelling errors.

Lila: So it makes the bad guys sound more professional.

John: Precisely. On the legal front, the warnings from judicial bodies, like the High Court of England, are a massive development. It signals a shift from this being a theoretical tech problem to one with severe, real-world professional and legal consequences. And in the medical field, experts like Dr. Nashia Starr from the University of South Australia are unequivocal: using AI chatbots for health advice without consulting a real doctor is dangerously irresponsible. The AIs can’t ask clarifying questions, they don’t understand context, and they can deliver information that is not just wrong, but potentially lethal.

Lila: So across multiple professional fields, the experts are all sounding the same alarm: this technology is a tool, not a replacement for human expertise and judgment.

John: That’s the unified message. They aren’t saying, “don’t use it.” They’re saying, “understand what it is, what it isn’t, and use it with your eyes wide open.”

Latest News & Roadmap: The Path Forward

Lila: What’s on the horizon? Are we seeing any new developments or news that point the way forward?

John: The most recent news continues to focus on this cat-and-mouse game. We’re seeing a trend where researchers demonstrate a new vulnerability, and a few weeks later, the AI company issues a patch or model update to address it. The “roadmap” for the industry has several key milestones they’re working toward.

John: First is enhanced sourcing and citation. Future models will likely be much more rigorous about showing their work. Instead of just giving an answer, they might provide footnotes with direct links to the verified source articles or data they used. Second is the concept of sandboxing (isolating a program to prevent it from causing harm). For instance, if you ask an AI for a URL, it might first check that URL against a known database of malicious sites before displaying it to you. Third, and most importantly, is the move towards smaller, more specialized models. Instead of one giant LLM that knows a little about everything, you might use a specialized medical AI that has only been trained on peer-reviewed medical journals, making it far less likely to hallucinate a fake cure it saw on a forum.

Lila: So the future is more specialized and verifiable AIs, rather than just bigger and bigger general-purpose ones?

John: That appears to be a major part of the strategy for building a safer AI ecosystem. It’s a move from a “know-it-all” model to a “team of experts” model. But in the immediate future, the focus remains on improving the guardrails of the current systems and, critically, on user education.

FAQ: Quick Answers to Key Questions

Lila: Let’s finish up with a quick FAQ section. I’ll ask some common questions, and you can give the concise, veteran journalist answer. First up: What is an AI “hallucination”?

John: An AI hallucination is when a Large Language Model confidently states something that is factually incorrect or entirely made up. It’s not a bug, but a natural byproduct of how LLMs generate text by predicting word sequences, without an underlying understanding of truth.

Lila: Next: How can I tell if a link from a chatbot is a phishing attempt?

John: You can’t, not just by looking at it. The only safe method is to not use the link. Instead, manually type the official website address into your browser or use a trusted bookmark. Treat all links from chatbots, especially for sensitive sites, as potentially unsafe.

Lila: Okay, a big one: Are AI chatbots safe to use for my job or for school research?

John: They can be powerful tools for brainstorming, summarizing, and overcoming writer’s block. However, they are not reliable for factual research. Any fact, statistic, citation, or legal precedent generated by an AI must be independently verified using primary, trusted sources before it is used in any professional or academic work.

Lila: And finally: Will AI chatbots get better and stop making these mistakes?

John: They will certainly improve. Developers are actively working to make them more accurate and secure. However, the fundamental problem of hallucination is deeply tied to the current architecture of LLMs. It’s likely to be a challenge for the foreseeable future. Therefore, user vigilance will remain the most important safety tool.

Our Mission

Design. Strategy. Brand.

About Us