AI vs. AI: Automated programs are writing better scam emails, and AI is spotting them
According to the FBI, email phishing attacks accounted for nearly $51 billion in losses over the past 10 years — and the number is only expected to grow with the introduction of artificial intelligence.
AI is already helping scammers craft convincing spam emails — devoid of the bad syntax and poor spelling that use to allow most people to spot the scam and send the email to the trash without even opening it.
Just ask Grant Walsh, the IT director for a North Carolina company called Flow Control Group. The company recently found itself on the receiving end of a business email compromise attack, a scam in which cybercriminals impersonate executives or clients and convince employees to transfer large funds.
“There was just an email that came in to finance: Hey, can you please update your wire information to X? That financial person updated the wire information, and out goes a check for a six-figure number, and away that money goes,” Walsh said.
It wasn’t just the money. Walsh noticed that the company was getting emails that looked like they were written by corporate higher-ups inside the company.
But they were fake.
“It can look like it’s coming from a CFO,” Walsh said. “That just makes me a little nervous, a little scared.”
It made Walsh think: If these emails were convincing now, just imagine what will happen as AI continues to improve.
“Where is it going to be a year from now?” he asked. So Walsh decided to fight fire with fire — or, AI with AI.
To do so, he called in Roy Darnell, a team lead at Perception Point, an Israeli cybersecurity company that is using AI to battle these kinds of business email compromise scams. Typically, Perception Point’s AI-based system spots these phishing attacks by identifying patterns.
“All you have to do is look for messages that were sent a million times the same way,” Darnell said.
But spotting these kinds of bulk emails are becoming less helpful because AI is tinkering with that too: it can write customized messages really quickly. So Darnell had to teach his AI models to go beyond simple word or pattern recognition.
“So let’s say you get an email and you want to try to understand whether it’s legitimate or if it’s malicious,” Darnell said. “The first thing you would do is try to break it apart.”
The AI creates a bunch of layers — filters, really — that look at IP addresses, what the message is, how it’s written, and even the sentiment behind it.
And because the AI is looking for and flagging so many things, it has a better chance of spotting a fake email, “if you have five, 10, 50 layers, it becomes virtually impenetrable,” Darnell said.
Grant Walsh pulled up one of the emails the system caught.
“This looks like it’s coming from our VP of marketing [and] going to our CFO, asking to change some banking information,” Walsh said.
But it turns out it wasn’t her email. And this isn’t a rare thing.
Just in the last month or so, Walsh said his company received 16 pretty convincing impersonation emails, but they didn’t get through. Thanks to this kind of AI defense, the emails ended up in the trash.
Stories You Might Like
The future of this podcast starts with you.
Every day, the “Marketplace Tech” team demystifies the digital economy with stories that explore more than just Big Tech. We’re committed to covering topics that matter to you and the world around us, diving deep into how technology intersects with climate change, inequity, and disinformation.
As part of a nonprofit newsroom, we’re counting on listeners like you to keep this public service paywall-free and available to all.
Support “Marketplace Tech” in any amount today and become a partner in our mission.
