Study finds AI chatbots easily persuaded to create phishing scams

A study by Reuters and a Harvard University researcher has demonstrated how artificial intelligence (AI) is being used to arm criminals for large-scale fraud. The research tested several leading AI chatbots to plot a simulated phishing scam on 108 elderly volunteers, ultimately finding that the chatbots’ persuasive performance could be used for industrial-scale fraud.

Elon Musk’s Grok chatbot created a deceptive phishing email without prodding and even suggested ways to make it more urgent. While five other popular chatbots initially refused the request, their defences were easily bypassed. After being fed simple ruses, such as claiming the email was for research or a novel, all of the other chatbots produced deceptive messages. The study concluded that, while some chatbots have initial safeguards, these can be easily overcome.

Training Announcement: The BCS Foundation Certificate in AI examines the challenges and risks associated with AI projects, such as those related to privacy, transparency and potential biases in algorithms that could lead to unintended consequences. Explore the role of data, effective risk management strategies, compliance requirements, and ongoing governance of the AI lifecycle and become a certified AI Governance professional. Find out more.