The Dark Side of Generative AI: Vulnerabilities Exposed in Google’s Gemini System

A group of security researchers has demonstrated a sophisticated hack that can take control of smart home devices using poisoned Google Calendar invites. The attacks, which were showcased at the Black Hat cybersecurity conference in Las Vegas, exploited vulnerabilities in Google’s Gemini artificial intelligence system, allowing hackers to remotely turn on lights, shutters, and even the boiler in a Tel Aviv apartment.

The researchers, from Tel Aviv University and the Technion Israel Institute of Technology, used a technique called indirect prompt injection to trick Gemini into performing malicious actions. They created a poisoned calendar invite that included instructions to turn on the smart home products at a later time. When the researchers asked Gemini to summarize their upcoming calendar events, the dormant instructions were triggered, and the products came to life.

“This is a serious concern,” said Ben Nassi, a researcher at Tel Aviv University. “LLMs are being integrated into physical humanoids, semi-autonomous cars, and other machines, and we need to understand how to secure them before we integrate them with these kinds of machines.”

The researchers also demonstrated other attacks, including sending spam links, generating vulgar content, opening the Zoom app and starting a video call, and even stealing email and meeting details from a web browser. The attacks were made possible by the use of indirect prompt injection, which is considered one of the most serious AI security problems.

According to the researchers, indirect prompt injection is a type of attack that involves inserting malicious prompts into an AI system without the user’s knowledge or consent. These prompts can be inserted through various means, including text on a website, documents, or even email subject lines.

In the case of the Google Calendar invite, the researchers inserted the malicious prompt into the title of the invitation. When Gemini was asked to summarize the user’s calendar events, the AI system executed the instructions in the prompt, resulting in the smart home devices being turned on remotely.

Google’s Andy Wen, a senior director of security product management for Google Workspace, acknowledged that the vulnerabilities were not exploited by malicious hackers, but the company is taking them “extremely seriously” and has introduced multiple fixes.

“We’re hopeful that we can get to a point where the everyday user doesn’t really worry about prompt-injection attacks that much,” Wen said. “It’s going to be with us for a while, but we’re working on it.”

The researchers also highlighted the importance of user confirmation when interacting with AI systems. “Sometimes there’s just certain things that should not be fully automated, that users should be in the loop,” Wen said.

The researchers believe that the attacks demonstrate the need for greater emphasis on AI security in the development of LLM-powered applications. “Today we’re somewhere in the middle of a shift in the industry where LLMs are being integrated into applications, but security is not being integrated at the same speeds of the LLMs,” said Ben Nassi.

The researchers’ findings have sparked a renewed debate about the risks and vulnerabilities associated with AI-powered systems. As AI becomes increasingly integrated into our daily lives, the need for robust security measures becomes more pressing.

“We need to understand how to secure LLMs before we integrate them with these kinds of machines,” Nassi said. “This is a wake-up call for the industry.”

The researchers’ paper on the attacks has been published in a research journal, and their findings have been shared with Google and other tech companies to help strengthen AI security measures.