Morris II: The first AI worm breaks free

[Teacher]

Models from OpenAI and Google are under attack. Will the industry giants be able to protect their users?

In a new study, a team of scientists has demonstrated the creation of a first-of-its-kind malicious AI worm that can automatically spread between generative AI agents, paving the way for potential data theft and spamming. This breakthrough points to a new kind of cyberattack that can be carried out in connected, autonomous AI ecosystems.

Researchers at Cornell Tech, including Ben Nassi, Stav Cohen, and Ron Bitton, developed the worm, named Morris II after the original Morris computer worm that caused havoc on the Internet in 1988. Experiments by experts have shown how such a worm can attack an AI-based mail assistant to steal data from emails and send spam, while violating some security measures in the ChatGPT and Gemini systems.

The study focuses on "hostile self-replicating requests" that cause the AI model to generate a new request in its response. This method resembles traditional SQL Injection and Buffer Overflow attacks, the researchers say.

To demonstrate the worm's capabilities, the researchers created a mail system that can send and receive messages using generative AI, connecting to ChatGPT, Gemini, and an open LLM called LLaVA. Experts found two ways of operating the system: using a text-based self-replicating query and embedding the self-replicating query in the image.

The study highlights that generative AI worms are a new security risk that should worry startups, developers, and tech companies. Although generative AI worms have not yet been discovered in the wild, security experts believe that the risk of their appearance in the future is quite high.

The researchers reported their findings to Google and OpenAI, emphasizing the need to develop more robust security systems and warning developers against using malicious input. Google declined to comment on the study, while an OpenAI representative noted that it is working to strengthen the resilience of its systems to such attacks.

Some ways to protect against potential AI worms already exist, such as using traditional security approaches and ensuring human participation in the decision-making process of AI agents. Experts emphasize the importance of developing secure applications and monitoring to prevent such threats.

 

[Teacher]

Experimental AI worm Morris II can steal data and send spam

A team of researchers from the United States and Israel created a worm designed for applications based on generative AI, and set it on a smart email assistant of its own design. The experiment helped to verify the existence of risks for the associated GenAI ecosystems.

An autonomous email agent sent incoming emails for processing to a service built on the basis of an AI model (ChatGPT 4.0, Google Gemini Pro, and the LLaVA opensource project were tested during the study). The attacks of the computer worm, named Morris II, used malicious self-replicating hints (stimuli that provoke responses with a new request).

Two types of emails were used as the attack vector: text and as an image file. In the first case, the inserted malicious incentive "poisons" the email assistant's database-the latter has to supplement the request with search results from external sources for greater accuracy and reliability of GenAI service responses.

As a result, the attack leads to a jailbreak, opening up the possibility of stealing confidential information from emails (names, phone numbers, bank card details, social insurance numbers, etc.). When new messages are received on the network, the generated response with such content is transmitted to other hosts and deposited there in databases.

In another Morris II scenario, a self-replicating stimulus is embedded in the image, and the output from the GenAI model dictates redirection of the message to other hosts. In this way, you can use a single email to distribute spam, propaganda materials, or content prohibited by law.

The developers were informed about the possibility of bypassing ChatGPT and Gemini restrictions. OpenAI decided that this is a new way of injecting incentives, relying on the absence of checks and filtering of user input. Google declined to comment, but the study attracted interest there.

The authors of Morris II expect attacks using analogs of their laboratory sample in the next two to three years. Due to the new threat, developers of AI assistants are advised to take risks into account in advance, and users are advised never to leave such assistants unattended.

• Video: