Man
Professional
- Messages
- 3,150
- Reaction score
- 692
- Points
- 113
Artificial intelligence in information security is such a broad topic that it’s easy to get lost in the wilds of speculative forecasts. The concept of AI covers everything from recommendation algorithms that push cat videos to robots that want to kill all humans. To avoid writing a low-budget sci-fi script instead of an article, it’s worth narrowing the focus.
Let's imagine that the AI world has frozen. No revolutions, no superintelligent AGI on the horizon. Let's focus on analyzing large language models (LLM) and their impact on information security. Of course, the release of a powerful new model could blow all these calculations to smithereens, but let's accept this risk. After all, you have to start somewhere.
In this article, I will look at the new opportunities that are opening up for attackers and defenders, and the challenges that the industry faces. Under the cut, you will find an exciting journey to the information security frontier.
Agent-based systems like AutoGPT are good for finding potential targets on social media, while the Red Reaper project , inspired by leaks about the capabilities of the Chinese hacker group ISOON, uses a cocktail of NLP, graph analysis, and GenAI to squeeze the most information out of stolen emails.
Every day, such tools become more numerous. For example, researchers have managed to adapt LLM to help users launch hacker utilities. “Maybe you’ll try this tricky parameter?” the AI whispers, offering context-sensitive command line arguments . And for dessert, there are interpreters of scan results, such as BurpGPT.
Results of comparison of LLM and specialized code analyzers (image source)
Language models can interpret the output of vulnerability scanners, identifying patterns and signatures associated with false positives. Agent systems based on them evaluate and prioritize vulnerabilities based on the likelihood of their successful exploitation.
LLMs make it easy to create personalized phishing texts. Professional experience, hobbies, favorite beer brand — now you don’t have to be Sherlock Holmes to collect, aggregate, and use the victim’s personal information. Instead of sending out similar “chain letters,” attackers can launch a pack of virtual social engineers, each of whom will select an individual approach to the victim. And if you add generative photos and synthesized voice to this, you get a real challenge to critical thinking. These technologies are so often used in Pig Butchering scams that the United Nations Office on Drugs and Crime (UNODC) published a report with entire sections devoted to LLMs and deepfakes.
Remember the recent commotion with the fake photo of the explosion at the Pentagon? While people were scratching their heads trying to figure out what was going on, stock bots were selling off. So thanks to one picture, the stock market was slightly shaken, and someone may have made a good profit.
Also at HOPE 2024, Erica Burgess described (starting at 3:56 in the recording) how AI can now create zero-day exploits. Or imagine a polymorphic payload that changes faster than the boss’s opinion on a deadline, extracting data via a Microsoft Teams webhook. That’s a quick rundown of BlackMamba, a tool that uses LLM to generate keyloggers.
Agent systems based on LLM demonstrated high capabilities for autonomous use of identified vulnerabilities, including SQL injections and XSS attacks. The creators of PentestGPT even claim that their development copes with tasks of easy and medium complexity on the HackTheBox platform and entered the top 1% of hackers on the platform.
LLMs can exploit one-day vulnerabilities with an 87% success rate when given information in the CVE database. And a follow-up to this paper suggests that teams of AI agents can autonomously exploit zero-day vulnerabilities. Another paper describes how LLMs implement sophisticated privilege escalation strategies in Linux environments.
Remember the good old malware builders and crypters? It's pretty much the same story, only with a trendy AI sauce. Professional malware developers have been spoiling customers with tools for creating their own malware strains for a long time now.
Using Malicious LLM (Image Source)
Thanks to LLM, the variety of malware will probably grow, but it is unlikely that we will see anything fundamentally new. Perhaps viruses will start writing poetry or telling jokes before encrypting the hard drive.
LLMs still need human skills and experience to create “worthy” malware. Neural networks have problems with logic , they need detailed instructions, like a first-grader in an IT class. “Don’t use this method of process injection, all antiviruses detect it.” “Try Google Drive for data theft, it is blocked so rarely that it’s embarrassing.” And most importantly, to train a specialized evil LLM, you need high-quality data. This is what causes the main difficulties in scaling social engineering and malware creation.
Thus, those who will benefit most from using AI in complex cyberattacks over the next year will have the resources and experience, as well as access to quality data. The intelligence services probably have an advantage here. They may have large enough malware repositories to unlock the potential of LLM in cyberattacks. But we shouldn’t write off seasoned cybercriminals either. They will also have something to further train open source models on. The darknet has been discussing the refinement and customization of LLM for use in cyberattacks for the second year already. Probably, sooner or later, the bad guys will make some progress.
Even the partial list of malicious LLMs is quite long. Some of them are pre-trained, others have had restrictions removed using special prompts.
But, oddly enough, the first to feel the benefits of large language models will be the greenest cybercriminals. In fact, they are already actively using them as an interactive reference. Publicly available LLMs largely cover their needs, especially in simple operations like spear phishing.
Blue Team has long been one step ahead here. Back when dinosaurs roamed the Earth (okay, in the 90s), they started implementing machine learning in anomaly detection systems (ADS) and intrusion detection systems (IDS). Then in antiviruses and spam filters, but there were still complex unsolved problems in incident response.
Depending on the context, even a serious vulnerability can be given a low priority. But a seemingly average one can easily become a queen if, by eliminating it, you can destroy an entire chain of attacks. But in practice, finding out the details of each alert takes a lot of time.
Almost every vulnerability alert is a quest, despite all the progress in incident response systems. But imagine if the necessary context arrived faster than you could say, “We have an alert, there may be a crime!” LLMs can help with this: extracting, aggregating, and presenting data in a convenient format.
Google security researchers are already writing incident reports using LLM. In their experiment, the model reduced the time it took to write incident summaries by 51%, while maintaining human-level quality.
The time left after incident analysis is spent by a typical security specialist on reporting and “demonstrating results.” Proving that we are not just eating our bread for nothing takes up to 20% of the working time. This epic affects everyone: from the average employee explaining why “qwerty123” is not a password to the security director convincing the board of directors that a new vulnerability is not a reason to dump the company’s shares.
It would seem that answering the question “How secure is our main server?” should be simple, but reporting requires collecting data from a bunch of unrelated systems with mysterious acronyms: CSPM, DSPM, ASPM, IAM…
This is another area where LLMs can be very useful. Imagine coming to work and your report is ready, your presentation is written, and your documentation has updated itself while you were sleeping. A hacker AI will write malware, and an AI on the security side will write a report about it – isn’t that a utopia?
Eli Burstein, the technical lead for cybersecurity and AI research at Google DeepMind, decided to test this. His team selected a bunch of vulnerabilities from Google’s codebase and asked the AI to play tech support. These patches were then sent to human programmers for review. Last year’s best model was able to fix 15% of bugs in plain C++ with input sanitization vulnerabilities. The problem was that it needed to not only fix the vulnerability, but also preserve the functionality of the program.
It doesn't sound very impressive, but it's more than would have been fixed without its help. Eli says that increasing the figure to 80% is quite possible. But to make the technology practical, it will require 90-95% of correctly fixed vulnerabilities and a fundamentally new language model. In general, this idea has prospects. Who knows, maybe soon our IDEs will not only highlight syntax, but also tease for crookedly written code.
It seems that in the end the situation will resemble an endless game of ping-pong between robots. The advantage will shift from one side to the other, but in the end, the “good” and “bad” AI will balance each other out. The ones who will suffer most are those who do not take advantage of the new technologies.
I would like to end on a moderately positive note, but by relying more and more on neural networks, we will face risks that we have not yet realized. No, this is not a machine uprising, I promised to do without futurology. Look, there are already two significant problems:
Of course, if you ask ChatGPT about Samsung's source code, the neural network won't answer, but this data will most likely end up in the training sample. And the guys from Google DeepMind have already proven that fragments of the original dataset can be extracted from ChatGPT. What's more, ChatGPT simply accidentally revealed the headers of someone else's chat history to other users.
These problems can be mitigated with clear corporate policies for the use of LLM and private local neural network models within the company's framework, but what to do with attacks on the neural network models themselves?
British scientists (no, really, British) from the Department of Science, Innovation and Technology have dug through a mountain of scientific papers and found 72 risk assessments related to AI. They have sorted out a whole zoo of vulnerabilities, including new ones specific only to AI. They are laid out by life cycle stages with an assessment of potential impact.
MITRE Atlas
Or here's another, more famous classification - MITRE Atlas (Adversarial Threat Landscape for Artificial-Intelligence Systems). It relies on tactics familiar to security professionals, for example, from web vulnerabilities, but contains many unique risks: generating "poison" data and introducing it into sets for training the model, extracting the dataset through multiple requests to the API of the machine learning model, selecting adversarial examples and who knows what else.
Here we are, on the threshold of the digital frontier, with an endless attack surface ahead. The adventure is just beginning, and it won't be boring. Tighten your stirrups and adjust your hats, developers and security folks - we're in for a thrilling journey into the unknown!
Source
Let's imagine that the AI world has frozen. No revolutions, no superintelligent AGI on the horizon. Let's focus on analyzing large language models (LLM) and their impact on information security. Of course, the release of a powerful new model could blow all these calculations to smithereens, but let's accept this risk. After all, you have to start somewhere.
In this article, I will look at the new opportunities that are opening up for attackers and defenders, and the challenges that the industry faces. Under the cut, you will find an exciting journey to the information security frontier.
Automating Evil: LLM as a Universal Hacker's Tool
In early 2024, Microsoft and OpenAI issued a joint statement about the removal of accounts used by hackers. It carefully wrote that LLMs provide “limited additional capabilities” to bad guys. It’s like saying, “We have a small nuclear reactor in the basement, but don’t worry, it only provides limited additional heat.” In essence, the two companies admitted in this release that LLMs are now part of the hackers’ arsenal. As early as October, OpenAI reported that it had shut down more than 20 cybercriminal operations that used ChatGPT. And I can assure you that despite the technology’s insecurity, hackers have found plenty of uses for LLMs.Lowering the entry threshold
Even research outside the cybersecurity field suggests that the biggest beneficiaries of the LLM revolution are newbies. Whereas script kiddies used to have to grind for years to pull off anything serious, now they just need to learn how to ask the right questions of their personal digital oracle. Tools like ChatGPT can, for example, generate attack scripts, allowing novice hackers to pull off more complex and sophisticated operations.Simplifying reconnaissance and scanning
Language models also affect how hackers gather information about their targets. With the advent of tools like ChatRTX, they can now communicate with databases in natural language. “Hey, database, where are the juiciest assets?”Agent-based systems like AutoGPT are good for finding potential targets on social media, while the Red Reaper project , inspired by leaks about the capabilities of the Chinese hacker group ISOON, uses a cocktail of NLP, graph analysis, and GenAI to squeeze the most information out of stolen emails.
Every day, such tools become more numerous. For example, researchers have managed to adapt LLM to help users launch hacker utilities. “Maybe you’ll try this tricky parameter?” the AI whispers, offering context-sensitive command line arguments . And for dessert, there are interpreters of scan results, such as BurpGPT.
Assistance in vulnerability detection and analysis
LLM's ability to explain complex things in simple terms also came in handy for hackers. The neural network can break down the workings of publicly known vulnerabilities and immediately find them in the code. In several cases, LLM outperformed specialized SAST tools in this task.
Results of comparison of LLM and specialized code analyzers (image source)
Language models can interpret the output of vulnerability scanners, identifying patterns and signatures associated with false positives. Agent systems based on them evaluate and prioritize vulnerabilities based on the likelihood of their successful exploitation.
Social engineering based on artificial intelligence
If a neural network can write poetry in the style of Pushkin, then it can imitate an FSB officer or pretend to be a neighbor who wants to borrow a couple of bitcoins.LLMs make it easy to create personalized phishing texts. Professional experience, hobbies, favorite beer brand — now you don’t have to be Sherlock Holmes to collect, aggregate, and use the victim’s personal information. Instead of sending out similar “chain letters,” attackers can launch a pack of virtual social engineers, each of whom will select an individual approach to the victim. And if you add generative photos and synthesized voice to this, you get a real challenge to critical thinking. These technologies are so often used in Pig Butchering scams that the United Nations Office on Drugs and Crime (UNODC) published a report with entire sections devoted to LLMs and deepfakes.
Remember the recent commotion with the fake photo of the explosion at the Pentagon? While people were scratching their heads trying to figure out what was going on, stock bots were selling off. So thanks to one picture, the stock market was slightly shaken, and someone may have made a good profit.
Working with malicious code
Want to port your favorite trojan from Python to Rust? No problem! LLM will do half the work for you, and even add comments to the code. Thanks to LLM, obfuscation and modification of malware has become faster, but the most interesting thing is generating malicious code "from scratch". ChatGPT, despite all its moral principles, can be persuaded to write a virus or exploit. This is proven, for example, by Aaron Mulgrew's article, mysteriously disappeared from the Forcepoint blog, but preserved in the web archive.Also at HOPE 2024, Erica Burgess described (starting at 3:56 in the recording) how AI can now create zero-day exploits. Or imagine a polymorphic payload that changes faster than the boss’s opinion on a deadline, extracting data via a Microsoft Teams webhook. That’s a quick rundown of BlackMamba, a tool that uses LLM to generate keyloggers.
Exploitation of vulnerabilities
Researchers are finding ways to use LLM directly for attacks. Austrians have taught neural networks to get into vulnerable virtual machines via SSH, for example, to increase privileges using incorrect settings in the sudoers file. Since last year, an evil version of TensorFlow has been lying on GitHub, designed to combine and compose exploits from different sources and platforms.Agent systems based on LLM demonstrated high capabilities for autonomous use of identified vulnerabilities, including SQL injections and XSS attacks. The creators of PentestGPT even claim that their development copes with tasks of easy and medium complexity on the HackTheBox platform and entered the top 1% of hackers on the platform.
LLMs can exploit one-day vulnerabilities with an 87% success rate when given information in the CVE database. And a follow-up to this paper suggests that teams of AI agents can autonomously exploit zero-day vulnerabilities. Another paper describes how LLMs implement sophisticated privilege escalation strategies in Linux environments.
From script kiddie to super spy: democratizing hacking
This flurry of publications may cause a slight panic, but let's understand the situation. Yes, hackers now have a new toy. LLM is like a multi-tool for them - it can write code, compose a phishing letter, but it can't brew coffee (for now). As a result, the scale, speed, and effectiveness of attacks are growing, but there has been no revolution.Remember the good old malware builders and crypters? It's pretty much the same story, only with a trendy AI sauce. Professional malware developers have been spoiling customers with tools for creating their own malware strains for a long time now.
Using Malicious LLM (Image Source)
Thanks to LLM, the variety of malware will probably grow, but it is unlikely that we will see anything fundamentally new. Perhaps viruses will start writing poetry or telling jokes before encrypting the hard drive.
LLMs still need human skills and experience to create “worthy” malware. Neural networks have problems with logic , they need detailed instructions, like a first-grader in an IT class. “Don’t use this method of process injection, all antiviruses detect it.” “Try Google Drive for data theft, it is blocked so rarely that it’s embarrassing.” And most importantly, to train a specialized evil LLM, you need high-quality data. This is what causes the main difficulties in scaling social engineering and malware creation.
Thus, those who will benefit most from using AI in complex cyberattacks over the next year will have the resources and experience, as well as access to quality data. The intelligence services probably have an advantage here. They may have large enough malware repositories to unlock the potential of LLM in cyberattacks. But we shouldn’t write off seasoned cybercriminals either. They will also have something to further train open source models on. The darknet has been discussing the refinement and customization of LLM for use in cyberattacks for the second year already. Probably, sooner or later, the bad guys will make some progress.
Even the partial list of malicious LLMs is quite long. Some of them are pre-trained, others have had restrictions removed using special prompts.
But, oddly enough, the first to feel the benefits of large language models will be the greenest cybercriminals. In fact, they are already actively using them as an interactive reference. Publicly available LLMs largely cover their needs, especially in simple operations like spear phishing.
SOC on Steroids: LLM as Security Assistant
White hat hackers have many tricks in their arsenal that they've learned from bad guys, so all the benefits of LLMs described here apply to offensive cybersecurity as well. Red teams will also see modest gains in social engineering, reconnaissance, and exfiltration capabilities in the near future, but LLMs can also strengthen internal security operations centers (SOCs).Blue Team has long been one step ahead here. Back when dinosaurs roamed the Earth (okay, in the 90s), they started implementing machine learning in anomaly detection systems (ADS) and intrusion detection systems (IDS). Then in antiviruses and spam filters, but there were still complex unsolved problems in incident response.
Providing context on an incident
So, when a vulnerability with a high CVSS score appears, security researchers don’t immediately sound the alarm. First, they ask a series of questions:- Is it possible to actually exploit this vulnerability?
- If so, who can do it? A skilled hacker or a cunning insider? Or maybe anyone who knows how to google?
- Are there any trump cards against this scourge? Maybe we'll tweak the WAF and that'll be it?
- How difficult is it to completely patch a hole?
Depending on the context, even a serious vulnerability can be given a low priority. But a seemingly average one can easily become a queen if, by eliminating it, you can destroy an entire chain of attacks. But in practice, finding out the details of each alert takes a lot of time.
Almost every vulnerability alert is a quest, despite all the progress in incident response systems. But imagine if the necessary context arrived faster than you could say, “We have an alert, there may be a crime!” LLMs can help with this: extracting, aggregating, and presenting data in a convenient format.
Google security researchers are already writing incident reports using LLM. In their experiment, the model reduced the time it took to write incident summaries by 51%, while maintaining human-level quality.
The time left after incident analysis is spent by a typical security specialist on reporting and “demonstrating results.” Proving that we are not just eating our bread for nothing takes up to 20% of the working time. This epic affects everyone: from the average employee explaining why “qwerty123” is not a password to the security director convincing the board of directors that a new vulnerability is not a reason to dump the company’s shares.
It would seem that answering the question “How secure is our main server?” should be simple, but reporting requires collecting data from a bunch of unrelated systems with mysterious acronyms: CSPM, DSPM, ASPM, IAM…
This is another area where LLMs can be very useful. Imagine coming to work and your report is ready, your presentation is written, and your documentation has updated itself while you were sleeping. A hacker AI will write malware, and an AI on the security side will write a report about it – isn’t that a utopia?
Bug fixes
Jason Clinton, Anthropic's chief information security officer, was reportedly thrilled to see a model reason about a vulnerability that didn't exist when it was trained. It's a sign that LLMs can identify potential vulnerabilities and suggest patches to fix them.Eli Burstein, the technical lead for cybersecurity and AI research at Google DeepMind, decided to test this. His team selected a bunch of vulnerabilities from Google’s codebase and asked the AI to play tech support. These patches were then sent to human programmers for review. Last year’s best model was able to fix 15% of bugs in plain C++ with input sanitization vulnerabilities. The problem was that it needed to not only fix the vulnerability, but also preserve the functionality of the program.
It doesn't sound very impressive, but it's more than would have been fixed without its help. Eli says that increasing the figure to 80% is quite possible. But to make the technology practical, it will require 90-95% of correctly fixed vulnerabilities and a fundamentally new language model. In general, this idea has prospects. Who knows, maybe soon our IDEs will not only highlight syntax, but also tease for crookedly written code.
Cyber apocalypse cancelled?
The scale of cyber attacks will increase as the number of hackers opens up opportunities for LLM. Red and Blue teams will have to keep their finger on the pulse, but security teams can also use the new opportunities to their advantage.It seems that in the end the situation will resemble an endless game of ping-pong between robots. The advantage will shift from one side to the other, but in the end, the “good” and “bad” AI will balance each other out. The ones who will suffer most are those who do not take advantage of the new technologies.
I would like to end on a moderately positive note, but by relying more and more on neural networks, we will face risks that we have not yet realized. No, this is not a machine uprising, I promised to do without futurology. Look, there are already two significant problems:
- The quality of the generated code. For example, a study of 800 programmers showed that specialists using Copilot make 41% more mistakes. In addition, the resulting code is also incomprehensible, it is more difficult to fix.
- Privacy. So, Samsung banned its employees from communicating with ChatGPT . Why? Someone decided that artificial intelligence would evaluate the source code of a proprietary program to fix bugs. It didn't, but it remembered.
Of course, if you ask ChatGPT about Samsung's source code, the neural network won't answer, but this data will most likely end up in the training sample. And the guys from Google DeepMind have already proven that fragments of the original dataset can be extracted from ChatGPT. What's more, ChatGPT simply accidentally revealed the headers of someone else's chat history to other users.
These problems can be mitigated with clear corporate policies for the use of LLM and private local neural network models within the company's framework, but what to do with attacks on the neural network models themselves?
British scientists (no, really, British) from the Department of Science, Innovation and Technology have dug through a mountain of scientific papers and found 72 risk assessments related to AI. They have sorted out a whole zoo of vulnerabilities, including new ones specific only to AI. They are laid out by life cycle stages with an assessment of potential impact.
MITRE Atlas
Or here's another, more famous classification - MITRE Atlas (Adversarial Threat Landscape for Artificial-Intelligence Systems). It relies on tactics familiar to security professionals, for example, from web vulnerabilities, but contains many unique risks: generating "poison" data and introducing it into sets for training the model, extracting the dataset through multiple requests to the API of the machine learning model, selecting adversarial examples and who knows what else.
Here we are, on the threshold of the digital frontier, with an endless attack surface ahead. The adventure is just beginning, and it won't be boring. Tighten your stirrups and adjust your hats, developers and security folks - we're in for a thrilling journey into the unknown!
Source
