The introduction of OpenAI’s ChatGPT was a tipping point in the field of artificial intelligence (AI) and, perhaps, a watershed moment in history. This platform’s myriad capabilities have captivated consumers and fueled engineers to leverage the features of generative AI to build applications with many use cases for businesses and consumers alike despite the lack of strong policies and effective risk management.1
As AI-based systems are integrated with routine business operations, some key questions need to be addressed. Do the applications provide correct results that are both accurate and fair? Will confidential data be protected when using these applications? According to a recent survey of IT professionals, 71 percent of the respondents believe that generative AI is likely to “introduce new security risks to data.”2
AI-based systems extend the attack surface and provide a new threat vector for hackers to exploit. Attackers can manipulate these systems and alter their behavior to produce malicious results. These threats are fundamentally different from traditional cyberattacks, as the underlying technologies, including algorithms, learning models, and training data sets, are targeted.3 AI security is still evolving, including regarding identifying vulnerabilities and implementing appropriate controls and countermeasures. It is important to formulate and evaluate mitigation plans for AI-related attacks as part of the risk management process as AI-based systems become more common.
Prompt Injection
The Open Worldwide Application Security Project (OWASP) has identified prompt injection as the greatest vulnerability for large language model (LLM)-based applications.4 Prompt injection refers to the manipulation of an application’s input parameters to hijack the output and provide desired results. Since this type of attack was discovered, it has gained the attention of application developers, especially those using LLMs, due to its serious repercussions. Researchers have shown that if employed correctly, prompt injection attacks can lead to leakage or exfiltration of sensitive data and allow for remote control or even manipulation of LLMs.5
Prompt injection operates similarly to structured query language (SQL) injection and cross-side scripting in web applications. An input prompt can be crafted with a specific series of words to tell the machine what to do and how to behave. Research has shown that attackers employ this technique in either direct mode or indirect mode (without a direct interface by injecting prompts within data that is likely to be retrieved) and use it to execute malicious code, extract sensitive data, or confuse the model.6
Currently, engineers are using mechanisms such as input validation, content filtering, and output encountering to thwart prompt injection attacks. Better construction of prompts, especially during the design phase, and enhanced testing can improve the application’s security. This is an evolving topic, and it is unclear whether these defenses are sufficient to guard against this serious threat.
The most important way to guard against these attacks is to implement strong data security controls, including access, encryption, and database hardening.
Data Poisoning
Data poisoning is a well-known vulnerability, and, as the name suggests, it involves intentionally polluting or compromising a training data set with malicious information. The quality of the training data influences the performance of the learning model and the accuracy of the application’s output. Attackers can tamper with training data in several ways, causing the machine to produce erroneous results. In the case of machine learning algorithms that use classifiers, the attacker injects bad data, causing the algorithm to return incorrect classifications, or the attacker gains access to the data set and tries to corrupt it. For example, attempts to tamper with the Gmail spam filter by corrupting its training data emphasize the extent of this threat.7
The most important way to guard against these attacks is to implement strong data security controls, including access, encryption, and database hardening. The database schema and indexes must be designed to reduce the attack surface. In addition, engineers must build the training data sets by appropriately scanning, filtering, and sampling the right data elements as well as periodically ensuring that the quality of the data set is maintained.
Machine Learning Supply Chain Attack
Thanks to the infamous SolarWinds8 and log4j9 incidents, there is increased awareness of software supply chain vulnerabilities, leading enterprises to enhance their application security scanning processes. The same concept pertains to AI applications, as learning models are often built using multiple open-source and third-party frameworks. Attackers can compromise the external frameworks and code libraries used to build models and create Trojan horses or back doors. Recently, the popular Python-based open-source machine learning framework PyTorch disclosed that one of its packages was affected by a supply chain attack, highlighting this type of threat.10 This specific attack type, referred to as dependency confusion in the software supply chain parlance, is mitigated by pulling packages only from verified or trusted sources or from private repositories (that may mirror other public repositories) and downloading packages via internal proxies only.11
The use of pretrained models also poses a significant risk, as the model’s integrity can be altered. Model scanning is a practice that is gaining traction as a tool for developers to identify vulnerabilities in third-party models or libraries and enhance the security of their LLMs. Open-source model scanning tools could be leveraged while developing AI applications that include external building blocks. Furthermore, it is suggested that AI applications, including LLMs, should be analyzed using software composition analysis tools (SCA), similar to other software products. This analysis should be part of the development process and should result in the creation of a detailed software bill of materials (SBOM).12 This would be an effective method to track the use of third-party models or libraries used in the development of applications and, thereby, manage any security vulnerabilities.
Inference
In inference attacks, the attacker tries to extract information about the data used to train the machine learning model. For example, the attacker might try to determine whether a specific data set was used as part of the training data (membership inference) or try to reconstruct the training data set itself (model inference). Such attacks have a significant impact from a privacy perspective, as they lead to the leakage of sensitive information from the training database. Possible mitigation strategies include changing the way the model learns from the training data by using a mathematical method called regularization or employing model obfuscation, which involves adding randomness to the model output to hide sensitive information and make it difficult for the attacker to identify confidential information used as part of the training data set.
Evasion
Another type of adversarial attack is model evasion, whereby a legitimate input is slightly modified to confuse the model and induce mistakes. For example, researchers have shown that making subtle changes to street signs by sticking small strips of material on them can confuse autonomous cars.13 In these cases, the learning model needs to be fine-tuned to be able to recognize the inputs correctly and produce the appropriate outputs. Continuous training and testing of models are mandatory to ensure accurate and repeatable outputs.
Conclusion
Research into the potential vulnerabilities of AI-based applications and their underlying training systems and machine learning algorithms is being actively pursued and is consistently revealing fascinating discoveries. However, as enterprises ramp up the development of AI-based applications, it is important to build a strong set of controls to mitigate the threat of attack. Further, as enterprises evaluate various AI-based products, it is critical to assess their ability to handle any attack from an adversary. This is an evolving space that must be constantly monitored as the practical applications of AI continue to grow.
Endnotes
1 ISACA, “Generative AI: Risks and Opportunities,” http://0ir7.51jiyangshi.com/resources/infographics/generative-ai-2023-an-isaca-pulse-poll-infographic
2 Jackson, T.; “Exploring the Security Risks of Generative AI,” Forbes, 19 April 2023, http://www.forbes.com/sites/forbestechcouncil/2023/04/19/exploring-the-security-risks-of-generative-ai/?sh=74afcc853594
3 Comiter, M.; “Attacking Artificial Intelligence: AI’s Security Vulnerability and What Policymakers Can Do About It,” Belfer Center for Science and International Affairs, Harvard Kennedy School, August 2019, http://www.belfercenter.org/publication/AttackingAI
4 OWASP, “OWASP Top 10 for Large Language Model Applications,” http://owasp.org/www-project-top-10-for-large-language-model-applications/
5 Greshake, K.; “llm-security,” GitHub, http://github.com/greshake/llm-security
6 Greshake, K.; S. Abdelnabi; S. Mishra; C. Endres; T. Holz; M. Fritz; “Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications With Indirect Prompt Injection,” 5 May 2023, http://arxiv.org/pdf/2302.12173.pdf
7 Bursztein, E.; “Attacks Against Machine Learning— An Overview,” Elie, May 2018, http://elie.net/blog/ai/attacks-against-machine-learning-an-overview/
8 Center for Internet Security, “The SolarWinds Cyber-Attack: What You Need to Know,” 15 March 2021, http://www.cisecurity.org/solarwinds
9 National Cyber Security Centre, “Log4j Vulnerability—What Everyone Needs to Know,” http://www.ncsc.gov.uk/information/log4j-vulnerability-what-everyone-needs-to-know
10 PyTorch Team, “Compromised PyTorch-Nightly Dependency Chain Between December 25th and December 30th 2022,” PyTorch, 31 December 2022, http://pytorch.org/blog/compromised-nightly-dependency/
11 Open Worldwide Application Security Project, “Dependency Chain Abuse,” http://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-03-Dependency-Chain-Abuse
12 Fruhlinger, J.; “What Is an SBOM? Software Bill of Materials Explained,” CSO, 19 July 2022, http://www.csoonline.com/article/573185/what-is-an-sbom-software-bill-of-materials-explained.html
13 Ackerman, E.; “Slight Street Sign Modifications Can Completely Fool Machine Learning Algorithms,” IEEE Spectrum, 4 August 2017, http://spectrum.ieee.org/slight-street-sign-modifications-can-fool-machine-learning-algorithms
Varun Prasad, CISA, CISM, CCSK, CIPM, PMP
Is senior manager of third-party attestation at BDO USA P.C.
