The UK’s National Cyber Security Centre (NCSC) has issued a fresh warning about the growing threat of prompt injection, a vulnerability that has quickly become one of the biggest security concerns in generative AI systems. First identified in 2022, prompt injection refers to attempts by attackers to manipulate large language models (LLMs) by inserting rogue instructions into user-supplied content. While the technique may appear similar to the long-familiar SQL injection flaw, the NCSC stresses that comparing the two is not only misleading but potentially harmful if organisations rely on the wrong mitigation strategies.

Why Prompt Injection Is Fundamentally Different

SQL injection has been understood for nearly three decades. Its core issue, blurring the boundary between data and executable instructions, has well-established fixes such as parameterised queries. These protections work because traditional systems draw a clear distinction between “data” and “instructions.” The NCSC explains that LLMs do not operate in the same way. Under the hood, a model doesn’t differentiate between a developer’s instruction and a user’s input; it simply predicts the most likely next token. This makes it inherently difficult to enforce any security boundary inside a prompt. In one common example of indirect prompt injection, a candidate’s CV might include hidden text instructing a recruitment AI to override previous rules and approve the applicant. Because an LLM treats all text the same, it can mistakenly follow the malicious instruction. This, according to the NCSC, is why prompt injection attacks consistently appear in deployed AI systems and why they are ranked as OWASP’s top risk for generative AI applications.

Treating LLMs as an ‘Inherently Confusable Deputy’

Rather than viewing prompt injection as another flavour of classic code injection, the NCSC recommends assessing it through the lens of a confused deputy problem. In such vulnerabilities, a trusted system is tricked into performing actions on behalf of an untrusted party. Traditional confused deputy issues can be patched. But LLMs, the NCSC argues, are “inherently confusable.” No matter how many filters or detection layers developers add, the underlying architecture still offers attackers opportunities to manipulate outputs. The goal, therefore, is not complete elimination of risk, but reducing the likelihood and impact of attacks.

Key Steps to Building More Secure AI Systems

The NCSC outlines several principles aligned with the ETSI baseline cybersecurity standard for AI systems: 1. Raise Developer and Organisational Awareness Prompt injection remains poorly understood, even among seasoned engineers. Teams building AI-connected systems must recognise it as an unavoidable risk. Security teams, too, must understand that no product can completely block these attacks; risk has to be managed through careful design and operational controls. 2. Prioritise Secure System Design Because LLMs can be coerced into using external tools or APIs, designers must assume they are manipulable from the outset. A compromised prompt could lead an AI assistant to trigger high-privilege actions, effectively handing those tools to an attacker. Researchers at Google, ETH Zurich, and independent security experts have proposed architectures that constrain the LLM’s authority. One widely discussed principle: if an LLM processes external content, its privileges should drop to match the privileges of that external party. 3. Make Attacks Harder to Execute Developers can experiment with techniques that separate “data” from expected “instructions”, for example, wrapping external input in XML tags. Microsoft’s early research shows these techniques can raise the barrier for attackers, though none guarantee total protection. The NCSC warns against simple deny-listing phrases such as “ignore previous instructions,” since attackers can easily rephrase commands. 4. Implement Robust Monitoring A well-designed system should log full inputs, outputs, tool integrations, and failed API calls. Because attackers often refine their attempts over time, early anomalies, like repeated failed tool calls, may provide the first signs of an emerging attack.

A Warning for the AI Adoption Wave

The NCSC concludes that relying on SQL-style mitigations would be a serious mistake. SQL injection saw its peak in the early 2010s after widespread adoption of database-driven applications. It wasn’t until years of breaches and data leaks that secure defaults finally became standard. With generative AI rapidly embedding itself into business workflows, the agency warns that a similar wave of exploitation could occur, unless organisations design systems with prompt injection risks front and center.

A new vulnerability scoring system has just been announced. The initiative, called the AI Vulnerability Scoring System (AIVSS), aims to fill the gaps left by traditional models such as the Common Vulnerability Scoring System (CVSS), which were not designed to handle the complex, non-deterministic nature of modern AI technologies.  AI security expert, author, and adjunct professor Ken Huang introduced the AIVSS framework, emphasizing that while CVSS has long been a cornerstone for assessing software vulnerabilities, it fails to capture the unique threat landscape presented by agentic and autonomous AI systems.  “The CVSS and other regular software vulnerability frameworks are not enough,” Huang explained. “These assume traditional deterministic coding. We need to deal with the non-deterministic nature of Agentic AI.”  Huang serves as co-leader of the AIVSS project working group alongside several prominent figures in cybersecurity and academia, including Zenity Co-Founder and CTO Michael Bargury, Amazon Web Services Application Security Engineer Vineeth Sai Narajala, and Stanford University Information Security Officer Bhavya Gupta.   Together, the group has collaborated under the Open Worldwide Application Security Project (OWASP) to develop a framework that provides a structured and measurable approach to assessing AI-related security threats.  According to Huang, Agentic AI introduces unique challenges because of its partial autonomy. “Autonomy is not itself a vulnerability, but it does elevate risk,” he noted. The AIVSS is designed specifically to quantify those additional risk factors that emerge when AI systems make independent decisions, interact dynamically with tools, or adapt their behavior in ways that traditional software cannot. 

A New Approach to AI Vulnerability Scoring 

The AI Vulnerability Scoring System builds upon the CVSS model, introducing new parameters tailored to the dynamic nature of AI systems. The AIVSS score begins with a base CVSS score and then incorporates an agentic capabilities assessment. This additional layer accounts for autonomy, non-determinism, and tool use, factors that can amplify risk in AI-driven systems. The combined score is then divided by two and multiplied by an environmental context factor to produce a final vulnerability score.  A dedicated portal, available at aivss.owasp.org, provides documentation, structured guides for AI risk assessment, and a scoring tool for practitioners to calculate their own AI vulnerability scores.  Huang highlighted a critical difference between AI systems and traditional software: the fluidity of AI identities. “We cannot assume the identities used at deployment time,” he said. “With agentic AI, you need the identity to be ephemeral and dynamically assigned. If you really want to have autonomy, you have to give it the privileges it needs to finish the task.”  

Top Risks in Agentic AI Systems 

The AIVSS project has also identified the ten most severe core security risks for Agentic AI, though the team has refrained from calling it an official “Top 10” list. The current risks include: 

• Agentic AI Tool Misuse 
• Agent Access Control Violation 
• Agent Cascading Failures 
• Agent Orchestration and Multi-Agent Exploitation 
• Agent Identity Impersonation 
• Agent Memory and Context Manipulation 
• Insecure Agent Critical Systems Interaction 
• Agent Supply Chain and Dependency Attacks 
• Agent Untraceability 
• Agent Goal and Instruction Manipulation 

Each of these risks reflects the interconnected and compositional nature of AI systems. As the draft AIVSS document notes, “Some repetition across entries is intentional. Agentic systems are compositional and interconnected by design. To date, the most common risks such as Tool Misuse, Goal Manipulation, or Access Control Violations, often overlap or reinforce each other in cascading ways.”  Huang provided an example of how this manifests in practice: “For tool misuse, there shouldn’t be a risk in selecting a tool. But in MCP systems, there is tool impersonation, and also insecure tool usage.” 

As businesses increasingly rely on AI chatbots, securing conversational AI is now mission-critical. Learn about common chatbot vulnerabilities, AI risk management strategies, and best practices — from data encryption and authentication to model protection — to safeguard user trust, privacy, and compliance in the digital era.

The post When Chatbots Go Rogue: Securing Conversational AI in Cyber Defense  appeared first on Security Boulevard.