AI penetration testing is not a prompt-writing contest. A useful engagement answers three questions: what can an attacker make the system do, what can they take, and would anyone notice? That requires a scope, a threat model, a reproducible corpus and a report that engineering can act on. This article describes a framework you can run against an LLM application, an agent, or an MCP deployment.

It complements the methodology in AI Red Teaming LLM Security and the commercial service described on the AI pentesting page.

Phase 0 — Scope and rules of engagement

AI systems blur the boundary between "application" and "content", so scope must be explicit:

  • In scope: models and versions, endpoints, retrieval sources, tools and their scopes, MCP servers, memory stores, admin consoles and logging pipelines.
  • Out of scope: the provider's underlying model weights, third-party SaaS internals, and production data beyond agreed test tenants.
  • Constraints: rate limits, cost caps, prohibited attack types (e.g. real phishing), data handling for captured prompts, and an emergency stop contact.
  • Success criteria: what counts as a finding — a policy bypass, a data exposure, an unauthorised tool call, or a reliable denial of service.

Phase 1 — Threat model

Map the system before attacking it. Two public catalogues give you coverage without reinventing taxonomy:

  • OWASP Top 10 for LLM Applications — prompt injection, insecure output handling, training-data poisoning, model denial of service, supply chain, sensitive information disclosure, insecure plugin design, excessive agency, overreliance and model theft.
  • MITRE ATLAS — adversary tactics and techniques specific to AI systems, useful for mapping findings to detection opportunities.

Then add the threats specific to your architecture: who can write to the knowledge base, which tools are reachable, what the agent can do on its own, and which identities it can assume. The AI agent security pillar is a good checklist for the agent layer.

Phase 2 — Build the test corpus

A penetration test is only repeatable if its inputs are versioned. Structure the corpus by objective, not by payload string:

  • Instruction override — direct injection, role-play, delimiter confusion, multilingual and encoded variants.
  • Indirect injection — payloads in documents, web pages, emails, tickets and tool results, including hidden text and zero-width characters.
  • System-prompt extraction — refusal loops, translation, summarisation and context-switch tricks (see prompt leaking).
  • Data exfiltration — PII, secrets, cross-tenant records, and outbound requests to attacker-controlled domains.
  • Tool abuse — unauthorised calls, argument manipulation, chained tools, and irreversible actions without approval.
  • Denial of service and cost — context flooding, recursive tool loops and oversized outputs.

Record, for each case: input, model and policy version, expected behaviour, observed behaviour, and a severity. Without versioning, a retest cannot distinguish a fix from a model upgrade.

Phase 3 — Application and model testing

Test the system as deployed, not the model in isolation. The same model behind a gateway with input filtering, output filtering and least-privilege tools behaves very differently from a raw API.

  • Baseline normal behaviour first, so you can recognise an anomaly.
  • Test the filters, not just the model: encoding, token splitting, homoglyphs, translation and multi-turn buildup.
  • Test output handling: does the application render model output as HTML, execute it, or pass it to another system?
  • Test retrieval: can you get a document indexed, and does its content influence the next action?
  • Measure reliability: a payload that works once in fifty attempts is a different finding from one that works every time. Report the rate.

Phase 4 — Agent and MCP testing

Agents add action to influence. The highest-value tests are about authority, not wording:

  • Privilege escalation. Can a low-privilege user cause a tool call that requires a higher privilege? Can the agent assume another identity?
  • Scope escape. Can arguments reference records, tenants or domains outside the caller's scope?
  • Confused deputy. Can the agent be made to act on behalf of a different user, or use its own broad credentials for an attacker's request?
  • Tool poisoning and rug pulls. For MCP: does the client pin the tool manifest? Does a description change take effect without approval? See MCP security mechanisms and layers.
  • Memory poisoning. Can a persisted fact change future behaviour across sessions or users?
  • Approval bypass. Can the model fabricate or skip a human-approval step for an irreversible action?
  • Secret exposure. Are credentials reachable from context, tool output or logs?

Least-privilege design is the control being tested; the implementation is described in Least Privilege for AI Agents.

Phase 5 — Automate what repeats

Manual testing finds novel issues; automation prevents regressions. Put the corpus in CI and run it on every model, prompt or policy change. A practical pipeline:

  1. Deploy the candidate version to an isolated environment with synthetic data.
  2. Run the corpus with fixed seeds and a cost cap; store results as JSON.
  3. Compare against the previous run; fail on new high-severity successes.
  4. Publish a dashboard of pass rate by category and model version.

Open-source building blocks — ReskSafety for the firewall, resk-llm for detection primitives and ReskPoints for telemetry — can host the target and the evidence. Open-source LLM security libraries lists the wider stack.

Phase 6 — Report and remediate

A finding is only useful with a reproduction and a fix. Use a consistent template:

  • Title and severity — CVSS v4 where possible, adjusted for AI-specific factors such as reliability and blast radius.
  • Preconditions — required access, data or user interaction.
  • Reproduction — exact input, model version, policy version, observed output.
  • Impact — what the attacker gains, in business terms.
  • Evidence — logs and traces correlated by request ID.
  • Remediation — a control, not a wording change. "Add a sentence to the system prompt" is not a fix.
  • Retest criteria — the specific case that must now fail to succeed.

Rank fixes by whether they remove a class of finding. Least-privilege tool authorization and output filtering usually eliminate more findings than any prompt change, while remaining effective across model upgrades.

FAQ

What is an AI penetration testing framework? A repeatable method covering scope, threat model, a versioned test corpus, application and agent testing, automation and a remediation-focused report — mapped to OWASP LLM Top 10 and MITRE ATLAS.

How is AI pentesting different from LLM red teaming? Red teaming explores adversarial behaviour broadly, often to improve safety. Penetration testing is scoped, evidence-driven and aimed at a specific deployed system, with severity and retest criteria.

Which tools are used for LLM red teaming? A versioned prompt corpus, an evaluation harness in CI, telemetry to correlate attempts, and open-source detection libraries such as resk-llm. The target itself should be instrumented with a firewall and audit logs.

How often should you pentest an AI system? Before launch, after any model or policy change, and at least annually. Because behaviour can change with a provider-side model update, continuous regression testing matters more than a single annual engagement.

What is the most common critical finding? Excessive agency: an agent with broad tools and inherited credentials, where a single injection or scope error leads to a real action — a send, a delete or a data transfer.

Next step

Start with the agent and MCP phases even if your application looks like a chatbot: that is where authority lives. If you need an external assessment, the RESK AI pentesting service runs this framework against production-like environments.

For the defensive counterpart, read LLM Security Monitoring & Observability.