Key facts.
- NIST AI RMF 1.0 gives a Govern-Map-Measure-Manage structure for AI risk; the 2026 AI Agent Standards Initiative adds identity, security, and interoperability for agents. source
- OWASP's Top 10 for Agentic Applications (2025) names goal hijack, tool misuse, identity and privilege abuse, and memory poisoning, the agent-specific risks a vendor must defend against. source
- ISO/IEC 42001:2023 specifies certifiable requirements for an AI Management System (AIMS), covering risk management, transparency, security, and continual improvement. source
- SOC 2 Type II covers security, availability, confidentiality, and privacy, a strong data-handling baseline; read the exceptions and confirm AI-relevant paths like inference and logging are in scope. source
What separates evidence from a pitch?
A vendor pitch shows you an architecture diagram and a clean demo. An evaluation asks for artifacts. For permission scoping, ask how the agent is constrained per tool and whether credentials are short-lived and revocable, then ask to see it enforced. For audit logging, ask for a sample trace that captures the reasoning, tool calls, arguments, and instruction source, exportable to your SIEM. For injection defenses, ask for red-team results against indirect injection through retrieved content, not just direct prompts. The difference between a vendor that has done this work and one that hasn't shows up the moment you ask for evidence instead of a slide.
Named frameworks make the bar objective. Instead of "is this secure," you ask: "Show me how you address OWASP's tool-misuse and memory-poisoning entries, where you sit against NIST's Manage function, and whether you hold or are pursuing ISO/IEC 42001." A vendor who can answer in those terms has built for production. One who can't is selling a demo.
What does a production-ready answer look like?
Concrete and evidenced. Tiered, revocable permissions with no standing high privilege. Tamper-evident logs with reasoning traces and incident-reconstruction detail. Instruction-boundary enforcement plus independent red-teaming with shared results. A distinct non-human identity integrated with your IAM. A clean SOC 2 Type II with AI paths in scope. Clear data-isolation and no-training-on-your-data terms. Contractual SLAs for breach notification, audit rights, and remediation. Every item verifiable before you sign, not after an incident.
| Evaluation area | Ask for evidence of | Framework anchor |
|---|---|---|
| Permission scoping | Least privilege, revocable, blast-radius limited | OWASP Agentic (privilege abuse) |
| Audit logging | Tamper-evident traces, SIEM export | NIST Manage; ISO 42001 |
| Injection defense | Indirect-injection red-team results | OWASP (tool misuse, goal hijack) |
| Identity | Distinct agent identity, IAM integration | NIST AI Agent Standards (2026) |
| Compliance | SOC 2 Type II, ISO/IEC 42001:2023 | SOC 2; ISO 42001 |
Score a vendor agent on scoping, audit logging, and injection defenses against NIST AI RMF and OWASP, and a frontier model doesn't settle it, since on AgentDojo GPT-4o falls to a targeted attack 48% of the time. (source)
This is also the bar the Pattern Intelligence Layer is built to meet. Reliability and security at the pattern level mean scoping, tracing, injection handling, and identity are enforced around the agent on every run and inspectable as evidence, not promises. When you evaluate a vendor against named frameworks and ask for the artifacts, you're asking whether security lives in the pattern or in the pitch. The first survives production and a model swap. The second survives the demo.
Frequently asked questions
What's the single most revealing question?
Ask to see a real audit trace from a production run, including reasoning and tool calls. A vendor who can produce one has built for incident response. One who can't hasn't.
Is SOC 2 enough on its own?
No. SOC 2 Type II is a strong baseline for data handling but not AI-specific. Pair it with ISO/IEC 42001 and the OWASP agentic risks to cover agent behavior, not just infrastructure.
Why test consistency, not just a demo?
Agents are non-deterministic. A frontier agent can pass once and fail on repeat. Evaluate under adversarial and repeated conditions, that's where production actually lives.