Resource
AI Agent Vendor Evaluation Checklist
For security, GRC, and architecture teams evaluating any AI agent platform before a pilot in a regulated environment.
How to use this
Fifteen yes-no questions. Take them into your next AI agent vendor evaluation and ask the vendor to answer each one with yes or no, plus one sentence of explanation. A "no" on any single question is a finding worth raising in design review. A "no" on any of the audit or privacy questions is hard to recover from at scale.
Vendor-neutral. Works for buy-vs-build decisions as well as vendor short-list reviews. Use it on yourself if you are building.
1. Identity
Who or what acted, and is that identity stable enough to audit.
| Identity | YES / NO | |
|---|---|---|
| 1 | Does every event carry a stable run_id that groups a full agent invocation end to end? | ☐ |
| 2 | Does every event carry a typed actor (user, agent, automation, connector, peer), not a freeform string? | ☐ |
2. Policy
What was permitted at the moment of action, and on what evidence.
| Policy | YES / NO | |
|---|---|---|
| 3 | Are policy decisions logged with the inputs the policy saw, not just the verdict? | ☐ |
| 4 | Are delegation grants (agent A acting on behalf of agent B) logged as first-class events, with explicit expiry? | ☐ |
3. Audit
Can a third party reproduce the record and prove nothing has been tampered with.
| Audit | YES / NO | |
|---|---|---|
| 5 | Is each event linked to its predecessor by a parent_event_hash over a documented canonical encoding? | ☐ |
| 6 | Is the canonical encoding documented and protected by a ratchet test that fails CI on accidental change? | ☐ |
| 7 | Can a replay verifier be run by someone outside the vendor, against raw events, with no proprietary dependency? | ☐ |
| 8 | Are events bound to downstream vendor audit logs via external_audit_refs (Gmail msg id, Slack ts, Drive revision)? | ☐ |
| 9 | Is there protection against insider tampering by an actor with primary-DB access (Merkle root published out of band, signed by a key the DBA cannot impersonate)? | ☐ |
4. Privacy
What the LLM actually saw, and what survives a Right-to-Erasure request.
| Privacy | YES / NO | |
|---|---|---|
| 10 | Is PII masking applied at the LLM boundary, and are real values restored before tool dispatch? | ☐ |
| 11 | Are PII values stored under per-subject keys that can be destroyed independently of the chain? | ☐ |
| 12 | Is there a documented Right-to-Erasure procedure that preserves chain integrity (crypto-shred receipt or equivalent)? | ☐ |
5. Reversibility
Refusals, retries, and the difference between a log and a system of record.
| Reversibility | YES / NO | |
|---|---|---|
| 13 | Are refusals and refused alternatives recorded, not silently discarded? | ☐ |
| 14 | Are retries recorded as their own events, with their own outcomes, linked to the original plan? | ☐ |
| 15 | Is the chain a system of record for compliance, separate from the application telemetry pipeline (OpenTelemetry, Datadog, Splunk are sinks, not the chain)? | ☐ |
Next step
If you are running an evaluation now and want a second pair of eyes from the team that wrote this, reply to the email you received with the checklist link, or write to [email protected]. We're happy to read vendor responses and tell you which questions are being answered honestly.
If you'd like to see how a deployment that clears all fifteen questions actually works in practice, the Sparcle Design Partner Program has a few slots open: sparcle.app/design-partners.