Resource

AI Agent Vendor Evaluation Checklist

For security, GRC, and architecture teams evaluating any AI agent platform before a pilot in a regulated environment.

By Sparcle. Distribute freely.

← Back to download page

How to use this

Fifteen yes-no questions. Take them into your next AI agent vendor evaluation and ask the vendor to answer each one with yes or no, plus one sentence of explanation. A "no" on any single question is a finding worth raising in design review. A "no" on any of the audit or privacy questions is hard to recover from at scale.

Vendor-neutral. Works for buy-vs-build decisions as well as vendor short-list reviews. Use it on yourself if you are building.

1. Identity

Who or what acted, and is that identity stable enough to audit.

  1. Does every event carry a stable run_id that groups a full agent invocation end to end?
  2. Does every event carry a typed actor (user, agent, automation, connector, peer), not a freeform string?

2. Policy

What was permitted at the moment of action, and on what evidence.

  1. Are policy decisions logged with the inputs the policy saw, not just the verdict?
  2. Are delegation grants (agent A acting on behalf of agent B) logged as first-class events, with explicit expiry?

3. Audit

Can a third party reproduce the record and prove nothing has been tampered with.

  1. Is each event linked to its predecessor by a parent_event_hash over a documented canonical encoding?
  2. Is the canonical encoding documented and protected by a ratchet test that fails CI on accidental change?
  3. Can a replay verifier be run by someone outside the vendor, against raw events, with no proprietary dependency?
  4. Are events bound to downstream vendor audit logs via external_audit_refs (Gmail msg id, Slack ts, Drive revision)?
  5. Is there protection against insider tampering by an actor with primary-DB access (Merkle root published out of band, signed by a key the DBA cannot impersonate)?

4. Privacy

What the LLM actually saw, and what survives a Right-to-Erasure request.

  1. Is PII masking applied at the LLM boundary, and are real values restored before tool dispatch?
  2. Are PII values stored under per-subject keys that can be destroyed independently of the chain?
  3. Is there a documented Right-to-Erasure procedure that preserves chain integrity (crypto-shred receipt or equivalent)?

5. Reversibility

Refusals, retries, and the difference between a log and a system of record.

  1. Are refusals and refused alternatives recorded, not silently discarded?
  2. Are retries recorded as their own events, with their own outcomes, linked to the original plan?
  3. Is the chain a system of record for compliance, separate from the application telemetry pipeline (OpenTelemetry, Datadog, Splunk are sinks, not the chain)?

Next step

If you are running an evaluation now and want a second pair of eyes from the team that wrote this, reply to the email you received with the checklist link, or write to [email protected]. We're happy to read vendor responses and tell you which questions are being answered honestly.

If you'd like to see how a deployment that clears all fifteen questions actually works in practice, the Sparcle Design Partner Program has a few slots open: sparcle.app/design-partners.