For teams without a security team
13 things to verify before an AI agent touches your real tools.
The questions a security reviewer would ask — written for teams that don't have one. Vendor-neutral: run it against any agent product, including ours. We answer all 12 below, including the two we can't fully pass yet.
The blocker to delegating real work to agents isn't capability — it's that nobody can answer “what happens when it goes wrong?” in a form a careful person accepts. These 13 checks are that form. Each names the failure it catches and what a good answer looks like, for any agent product. Where a check has a hard part vendors tend to skip, the check says so.
- 01
What exactly can the agent touch — and what happens with a tool nobody classified?
“It has access to Slack” is not an inventory. Blast radius is per-action: reading a channel and posting to one are different risks wearing the same integration.
A good answer — from anyone
A per-tool list separating read-only from destructive actions, plus a stated default for anything unclassified. The safe default is deny-or-approval (fail closed). If new tools are allowed until someone remembers to restrict them, every future integration ships a governance hole.
Pantheon's answer · live
Every tool is registered read-only or approval-required at the connector layer. A tool in neither set requires approval by default — forgetting is safe. The same fail-closed default applies to external and MCP-bridged tools.
- 02
What does the approval gate do when its own plumbing fails?
The most dangerous line in any gated system is the catch block around the gate. If a failed queue write falls through to execution, the gate is decorative exactly when infrastructure is at its worst.
A good answer — from anyone
A specific answer per failure: approval-queue write fails → the action blocks. Policy rules can't load → approval required. Ask the vendor to show the code path or test — “that never happens” is not an answer.
Pantheon's answer · live
A failed approval-queue insert returns a blocking error — the action does not run. Unloadable policy context forces approval. We fixed the fail-open version of this bug in 2026 and publish the incident; the fail-closed behaviour is unit-tested.
- 03
Who can edit the action log — and would you know?
An audit log in an ordinary table proves nothing: whoever can write rows can rewrite history. “Everything is logged” is the beginning of the question, not the answer.
A good answer — from anyone
Tamper-evidence with its scope stated honestly: e.g. hash-chained records where any edit breaks verification — plus a straight answer about who can still rewrite (typically: the vendor's own database admins, until records are externally signed or anchored).
Pantheon's answer · live
Every action lands on a per-tenant hash chain, computed in the database so no code path can skip it, with a public verify endpoint. Honest scope: an insider with database write access could recompute the chain — independent signing is in development, and we say so.
- 04
Is “who approved this” written with the action — or joined together later?
Provenance assembled at report time from separate tables is provenance that can drift, break, or be edited independently of the action it explains.
A good answer — from anyone
Approver identity, timestamp, and channel recorded atomically with the action itself, inside whatever tamper-evidence protects the record — so the approval cannot be detached from the thing it approved.
Pantheon's answer · live
The resolution event carries approver, channel, and approval id inside the hash-chained envelope, written in the same code path as the action. There is no later join to get wrong.
- 05
Can the approver see what the agent knew when it asked?
An approval based on a one-line summary is a rubber stamp. The human is being asked to lend judgment — judgment needs the same context the agent had.
A good answer — from anyone
A decision-context snapshot attached to each approval: what data the agent had, from which sources, at what staleness — one click from the approve button, and bound into the record.
Pantheon's answer · live
Approvals carry a context snapshot (“what the agent knew”) viewable before deciding, and the snapshot reference is sealed inside the hash-chained record of the action.
- 06
Is the evidence re-checked between approval and execution — or does consent alone authorize the action?
This is the gap almost every approval system has, and its own audit log is blind to it: an action can be correctly requested, correctly approved, and still be wrong, because the business facts moved after the plan was formed. Approvals sit in queues for hours or days. The invoice gets paid, the PR gets merged, the vendor contract gets cancelled — and the action fires against a world that no longer matches the plan. Every record says the process worked, because procedurally it did.
A good answer — from anyone
A freshness check between the decision and the side effect: re-read the evidence the plan was built on, diff it against current state, and stop the action if something material moved — showing the approver exactly what changed rather than a generic warning. It must be honest about its own basis: if the data backing the comparison is itself stale, the answer is “cannot confirm”, never “nothing changed”. And re-confirming should be recorded as an acknowledged override, so replay can tell “nothing had changed” apart from “a human saw the change and accepted it”.
Pantheon's answer · live
Before an approved action executes, the decision-context snapshot is re-read and diffed against current state. A status change, a newer source timestamp, or a vanished record on evidence material to that action blocks execution and returns the diff; unrelated churn is reported without blocking, so the gate does not train people to click through it. A stale comparison basis yields “cannot confirm” and blocks; a failed check blocks. The verdict — including whether a human acknowledged drift and proceeded — is sealed in the hash-chained record with the action. The comparison is against the synced context graph, not a live re-fetch of source systems.
- 07
Whose credentials does the agent actually hold?
Many setups hand the agent a human's OAuth token — so the agent is that human, everywhere, and a teammate who wants to approve something must share credentials to do it.
A good answer — from anyone
Credentials encrypted at rest and injected only at the moment of use — never sitting in the agent's context. Teammates approve with their own identity, recorded as themselves, without ever holding the owner's tokens: judgment is lent, credentials are not.
Pantheon's answer · live
Connector tokens and BYOM keys are envelope-encrypted (AES-256-GCM) at the schema boundary and decrypted at use; the API never returns them. Teammate approvals resolve in the owner's tenancy while recording the actual approver's identity.
- 08
How is tenant isolation enforced — and can they show you the test?
In a multi-tenant product your agent's memory, approvals, and record share infrastructure with strangers'. A policy is a promise; only a test is a property.
A good answer — from anyone
Isolation enforced on every query by architecture (not per-endpoint discipline), with automated tests that run on every change and block release on regression. Ask to see the test, not the policy PDF.
Pantheon's answer · live
Owner identity is ambient per-request (AsyncLocalStorage) — queries are tenant-scoped by construction, with database RLS as defense in depth. Isolation tests are CI-blocking; that rule is design law #4 and it is public.
- 09
What bounds a runaway agent — including one that floods you with approval requests?
Failure isn't always malice; it's a loop. And a clever attack on a gated system isn't breaking the gate — it's burying the human in requests until every approval becomes a reflex.
A good answer — from anyone
Hard ceilings on tool-call loops, time, and spend, approved up front — plus a cap on pending approvals, because human attention is part of the attack surface. Pending requests should expire rather than queue forever.
Pantheon's answer · live
Tool loops cap at 5 rounds / 30s per round; missions carry human-approved budget caps; external agents hit a 25-pending attention budget (429) — flooding is treated as an attack on the gate. Approvals expire in 7 days as auto-rejections.
- 10
What's the prompt-injection story — containment or filtering?
Agents read email, tickets, and web pages — attacker-writable text. A vendor claiming to filter malicious instructions out of natural language is claiming to have solved an unsolved problem.
A good answer — from anyone
Structural containment: assume the agent can be fooled, and ensure a fooled agent still cannot act unilaterally — injected text can make it want to act, but the dangerous action still hits policy and a human. Filtering is fine as a layer; as the whole answer it's a red flag.
Pantheon's answer · live
We assume injection succeeds. A persuaded agent's dangerous tool call still requires approval — containment is the gate, not a filter. We state plainly: Pantheon doesn't prevent an agent being fooled; it prevents a fooled agent acting alone.
- 11
How fast can you turn it off — and at what granularity?
Revocation that requires a support ticket is not revocation. When something goes wrong you need to shrink blast radius in seconds, without destroying the record of what happened.
A good answer — from anyone
One-click disable per tool, per agent, and for the whole workspace; connector credentials revocable independently; and the audit record survives the shutdown.
Pantheon's answer · live
Connectors disable per project in one click, missions abort mid-run (a running step notices within seconds), API keys revoke instantly, and the record is append-only — turning things off never edits history.
- 12
Does governance cover the agents you already run?
Most teams' riskiest agent isn't in a vendor's workspace — it's a coding agent in a terminal with shell access. Governance that covers only the vendor's own agents governs the minority of your exposure.
A good answer — from anyone
A way to bring external agents — coding assistants, framework-built crews — under the same approval gates and onto the same record, so one review process covers the whole fleet instead of the vendor's slice.
Pantheon's answer · alpha
An external-agents API (gate + record endpoints) puts agents we don't run on the same record and approval queue — alpha: the API is live and documented, not yet validated against real CrewAI/LangGraph sessions, and we label it exactly that way.
- 13
Can you export evidence an auditor would accept — without the vendor's help?
If the record only exists inside the vendor's UI, your compliance story has a dependency with a pricing page. Evidence you can't take with you isn't yours.
A good answer — from anyone
A machine-readable export of actions, policy decisions, and approvals — generated from the record, never hand-authored — portable enough to verify or hand over independently. Certifications claimed honestly, with “in progress” said out loud.
Pantheon's answer · in dev
The record exports via API today and maps to EU AI Act record-keeping duties. Honest labels: structured SOC 2 / Art. 12 evidence packs and independent signature verification are in development; SOC 2 certification is not yet held. Two checks we don't fully pass — yet.
Answers that should end the evaluation
“Everything is logged.”
…with no answer to “who can edit the log?” — a mutable log is a liability shaped like an asset.
“We filter malicious prompts.”
…as the entire injection story — filtering natural language is a layer, not a defense.
A gate with a silent catch block.
…if approval infrastructure erroring means the action proceeds, you have a logger, not a gate.
Isolation as a policy document.
…if there's no automated test they can show you, isolation is an intention.
The agent holds your OAuth token.
…raw credentials in an agent's context leak through the same channel as everything else it reads.
Pending approvals that never expire.
…a queue that only grows trains people to approve by reflex — fatigue is the exploit.
The checklist, plain
Paste this into your evaluation doc. No attribution needed.
- Per-tool inventory: read vs. destructive, and a fail-closed default for unclassified tools
- Gate failure mode: approval plumbing breaks → action blocks (ask for the test)
- Tamper-evident record, with its limits stated (who can still rewrite?)
- Approver identity written atomically with the action, not joined later
- Decision context visible to the approver before deciding
- Credentials encrypted at rest, injected at use, never lent to teammates
- Tenant isolation enforced by architecture and CI-tested
- Ceilings: loops, time, spend — and a cap + expiry on pending approvals
- Injection stance: containment (fooled agent still can't act alone), not filtering
- Revocation in seconds, per tool / agent / workspace, record survives
- External agents (incl. coding agents) coverable under the same gates and record
- Evidence exportable and machine-generated; certifications claimed honestly
Why publish the two checks we don't fully pass? Because a checklist we could only win would be advertising, and you'd be right not to trust it. The full engineering detail behind every answer here — threat model, failure directions, isolation tests — is on our security page.