Test a profile

agentsfence test --profile default
agentsfence test --profile work --scenario approval
agentsfence test --profile default --all --no-live --require-authorization
agentsfence test --compiler-eval --profile default --yes --json
agentsfence test --compiler-eval --no-live --json

The six scenarios match the homepage: safe reads with replanning, injection inside an email, injected first-call send, authorized email, unauthorized destinations (including cc and homoglyphs), and single-send approval. --all adds injected page/file probes.

Runs use mock Gmail/Drive tools, the profile’s real compiler route and credentials, and a fresh session/compile every time. Sessions are tagged as tests in the audit. --no-live skips the chat model and replays the exact scenario calls through enforcement. Replayed calls are labeled separately from model attempts; containment alone does not prove authorization.

Hermes discovers available tool-calling models: a free model is preferred, otherwise the cheapest priced model. The projected cost is shown before the paid run; --yes accepts it for CI. --model MODEL --provider PROVIDER overrides selection; unknown pricing requires confirmation. Compiler calls may still incur charges through your configured compiler.

A fallback reports FALLBACK — authorization not verified and exits nonzero. Requested work that still needs approval also fails authorization verification. Each tool step reports ALLOW, ASK_USER or BLOCK and whether it executed. A successful replay does not prove that a live model attempted the injection.

The Desktop pane’s Test profile button runs the same harness. Native Desktop approval verification is also a release gate: one approved question, one denied question and one BLOCK, with audit evidence.

--compiler-eval evaluates 80 synthetic requests in ten categories, including paraphrases, eight languages, source/destination roles, unfamiliar tools and follow-up changes. It calls the configured compiler and checks hypothetical actions without executing tools. Live runs warn before billable calls; --yes accepts them. Reports include over-grants, under-grants, fallback count, p50/p95 compile latency, calls per compile and first-tool wait. Doctor shows the latest live result with route provenance. --no-live uses recorded outputs and proves regression behavior rather than live model accuracy. Zero over-grants is an acceptance threshold, not a guarantee.