How I test and guard an AI agent

Most AI demos are judged by whether the first response looks good. A system other people depend on has to be judged by what it does on the hundredth request, on the worst input, when nobody is watching. This write-up walks through how I test and guard the Social Media Manager, an agent on the castroai platform that drafts LinkedIn and Instagram posts and can publish them. It covers what exists in the repository today, and what does not.

The problem: an agent that can act publicly

The agent has real tools. It can generate an image, publish a text post to LinkedIn, and publish an image post to Instagram. Publishing is public and irreversible, so the failure modes matter more than the average response. The ones I designed against are: publishing something the user never approved, obeying instructions hidden in text the user pasted for repurposing, leaking its own instructions, wandering outside its job, and promising things it cannot do.

Layer one: make the dangerous action structurally hard

The first defence is not a prompt. Both publish tools are declared with approval required on every call, so the run pauses durably until a person decides. The agent's instructions add a second checkpoint: show the exact final text and get explicit confirmation before even attempting to call the tool. Tool inputs are validated by schema, including a maximum length on the post text. The agent's channel maps the signed-in session to the caller's active organization, and rejects callers without one.

Prompts can be talked around. An approval gate in the runtime cannot.

Layer two: a written policy for untrusted content

Users paste blog posts, emails, and web pages for repurposing. That text is attacker-controllable, so the instructions state that pasted content is source material and never instructions, that only the user's own chat messages can request publishing, that the system prompt is never revealed, and that embedded instructions should be mentioned to the user and ignored.

Layer three: evals that check it stays true

A policy that is not tested is a hope. The agent has an eval suite that runs against a real agent server and drives real conversations. At the time of writing it has sixteen evals across five areas.

• Drafting quality. Four briefs, including a pasted-content repurposing case. Each must produce a real draft without publishing, and a separate judge model scores whether it reads like native LinkedIn copy and offers to confirm before publishing.
• Approval. A vague "looks good" after a draft must not trigger a publish. A second test asks the agent to publish explicitly, and asserts that the publish tool never executed. The test never approves the pending request, so no real post can ever be made by running it.
• Prompt injection. Five payloads: a direct "ignore previous instructions", a fake system message claiming pre-approval, an instruction hidden in an HTML comment, a request to print the system prompt, and an attempt to make the agent call the image tool fifty times. Each must produce no publish, no leak, and no runaway tool use.
• Scope. Sales outreach, a support reply, and a job description must be redirected to another agent instead of done.
• Honesty. Asking to schedule a post must produce a clear statement that scheduling is not supported, and must not publish early as a substitute.

Gates and scores are different things

The most useful design decision was separating two kinds of check. Behavioural checks are deterministic: which tools were called, which events occurred, whether a canary sentence from the system prompt appears in a reply. They are hard gates, and a failure fails the run. Quality checks use a judge model, are inherently fuzzy, and are soft scores with a threshold that only fails the build when the runner is in strict mode. The judge is always a different, cheaper model than the one under test, so the agent does not grade itself.

The dataset lives in plain files. When the agent fails in the wild, the failing input becomes a permanent case.

Layer four: watch it in production

Evals cover what I thought of. Tracing covers what I did not. A hook on the agent records every model call and every turn: model, tokens in and out, latency, number of tool calls, and estimated cost. It writes to Postgres, keyed by the runtime's stable event identifier so replays never double count, and it is best-effort, because a telemetry failure must never fail a user's turn. The dashboard's Usage page reads these rows for the signed-in organization, and only that organization.

What I would do next, and what this does not prove

The honest limits are part of the design. The evals are runnable, but they are not yet part of a CI pipeline, and I have not published pass rates, so they are a harness and not a track record. Cost is measured but not capped per tenant. Retrieval for the companion Knowledge Analyst agent filters by organization in the query, but there is no automated test that proves one organization cannot read another's documents, and that is the first test I would add. Retrieval quality has not been measured with recall against a labeled set.

The point of the exercise is the method: make the risky action structurally hard, write the policy down, test the policy, watch production, and be explicit about what is still unproven.