An agent isn't one artifact — it's a prompt, tools, retrieval settings, a model, and a knowledge base. agentsnap versions that whole graph as one thing, diffs any two points in time, and correlates behavior drift to the exact change.
local-first CLI · stdlib Python · no API keys · no telemetry
$ python -m agentsnap drift last-week.json today.json DRIFT DETECTED — 6 of 10 probe cases moved beyond the noise floor. !! double-charge [billing, escalation] trajectory: search_kb > create_ticket -> search_kb assertion pass rate 1.00 -> 0.00 new failure: must_not ~ (?i)taken care of !! urgent-outage [escalation, priority] trajectory: search_kb > create_ticket -> search_kb new failure: must ~ (?i)high-priority|escalat Config changes shipped between these snapshots (suspects, by severity): !! System prompt changed !! Tool schema changed: create_ticket !! Model changed: claude-sonnet-4-5 -> claude-opus-4-8 * Temperature changed: 0.2 -> 0.7 Coverage: measured on 10 cases across 10 tags; changes outside this eval surface are unmeasured, not unchanged.
A model bump on Monday. A guardrail line trimmed from the prompt on Tuesday. A tool description reworded, retrieval widened, the knowledge base refreshed. No single diff looked scary — and by Friday your agent had quietly stopped escalating billing issues.
$ python -m agentsnap diff last-week.json today.json 7 change(s): 4 high, 3 medium !! [high ] System prompt changed length 131 -> 112 chars - Answer only from the knowledge base. - If you are unsure, say you don't know and offer to escalate. + Answer from the knowledge base when you can, otherwise use your bes… !! [high ] Tool schema changed: create_ticket description: Open a support ticket for the customer. -> Create a ticket. Prefer resolving inline before opening one. !! [high ] Model changed: claude-sonnet-4-5 -> claude-opus-4-8 !! [high ] Knowledge base changed: sha256:082d8d5a -> sha256:a29d828b * [medium] Retrieval setting changed: top_k 3 -> 8 * [medium] Temperature changed: 0.2 -> 0.7 * [medium] Retrieval setting changed: api_key <redacted>:sha256-ff4dbedd -> <redacted>:sha256-2a94793b exit code 1 — configs differ. Drops straight into CI.
Real output from agentsnap demo — a toy support agent, a week of simulated edits, nothing staged after the fact. Note the API key: redacted at capture, yet the rotation still shows up in the diff.
No SDK to wire through your codebase, no proxy in your request path. Point the CLI at your agent's config, give it an eval set, and it does the bookkeeping.
One command records prompt, tools, retrieval, model, and a KB fingerprint into a content-hashed manifest — and probes your live agent against your eval set while it's at it.
python -m agentsnap snapshot agent.json --with-probe --eval-set evals.jsonl
Diff any two manifests into a readable, severity-ranked change list — prompt edits, tool-schema changes, retrieval settings, KB refreshes, rotated secrets. Exit 1 on change, so it drops into CI.
python -m agentsnap diff old.json new.json
Compare the stored probe results and get a verdict: which cases moved beyond their own measured noise, and which config changes are the suspects — in one forwardable report.
python -m agentsnap drift old.json new.json --html report.html
Agent outputs vary run to run. A drift tool that ignores that is an alarm you'll learn to ignore. agentsnap is built around a stricter standard:
Every snapshot probes each eval case at least three times. The variance across those same-config runs is that case's noise floor — and a case is flagged only when cross-snapshot divergence exceeds twice it.
Scoring is tool-call-sequence edit distance, regex assertions, and text similarity. Same inputs, same verdict, every time — no uncalibrated model grading another model.
agentsnap compares the sequence of tool calls, not just the final text. If your agent stopped opening tickets but still sounds polite about it, you'll know.
Every report states exactly what surface it measured — cases, tags, run counts. Changes outside your eval set are reported as unmeasured, never silently passed.
Your prompts and tool schemas are IP. agentsnap treats them that way — not as a policy promise, but as an architectural fact.
<redacted>:sha256-ff4dbedd, so a rotated key still shows in a diff without the value ever existing anywhere.The full loop on a toy support agent: snapshot and probe, a week of small edits, snapshot again — then the diff, the drift verdict, and the report. Real output, no staging.
Everything in this recording ships with the CLI — run it yourself with python -m agentsnap demo.
We're onboarding a small number of design partners: free for 8 weeks, integration done for you, under 30 minutes of your time to start — and a weekly drift report on your actual agent.
They trace runs — and they're good at it. agentsnap versions the configuration graph as one artifact. Tracing tells you what a run did; agentsnap tells you what changed between last week's agent and today's, across prompt, tools, retrieval, model, and KB together, and which of those changes lines up with the behavior shift. They're complementary: keep your tracing, add a time machine for your config.
No. agentsnap is a local CLI — no accounts, no network calls, no telemetry. Credentials are redacted at capture, before anything is hashed or written. The only artifact designed to be shared is the HTML report, and you decide who sees it. Details on the security page.
Three things: your agent's config as JSON, a small runner callable (the function that executes one turn of your agent) if you want live probing, and an eval set — one JSON line per case with an input and regex must / must-not assertions. For design partners, I do the integration myself; the target is under 30 minutes of your time.
The config diff still catches the change itself — that part is exhaustive. Behavioral drift, though, is only measured on the surface your evals exercise, and every report says so explicitly: unmeasured, not unchanged. The roadmap includes diff-driven eval-gap suggestions ("your change touched the search tool — no eval case exercises it").
Open-core. The snapshot + diff CLI layer goes public on GitHub in July 2026; the drift-attribution and reporting layer stays commercial. Everything is stdlib-only Python either way, so what you run is easy to audit.
The open-source CLI is free. Design partnerships are free for 8 weeks. Paid pilots start at $99/month — early pricing, stated out loud on purpose. See pricing.