How agentsnap works
From a config file to a defensible answer to "which change did this?" — manifests, typed diffs, probe-at-snapshot-time, and drift scoring calibrated against a measured noise floor.
The whole config graph, as one versioned artifact.
Each snapshot records everything that controls your agent's behavior:
- Model version and temperature
- System prompt, verbatim
- Tools — name, description, and parameter schema for each
- Retrieval settings — top-k, similarity, embedding model
- KB fingerprint — a hash of the knowledge base contents
- Probe results (optional) — live behavior measured at snapshot time
A stable content_hash is computed over the canonicalized config: two identical configs hash equal regardless of key or tool ordering, and any meaningful change moves the hash. Probe results never affect it — the hash identifies the configuration.
{
"schema": 2,
"label": "prod-2026-07-04",
"content_hash": "sha256:f7c4f4b4…",
"agent": {
"model": "claude-sonnet-4-5",
"temperature": 0.2,
"system_prompt": "Answer only from the knowledge base. …",
"tools": [
{ "name": "search_kb", … },
{ "name": "create_ticket", … }
],
"retrieval": {
"top_k": 3,
"api_key": "<redacted>:sha256-ff4dbedd"
},
"kb_hash": "sha256:082d8d5a…"
},
"probe_results": { "cases": 10, "runs_per_case": 3, … }
}
Not a wall of JSON. A typed, severity-ranked change list.
Diffing two manifests produces changes you can read out loud in a standup: "system prompt changed, guardrail line dropped, high severity." Prompt edits arrive as line-level diffs; tool-schema changes are reported distinctly from prompt changes; a rotated secret shows up by digest without the value ever being stored.
high vs medium
Model, prompt, tool-schema, and KB changes rank high; parameter nudges like temperature or top-k rank medium — so triage starts sorted.
Exit codes that mean something
0 — identical. 1 — configs differ. 2 — error. Gate a deploy on "the agent config changed and nobody said so" with one line in your pipeline.
--json on every command
Human-readable by default, structured output when you want to feed dashboards, bots, or your own tooling.
Behavior is measured live, at snapshot time.
A manifest is a description, not an executable agent — so agentsnap doesn't pretend it can time-travel. With --with-probe, each snapshot runs your eval set against the live agent and stores the results in the manifest. Drift is then a comparison of two stored measurement blocks — reproducible forever.
Three deterministic signals
- Trajectory — edit distance between tool-call sequences. A right answer reached through a broken path is still drift.
- Assertions — regex must / must-not checks per case. "Must mention escalation" is a contract, not a vibe.
- Output similarity — text similarity as a soft, supporting signal — never the sole trigger.
The noise floor
Each snapshot probes every case at least 3 times. The variance across those same-config runs is that case's noise floor. A case is flagged only when cross-snapshot divergence exceeds 2× its own noise — we measure our own noise before accusing your config. No LLM judge anywhere in the loop.
!! yearly-discount [billing, sales] trajectory: search_kb > create_ticket -> search_kb trajectory diverged 0.50 (noise 0.00) output diverged 0.88 (noise 0.00) assertion pass rate 1.00 -> 0.00 new failure: must ~ (?i)escalat|ticket new failure: must_not ~ (?i)10% off Stable within noise: order-status, premium-support, refund-policy, shipping-time Mean probe latency: 698ms -> 1238ms
One page you can forward to whoever owns the agent.
Verdict at the top, config diff next, flagged cases with their numbers, and a method section that says exactly how the conclusion was reached — including a coverage statement admitting what wasn't measured. Self-contained HTML, generated locally.
A real report from the demo — scroll it. Footer reads: "Generated locally by agentsnap — your config never left this machine."
The whole surface, honestly small.
Stdlib-only Python. No API keys, no accounts, no daemon. The CLI goes open-core on GitHub in July 2026; design partners get it today.
Snapshot a config
python -m agentsnap snapshot my_config.json --label prod-2026-07-04
Snapshot and probe the live agent
python -m agentsnap snapshot my_config.json --with-probe --eval-set my_evals.jsonl --runner my_package.agent:run_agent
Diff two manifests (exit 1 when they differ)
python -m agentsnap diff snapshots/old.json snapshots/new.json
Score drift and write the report
python -m agentsnap drift snapshots/old.json snapshots/new.json --html report.html
Eval sets are one JSON line per case
{"id": "double-charge", "input": "I was charged twice for my order", "assertions": {"must": ["(?i)escalat"], "must_not": ["(?i)taken care of"]}, "tags": ["billing", "escalation"]}
What's next — built on demand, not on speculation.
A standing rule from day one: nothing beyond the core ships until a named team asks for it in a call. Here's what's committed:
Packaging & onboarding
pip install agentsnap, an agentsnap init scaffold, and a documented config schema — so first snapshot takes minutes, not an afternoon of reading source.
Drift timeline report
Drift across many snapshots, not just two — plus eval-gap suggestions: "your change touched the search tool; no eval case exercises it."
First stack adapter
Config extraction and trace ingestion for one stack — chosen by the first design partner's actual infrastructure, not by a market map.
Rollback
Restore the full config graph to any snapshot, component-wise. Attribution proves its value first; write access to your deploy pipeline is earned, not assumed.
Want this running on your agent?
First report on your own config is free and takes about 30 minutes of your time — I do the integration.