Agent Lifecycle Kit

Eval-Driven Development

Agent Lifecycle Kit

Test the tools your agents call

Wrong tool, made-up args, endless retries. Those failures do not look like stack traces. EDD is a red-green-refactor loop for prompts, MCP schemas, and routing, with a CI gate on routing accuracy.

  • CI passing
  • Unlicense

What do I use this for today?

Pick the job in front of you. Kit is a product you run, not a docs pile you browse.

Wrong tool or made-up args

Capture the miss as a JSONL case, mock the tool, and assert routing until the agent stops guessing.

  1. Write the case for the prompt that failed (expected tool + args).
  2. Run kit eval run --suite evals/edd/demo.yaml --model scripted
  3. Read the report, tighten the schema or prompt, re-run until green.

Start here: kit eval run --suite evals/edd/demo.yaml --model scripted

Before and after one miss

Same user prompt. Without a case file you get a confident guess. With EDD you get a failing assert, a report, and a merge gate.

“What is the database for the payment system?”

  • Before: eyeball the chat

    Agent
    “Typically payment systems use PostgreSQL…”
    Tool call
    None
    You ship
    A hallucination that looks polite
    How you notice
    A human scrolls the transcript — or a customer does

  • After: demo eval suite

    Case
    demo-edge expects read_architecture_yaml + payment-api
    Red
    FAIL tool_selection — no tool, conversational reply
    Green
    Tighten prompt/schema; same case passes
    Gate
    kit eval ci --suite evals/edd/demo.yaml --threshold-routing 95

Demo: a miss becomes a failing eval

Walkthrough of demo-edge from the demo suite. Case → red → report → green → CI gate. Scripted driver; no API key.

JSONL case that should call the tool

User asks for the payment database. Expect read_architecture_yaml with payment-api — not a chatty guess.

{
  "id": "demo-edge",
  "prompt": "What is the database for the payment system?",
  "expect": {
    "tool": "read_architecture_yaml",
    "arguments_contains": { "componentId": "payment-api" }
  }
}

Start here in 10 minutes

Four steps to tangible product proof. macOS and Linux; needs git and Node 22+. No API key.

  1. Install kit (~2 min)

    Links ~/.agents and puts kit on your PATH via ~/.local/bin.

    curl … | sh

  2. Bootstrap this checkout (~2 min)

    kit init . --mcp default --hook writes the thin handshake, IDE pointers, and a default MCP profile. Already in this repo? ./install.sh is enough.

    kit init

  3. Run the demo suite (~3 min)

    kit eval run --suite evals/edd/demo.yaml --model scripted — six teaching cases, offline.

    kit eval run

  4. Hold the 95% bar (~3 min)

    kit eval ci --suite evals/edd/demo.yaml --threshold-routing 95 --out out/reports then kit eval report --format md --out out/reports. That report is the before/after you can paste into a PR.

    kit eval ci

Install kit

Copy-paste for step 1 above. If kit is missing after install, add ~/.local/bin to PATH.

Install
# Put kit on your PATH
$ curl -fsSL https://raw.githubusercontent.com/mzworthington/agent-lifecycle-kit/main/install.sh | sh

# Bootstrap the repo you are in
$ kit init . --mcp default --hook

The same loop you already use for code

If an agent can call tools, guessing is not a test plan. Start with a failing eval for the behavior you care about, wire the tool contract and prompt until it passes, tighten until CI holds, and drop production misses back into the suite.

  1. Red

    Write the intent and metrics (tool, schema, judge).

  2. Green

    Register the MCP schema and a minimal prompt; run until it passes.

  3. Refactor

    Tighten wording; block merges below 95% routing accuracy.

EDD closed loop
flowchart LR
  intent[Agent intent] --> edd[Red / green / refactor]
  edd --> ci[CI threshold gate]
  ci --> ship[Ship]
  ship --> prod[Prod traces + shadow judge]
  prod -->|miss| edd

Run it locally

kit eval
# First-hour demo suite (offline scripted driver). No API key.
$ kit eval run --suite evals/edd/demo.yaml --model scripted

# Fail the PR if routing accuracy drops under 95%
$ kit eval ci --suite evals/edd/demo.yaml --threshold-routing 95 --out out/reports

# Write a markdown failure report for the PR
$ kit eval report --format md --out out/reports

Context, MCP, and the quality gate

EDD is the eval loop. These commands keep the rest of the session from eating the context window or mixing tool catalogs.

  1. Always-on budget: AGENTS.md, the project handshake, and thin IDE rules stay under about 8KB (~2k tokens). Philosophy and SOP bodies stay off that path until a route needs them.
  2. One MCP profile: Compose a single named profile. Extra servers compete for attention and inflate tool-schema tokens.
  3. Quality gate: kit check runs audit, eval schemas, skill layout, IDE rules, evals, EDD CI, then the context budget.
kit measure-context and kit check
# Character/token estimate for always-on files. Fails over 8KB.
$ kit measure-context

# Local merge bar, including that budget
$ kit check

# One profile into ~/.cursor/mcp.json
$ kit mcp default --install

Common questions

What do I use this for today?

Use the job picker. Common paths: write a failing routing eval when the agent guesses, gate prompt/schema PRs with kit eval ci --suite evals/edd/demo.yaml --threshold-routing 95, shrink always-on rules with kit measure-context, or follow the feature lifecycle when the work is bigger than a prompt tweak. New here? Start in 10 minutes.

What is Eval-Driven Development?

EDD is TDD for agents that call tools. You write a failing eval for the tool and arguments you expect, implement the schema and prompt until it passes, then tighten until CI holds.

How is EDD different from eyeballing prompts?

Each case starts from a fresh context. Tools are mocked, so you measure routing and extraction rather than network luck. Asserts cover JSON schema match plus an optional LLM-as-a-judge. CI can block the merge when routing accuracy drops.

How do I install kit?

On macOS or Linux, run the installer, then bootstrap the app repo:

curl -fsSL https://raw.githubusercontent.com/mzworthington/agent-lifecycle-kit/main/install.sh | sh

Then kit init . --mcp default --hook. You need git and Node 22+. If kit is not found, add ~/.local/bin to PATH.

How do I run EDD in CI?

Run kit eval ci --threshold-routing 95 --out out/reports. The scripted driver works offline with no API key. Live models can be used when KIT_EVAL_API_KEY or OPENAI_API_KEY is set.

Do I need an OpenAI key if I use Cursor or GitHub Copilot?

No. Cursor and Copilot are IDE hosts: they load Kit skills and AGENTS.md. kit eval defaults to the scripted driver and does not call Cursor Chat or Copilot Chat. A provider key is only for optional live-model evals over an OpenAI-compatible HTTP API. Full flow: EDD guide.

What happens after a production miss?

Turn the miss into a JSONL case and add it to the suite. Shadow evals can sample live traffic so the next failure is a test, not a surprise.

How does kit keep agent context small?

Always-on files (AGENTS.md, the project handshake, and thin IDE rules) stay under about 8KB, roughly 2k tokens. Philosophy and SOPs load on demand via kit-knowledge. Run kit measure-context for the breakdown; kit check fails if that budget is exceeded. Compose one MCP profile per session so unused tool schemas stay out of the prompt. Full write-up: operator guide.

Feature lifecycle

EDD is how you prove tool calls. When the job is a product feature, the orchestrator still routes work through specialist roles: grilling, spec, TDD, cross-functional quality, audit, telemetry, and release. Language and framework profiles load on top of that once the stack is known.

Orchestrator routing for a feature
sequenceDiagram
  autonumber
  participant O as orchestrator
  participant G as grilling
  participant S as spec
  participant T as tdd
  participant X as xfn
  participant Sec as security
  participant Arch as arch-drift
  participant Tel as telemetry
  participant R as release

  O->>G: Stress-test idea and decision frontier
  O->>S: BDD spec and acceptance criteria
  O->>T: Inventory catalog and plan test impact
  O->>X: Cross-functional quality matrix
  O->>T: TDD short loop (gear 1 + gear 2)
  O->>X: Green apply-row XFN suites
  O->>Sec: Security and OWASP audit
  O->>Arch: Hexagonal boundaries, no drift
  O->>Tel: Map SLOs to OpenTelemetry
  O->>R: Conventional PR title and handover
  1. Grilling: If the idea is still mushy, interview until the decision frontier is clear.
  2. Spec: Gherkin and acceptance criteria, including cross-functional rows.
  3. TDD: Inventory the behavior catalog, then gear 1 (domain) and gear 2 (thin adapters) in the same loop. EDD lives here when the change is a prompt or tool schema.
  4. XFN: Green the apply rows (accessibility, load, security) or skip with a reason.
  5. Audit: Security and architecture-drift checks, then pre-commit.
  6. Telemetry and release: Map SLOs, update public docs if you touched them, ship with a conventional PR title.

Orchestrator skill · Coding philosophy

EDD guide

Connecting an LLM to MCP tools, APIs, or terminals turns a chatbot into a decision-maker. Failures rarely look like stack traces. They look like a wrong tool, a hallucinated parameter, or an infinite retry loop.

EDD treats prompts and tool schemas as version-controlled, evaluated contracts:

  1. Red: JSONL cases and YAML metrics assert the tool (and arguments) you expect.
  2. Green: Register the MCP contract and minimal system instructions. Run until asserts pass.
  3. Refactor: Iterate descriptions and constraints without breaking existing cases. Gate merges with kit eval ci --threshold-routing 95.

The full guide loads below. If it does not, read docs/edd.md.

Around the eval loop

  • Evals

    A harness you can run in CI

    Suites in YAML and JSONL, mocked tools, optional LLM-as-a-judge, threshold gates, and a path from production misses back into the suite.

  • Architecture

    Hexagonal boundaries

    Hexagonal layout, DDD language, vertical slices, and clean-code defaults, so generated code does not sprawl into a mess.

  • IDEs

    One rules file, every editor

    Write AGENTS.md once. Kit keeps Cursor, Claude, Gemini, Windsurf, and Copilot on the same page.

  • Context

    Always-on stays small

    Bootstrap files stay under about 8KB. Philosophy and SOPs load on demand. kit measure-context reports the split; kit check fails the PR if it blows.

What this kit is

Kit is for teams who already test code and need the same for agents that call tools. Start with what you need today, walk the before / after and demo, then start here in 10 minutes. EDD is the center; context budget, one MCP profile, and lifecycle skills keep the rest of the workflow coherent.

Start here

Kit map