Success Criteria

---
name: success-criteria
description: >-
  Turns a plan or spec into machine-readable success criteria in qa-checklist.yaml. Use when: "define success", "success criteria", "how do we measure", "acceptance criteria", "definition of done", "generate checklist", "what does done look like", "create acceptance criteria", "make a QA checklist". PROACTIVE for a new feature with no metrics, and on feedback changing scope (affected checks only). LLM or agent features then hand off to /ai-done.
author: the repository owner
contributors: []
---

# Success Criteria Generator

Plans say what to build. Success criteria say how to know it's built correctly. This skill reads any plan, spec, or task list and produces a structured YAML checklist where every item is either automatically testable (run a command, check the output) or code-auditable (read specific files, verify specific logic).

The output format is designed to feed directly into the QA checklist loop (self-loop, or a continuation-enforcing loop plugin when one is enabled), which iterates until all checks pass.

> **AI / LLM / agent features:** this skill produces the *deterministic* definition of done. If the feature has a probabilistic surface (an LLM call, an agent, generation, or recommendation), hand off to `/ai-done` after this checklist exists. `/ai-done` reuses this checklist as its deterministic layer (L1) and adds the eval bands, failure-triage playbook, tripwires + rehearsed rollback, and post-ship loop that deterministic checks cannot cover.

## Why This Matters

Bad success criteria are the root cause of QA loops that never close. If criteria are vague ("app works well"), subjective ("UI looks good"), or incomplete (missing edge cases), the QA agent has no clear target. Every iteration becomes a judgment call, and judgment calls are where drift happens.

Good success criteria are:
- **Specific**: "GET /health returns HTTP 200" not "API works"
- **Measurable**: A command you can run, a file you can read, a condition you can check
- **Complete**: Covers every requirement in the plan, not just the obvious ones
- **Independent**: Each check can pass or fail on its own

## Reactive Triggers (run WITHOUT being asked)

This skill should auto-trigger whenever the user gives feedback that changes what "done" looks like:

- **Feature change**: "actually, make it do X instead of Y" → update affected checks
- **Bug report**: "this doesn't work" → add or modify the relevant check, then re-run the affected checks
- **Scope addition**: "also add Z" → add new checks to the checklist
- **Scope removal**: "we don't need W anymore" → mark those checks as blocked/ARCHIVED

When auto-triggered, run the **incremental update** path (see "Updating Existing Checklists" section), not a full regeneration. Update only the checks affected by the user's feedback, then re-run the affected checks.

## Procedure

### Step 1: Read the Plan

Accept a plan file path as argument: `/success-criteria path/to/plan.md`

If no path given, look for the most relevant plan:
1. Check if `docs/qa-checklist.yaml` already exists (if so, offer to regenerate or update)
2. Look for plan files in `docs/`, `<NOTES_LINK>/`, and `~/.claude/plans/` matching the current project
3. If multiple plans exist, ask which one to use (or use the plan index if available)

Read the entire plan. Extract every requirement, feature, endpoint, UI flow, and constraint mentioned.

### Step 2: Read the Codebase Structure

Understanding the actual code is essential for generating useful criteria. Without it, you'd produce generic checks that don't map to real files.

1. Read the project's file tree (focus on `src/`, `lib/`, `app/` directories)
2. Identify key entry points (main.ts, App.tsx, index.ts, main.dart, etc.)
3. Note the tech stack (this determines what automated checks are possible)
4. If a previous checklist exists, read it to preserve check IDs and any manual annotations

### Step 3: Classify Requirements

For each requirement from the plan, determine its type:

**Automated**, Can be verified by running a command and checking output:
- API endpoints (curl + status code check)
- Build commands (tsc, flutter analyze, npm run build, exit code 0)
- Test suites (flutter test, npm test, all pass)
- File existence checks (ls path/to/file)
- Database state (query + expected result)
- Process health (port listening, PID exists)

**Code-audit**, Requires reading source code and evaluating logic:
- UI conditional rendering ("if the cart is empty, show the empty-cart panel")
- Event handlers ("keyboard shortcut 'a' calls handleAddToCart")
- Data flow ("the checkout form posts to /orders")
- Business logic ("a discount code applies once per order and never below the floor price")
- Error handling ("API failure shows user-friendly message, not stack trace")

The classification matters because automated checks are faster, more reliable, and can run without human judgment. Maximize automated checks, only use code-audit when there's no command that can verify the requirement.

### Step 4: Generate the Checklist

Output to `docs/qa-checklist.yaml` in this format:

```yaml
project: {project-name}
created: {today's date}
source_plan: {path to plan file used}
max_retries: 3
build_command: "{the project's build/compile command}"
summary:
  total: {count}
  passed: 0
  failed: 0
  blocked: 0
  pending: {count}

checks:
  - id: {CATEGORY}{NUMBER}
    name: "{human-readable description}"
    type: automated
    command: "{exact bash command to run}"
    expected: "{expected output or comparison value}"
    comparison_mode: exact   # exact | regex | numeric | exit_code
    side_effects: false      # true if the check writes data or costs money
    status: pending
    retries: 0
    blocked_reason: null

  - id: {CATEGORY}{NUMBER}
    name: "{human-readable description}"
    type: code-audit
    files: [{list of files to read}]
    criteria: "{precise, unambiguous condition to verify}"
    status: pending
    retries: 0
    blocked_reason: null
```

Read `references/checklist-format.md` for the complete format specification, including the `expected` field's comparison modes.

### Step 5: Organize by Category

Group checks into logical categories using letter prefixes. Choose categories that match the plan's structure. Common patterns:

- By feature area: A (Onboarding), B (Dashboard), C (Actions), D (Navigation)
- By layer: A (API), B (Frontend), C (Database), D (Build)
- By priority: A (P0 Critical), B (P1 Important), C (P2 Nice-to-have)

Number checks sequentially within each category: A1, A2, A3, B1, B2, etc.

### Step 6: Quality Review

Before presenting the checklist, verify:

1. **Coverage**: Every requirement from the plan has at least one check. Read the plan again and cross-reference.
2. **Precision**: Automated check commands are syntactically correct and will actually work. Code-audit criteria are specific enough that two people would agree on pass/fail.
3. **Independence**: No check depends on another check passing first (order shouldn't matter).
4. **Build gate**: At least one check verifies the project compiles/builds cleanly.
5. **No gaps in IDs**: Categories are contiguous (A1, A2, A3, not A1, A3, A5).

### Step 7: Present Summary

Show the user:

```
SUCCESS CRITERIA GENERATED
==========================
Source plan: {plan file}
Output: docs/qa-checklist.yaml

Categories:
  A. {name} ({count} checks: {auto} automated, {audit} code-audit)
  B. {name} ({count} checks: {auto} automated, {audit} code-audit)
  ...

Total: {count} checks ({auto_total} automated, {audit_total} code-audit)
Build gate: {build_command}

Ready for the QA checklist loop.
```

## Comparison Mode & Side Effects

Every automated check MUST have explicit `comparison_mode` and `side_effects` fields.

### comparison_mode (required for automated checks)

| Mode | `comparison_mode` | `expected` value | Example |
|------|-------------------|------------------|---------|
| Exact string | `exact` | `"200"` | HTTP status code |
| Regex match | `regex` | `"/No issues found/"` | Substring pattern |
| Numeric comparison | `numeric` | `">= 100"` | Test count baseline |
| Exit code | `exit_code` | `"0"` | Build must exit clean |

### side_effects (required for automated checks)

Set `side_effects: true` when a check:
- Writes to a database (POST/PUT/DELETE endpoints)
- Calls external APIs that cost money (LLM generation, SMS, email)
- Mutates application state that other checks depend on

The checklist loop handles side-effect checks differently: they run **once, at the end**, outside the retry loop. This prevents data pollution and wasted credits.

Read-only checks (GET endpoints, build commands, file scans) are always `side_effects: false`.

## Project Templates

The public copy ships no bundled project templates. When a project already has a checklist
that works, read it first and adapt it rather than starting from scratch: that preserves
proven check patterns and stable IDs. `references/checklist-format.md` documents the schema
every checklist must follow.

## Updating Existing Checklists

If `docs/qa-checklist.yaml` already exists:

1. Read the existing checklist
2. Preserve check IDs that still apply (don't renumber)
3. Mark removed checks as ARCHIVED (don't delete, the QA run log may reference them)
4. Add new checks with the next available ID in each category
5. Update the summary counters
6. Show a diff: "Added X checks, archived Y checks, modified Z checks"

This matters because the QA runner tracks progress by check ID. Renumbering breaks the run log.

# QA Checklist YAML Format Specification

## Overview

The `qa-checklist.yaml` file is the shared state between `/success-criteria` (which generates it) and the QA checklist loop (which executes against it). It tracks every success criterion, its verification method, and its current status.

## Schema

```yaml
# Header — project metadata
project: string          # Project identifier (e.g., "<PRODUCT_REPO>")
created: date            # YYYY-MM-DD when checklist was generated
source_plan: string      # Path to the plan file used to generate criteria
max_retries: integer     # Max fix attempts per check before marking blocked (default: 3)
build_command: string    # Command that verifies the project compiles/builds

# Summary — updated by the checklist loop after each iteration
summary:
  total: integer
  passed: integer
  failed: integer
  blocked: integer
  pending: integer

# Checks — the actual success criteria
checks:
  - id: string           # Unique ID: {LETTER}{NUMBER} (e.g., A1, B3, H10)
    name: string         # Human-readable description
    type: enum           # "automated" or "code-audit"

    # For automated checks:
    command: string      # Exact bash command to run
    expected: string     # Expected output (see comparison modes below)
    comparison_mode: enum  # "exact" | "regex" | "numeric" | "exit_code" (explicit, don't infer from string)
    side_effects: boolean  # true if the check mutates state (writes to DB, calls external APIs, costs money). Default: false.
                           # the checklist loop runs side_effects checks ONCE at the end, outside the retry loop.
                           # Read-only checks (GET endpoints, build commands, file scans) are always false.

    # For code-audit checks:
    files: list[string]  # File paths to read
    criteria: string     # Precise condition to verify in the code

    # Status tracking (managed by the checklist loop):
    status: enum         # "pending" | "passed" | "failed" | "blocked"
    retries: integer     # Number of fix attempts so far
    blocked_reason: string|null  # Why this check can't be fixed (null if not blocked)
```

## Comparison Modes

Each automated check MUST have an explicit `comparison_mode` field. Do not infer the mode from the `expected` string, always set it explicitly.

| Mode | `comparison_mode` | `expected` value | Semantics |
|------|-------------------|------------------|-----------|
| Exact string | `exact` | `"200"` | stdout must exactly equal this string (trimmed) |
| Regex match | `regex` | `/No issues found/` | stdout must match this regex |
| Numeric >= | `numeric` | `">= 100"` | numeric stdout must satisfy the comparison |
| Exit code | `exit_code` | `"0"` | command's exit code must equal this number |

## Status Values

| Status | Meaning | Set by |
|--------|---------|--------|
| `pending` | Not yet attempted | success-criteria (initial) |
| `passed` | Check verified successfully | checklist loop |
| `failed` | Check attempted but failed | checklist loop |
| `blocked` | Cannot fix after max_retries attempts | checklist loop |

## ID Convention

- Letters = categories (A, B, C, ..., Z)
- Numbers = sequential within category (1, 2, 3, ...)
- IDs are stable, once assigned, never renumber
- When removing a check, mark it `blocked` with reason "ARCHIVED" rather than deleting
- When adding checks to an existing category, use the next available number

## Example

```yaml
project: <PRODUCT_REPO>
created: <YYYY-MM-DD>
source_plan: "<PLAN_PATH>"
max_retries: 3
build_command: "cd '<PRODUCT_REPO>' && npm run build"
summary:
  total: 3
  passed: 1
  failed: 1
  pending: 1

checks:
  - id: A1
    name: "The catalog page lists every ceramic mug in stock"
    type: automated
    command: "curl -s http://localhost:<APP_PORT>/api/mugs | jq 'length'"
    expected: "3"
    comparison_mode: exact
    side_effects: false
    status: passed

  - id: A2
    name: "Adding a mug to the cart updates the cart badge"
    type: code-audit
    files: [src/Cart.tsx]
    criteria: "Cart.tsx increments the count when addItem() runs, and the badge reads the same store."
    status: failed

  - id: B1
    name: "The build compiles with zero errors"
    type: automated
    command: "cd '<PRODUCT_REPO>' && npm run build"
    expected: "0"
    comparison_mode: exit_code
    side_effects: false
    status: pending
```