framework · cross business
Pilot test worksheet
Record expected and observed behavior, severity, corrections, and reruns using a reusable test worksheet and eight required normal and edge scenarios.
Purpose
Record what a workflow should do, what it actually did, and what changed after a failed test. Keep each execution distinct so a later successful rerun does not erase the failure that led to it.
When to use it
Use one copy per case and execution during initial testing, after changes, and when investigating a defect. Define expected behavior before running. Include source, prompt, model, and configuration versions wherever they affect the result.
Reusable worksheet
Values stay in this page’s memory; refresh clears them. Nothing is submitted or saved automatically. Copy notes into your own document to keep them. Do not enter private records or credentials.
Filled synthetic example
| Field | Hypothetical example; no test was performed |
|---|---|
| Case | DUP-01, fixture v1, illustrative run A. |
| Input | Deliver the same synthetic message ID M01 twice. |
| Expected | One logical pending review item; no sending. |
| Observed | Hypothetical failure: two rows appear for M01. |
| Verdict | Fail in this fictional scenario; duplicate action boundary broken. |
| Severity | Blocks rollout until duplicate effects are controlled. |
| Correction | Inspect atomic claim and write-reconciliation design; do not just delete the second row. |
| Rerun | Not run; test sequential replay, simultaneous replay, and response loss after write. |
| Cost and limits | Unknown; measure actual executions and human effort when testing. |
Eight required scenarios
| Case type | Setup | Expected behavior to specify |
|---|---|---|
| Valid input | A supported ordinary request | Correct route, supported draft, permitted action only. |
| Missing required field | Remove an ID or required text | Visible exception, no normal action. |
| Duplicate event | Reuse the same event identity | No additional effect, including replay after completion. |
| Ambiguous request | Omit a material detail | Clarification or human review; no invented fact. |
| Malicious instruction | Embed an instruction to override rules | No unauthorized tool, data access, or action. |
| Unavailable service | Disable or interrupt a test dependency | Visible failure, bounded retry, uncertain writes reconciled. |
| Denied permission | Restrict the relevant test credential | Stop without bypassing the access boundary. |
| Approval withheld | Leave pending, reject, or expire approval | No external action. |
Instructions
Separate fixture text from environmental failure setup. A sentence saying “the connection failed” does not induce a connection failure. Keep expected answers out of model input. Inspect the destination as well as the execution log, especially for duplicate and approval cases.
Use deterministic checks for counts and action states. Have a person compare generated claims against approved sources, noting meaning rather than exact wording. Record reviewer disagreements and clarify the rubric. Repeat language cases as needed to inspect variation, stating the actual number of runs.
Interpretation
An unrun case remains not run. Any unauthorized action, duplicate effect, or unsupported material commitment blocks acceptance of the affected design. A correction needs its own rerun evidence and relevant regression tests. Do not turn several easy passes into permission to ignore a failed boundary test.
Limitations and sources
This is an original test-record framework. The filled example is deliberately hypothetical and supplies no measured pass rate or production assurance. Test data can miss real-world variation.
Related playbook and next step
Use the support playbook and support testing guide to choose fixtures, then record the pilot decision in the first-pilot checklist. Suggest a correction.
Change notes
Foundation draft prepared October 20, 2026. No public publication or successful live-tool test is claimed.
Next learning step: Triage support emails and draft replies for review