unit U4 — 4 of 4

Eval hooks

golden sets, regression checks

A prompt is code, and like any code it regresses. Tighten a system instruction, bump the model version because the provider deprecated the old one, or add a schema field, and the output can quietly get worse on inputs you never re-tested. A golden set — a fixed collection of representative inputs paired with known-good outputs — turns that invisible risk into a measurable one. Run the current prompt over the set, compare, and you have a pass rate you can gate a deploy on.

Wire it as its own flow. A Schedule trigger, or a manual run before every prompt change, loops the golden inputs through the exact live prompt, then a Code Piece compares each result to its expected output — exact match for enums, field-by-field for structured objects, or a cheap semantic score for free text. Aggregate into a pass rate, write it to a store the run history can surface, and fire an alert through a Slack or email step when the score drops below your threshold.

Where it breaks: shipping prompt changes on vibes. Without a golden set, the only regression test is production traffic, and the only alarm is a client noticing that categorisation got sloppy last Tuesday. By then you have dirty data to backfill and trust to rebuild. Grow the set every time a real failure slips through, so each incident becomes a permanent guard.

worked example

A golden-set fixture pinning known-good triage outputs so a prompt or model change that regresses them fails the check.

[
  {
    "id": "gold-001",
    "input": "Card declined twice, still charged. Furious.",
    "expected": { "category": "billing", "priority": "high" }
  },
  {
    "id": "gold-002",
    "input": "Any plans for a dark mode toggle?",
    "expected": { "category": "feature", "priority": "low" }
  },
  {
    "id": "gold-003",
    "input": "App crashes on export since the update.",
    "expected": { "category": "bug", "priority": "high" }
  }
]

field checklist

Pin representative inputs with known-good outputs as a golden set.
Re-run the set before shipping any prompt or model change.
Compare per field, not on the whole blob of text.
Alert through Slack when the pass rate drops below threshold.
Add every escaped production failure back into the set.

common failure — Silent regression after a model bump

A provider deprecated the pinned model, so the team swapped the version id and shipped without re-testing. The new model read terse tickets as low priority, and high-priority billing complaints sat unrouted for three weeks before a client escalated. There was no golden set, so nothing caught the drop. Build one from real tickets, gate the deploy on its pass rate, and no model swap reaches production unchecked.

check your understanding

A provider deprecated the pinned model, the team swapped the version id, and high-priority billing complaints sat unrouted for three weeks. What would have caught this before it shipped?

sandbox validation

The check above confirms you followed the unit. Marking the module COMPLETED takes more: build the automation in your own engine and submit the exported flow and its run evidence, signed, to your unique validation URL. See the module page for that spec.