unit U3 — 3 of 5
Golden runs
real payloads pinned to expected output
A golden run is a real payload paired with the output it should produce. Collect a handful that cover the shapes you actually see — the clean lead, the one with a missing phone number, the one where the company field is an empty string, the duplicate — and you have a regression test for the whole flow. Replay them after every change and you find out whether you broke something before the client does.
Build it as its own flow: a trigger you fire manually, a loop over the stored golden payloads, and a comparison step that checks each result against its expected output field by field. Compare the fields that matter rather than the whole blob — a timestamp or a generated id will differ on every run and would fail every comparison for no reason. Aggregate to a pass count you can read at a glance, and keep the failures visible with enough detail to see which field diverged.
Where it breaks: a golden set built from imagination. Payloads you wrote yourself all have every field populated and spelled the way you expected, so the set passes forever while production keeps breaking on the records you never thought of. Harvest them from real traffic, and every time something breaks in production, add that payload to the set so the same failure can never return unnoticed.
worked example
A golden set for a lead-intake flow — the awkward records are the ones worth pinning.
[
{
"id": "golden-clean",
"payload": { "email": "[email protected]", "phone": "+4477", "company": "Acme" },
"expect": { "route": "sales", "score": 80, "valid": true }
},
{
"id": "golden-empty-company",
"payload": { "email": "[email protected]", "phone": "+4477", "company": "" },
"expect": { "route": "review", "score": 40, "valid": true }
},
{
"id": "golden-missing-email",
"payload": { "phone": "+4477", "company": "Acme" },
"expect": { "route": "reject", "valid": false }
}
]
// compared per field - never on the whole object:
// generated ids and timestamps differ every run and would fail foreverfield checklist
- Harvest golden payloads from real traffic, not from imagination.
- Include the awkward records: empty strings, missing fields, duplicates.
- Compare per field, skipping generated ids and timestamps.
- Replay the whole set before every publish.
- Add every production failure to the set so it cannot return.
common failure — A green suite that tested nothing real
A team wrote twelve test payloads by hand and every one had all fields populated and correctly typed. The suite passed for months while production kept failing on records with empty company strings — a shape nobody had imagined. Harvest golden payloads from real webhook logs, deliberately include the malformed ones, and grow the set from each incident so every escaped bug becomes a permanent guard.
check your understanding
Your golden-run comparison fails every single time, on every payload, even ones you have not changed. What is the most likely cause?
next unit opens once this is passed
sandbox validation
The check above confirms you followed the unit. Marking the module COMPLETED takes more: build the automation in your own engine and submit the exported flow and its run evidence, signed, to your unique validation URL. See the module page for that spec.