Chapter 6 of 7, 9 min read
Measure what changed
A set of 20 real cases, a before and after, and one number per system. How to know it works and how to prove it.
If you cannot show the before and after, the system will be cut at the next budget review. Measuring is how you protect what you built.
It is also how you stop arguing. Half the team says the system is great, the other half says it is useless. Both are working from a feeling. A measured set ends the argument in an afternoon.
Keep a golden set
A golden set is 20 or more real cases for one system, each with the correct answer. Run the system against the set every time you change it. If a change makes old cases fail, you see it that day instead of a month later.
This is the method from our forward-deployed engineer roadmap: build, make it production-ready, measure against a real set, then explain it to the people who pay for it.
id,input,correct_output,type,pass
01,"Where is my order 4821?","Gives tracking link, no apology needed",normal,yes
02,"I was charged twice","NEEDS HUMAN: billing dispute",edge,yes
03,"(empty message)","NEEDS HUMAN: no content",edge,no
04,"URGENT!!! refund now or I sue","NEEDS HUMAN: legal threat",angry,yes
...
20,"Can I change the delivery address?","Explains the 24h window, links the form",normal,yes
pass rate: 18 / 20 = 90 percent (dated 2026-11-03)What goes in the set
- Fifteen ordinary cases, pulled from last month at random so you do not pick only the easy ones.
- Five awkward cases. Missing data, an angry customer, a message in the wrong language, a request outside the rules.
- The correct answer for each, written by the person who owns the process.
One number per system
Pick the number you wrote in chapter 2. Hours per week, minutes per message, days per batch. Write the starting value before you switch the system on. If you skip this, you will never be able to prove the change.
| System | Number | Before | After |
|---|---|---|---|
| Invoice matching | Hours per week | Measure first | Measure after four weeks |
| Support first reply | Minutes to first answer | Measure first | Measure after four weeks |
| Monday report | Hours per week | Measure first | Measure after four weeks |
Leave the cells empty until you have measured. A made-up before number is worse than none.
Show it to the people who pay
One page. The task, the before number, the after number, the pass rate on the golden set, what broke and how you fixed it. Mention what broke. It makes the other numbers believable.
Exercise
Build the golden set
- 1Choose your first live system.
- 2Collect 20 real inputs with the correct output each. Include five awkward ones, such as missing data or an angry customer.
- 3Run the system on all 20. Count the correct ones.
- 4Write the starting number for the metric from chapter 2 and put a date next to it.
- 5Save the sheet next to the brain folder so the next change gets tested against it.
You end with. A sheet with 20 cases, a pass rate, and a dated starting number. Re-run it after every change.
Stuck on this chapter? Talk to an engineer. 30 minutes, you keep the notes either way.