LESSON 6 OF 6
Roll it out without breaking things
BY THE END OF THIS LESSON
Run a pilot that produces a decision rather than an opinion.
Most pilots decide nothing
- Nobody agreed what would be success
- The loudest person decides
- No information
A team tries a tool for a month, and at the end everyone has a vibe about it. Nobody agreed in advance what would count as success, so the loudest person decides. The month produced no information.
Write the success criteria down first
Before starting: what measure, what current value, what threshold, and by when.
"Ticket first-response time, currently 4 hours, we continue if it is under 90 minutes with no drop in satisfaction, measured over four weeks."
If you cannot state the current value, measure that first. A pilot without a baseline can only ever produce anecdotes.
Pilot narrow, then widen
One team, one task, four weeks. Narrow enough that failure is cheap and the cause is obvious.
Include a stop rule — the condition under which you abandon it — and agree it before you start. Written down in advance, it is a sensible decision. Invented afterwards, it is an argument.
WORKED EXAMPLE
VIBES
"Let's try AI in support for a month and see how it goes."
DECIDABLE
Pilot: AI drafts first replies for tier-1 tickets. One team of four, four weeks. Baseline: median first response 4h 10m, CSAT 4.4, agent handles 38 tickets/day. Continue if: first response under 90m, CSAT no lower than 4.3, tickets/day up at least 20%. Stop immediately if: any invented policy reaches a customer, or CSAT drops below 4.1 in any week. Reviewer: Priya reads every draft in weeks 1–2, spot-checks after.
At the end of four weeks this produces a decision that does not depend on who is most persuasive in the room. The stop rule protects the customer relationship while you learn.
YOUR TURN
Design a pilot that produces a decision.
RUN THIS
Help me design a four-week pilot for one AI use case in my business. Ask me what I would measure and what it is today. If I do not know the current number, tell me plainly that I need to measure it before starting rather than helping me around it. Then produce: the scope, the baseline, the continue criteria, the stop rule, and the named reviewer. Use case: [describe it]
What a good result looks like
If you cannot fill in the baseline, that is the finding — measure for a week before you start anything.
KNOWLEDGE CHECK
Answer all 3 correctly to complete this lesson.
1. Why do most AI pilots fail to produce a decision?
2. You cannot state the current value of your chosen measure. What should you do?
3. Why agree a stop rule before the pilot starts?
REMEMBER THIS
Baseline, threshold and stop rule — written down before you start, or the pilot decides nothing.