Piloting an Automation: Running a Test You Can Trust the Result Of

A pilot is meant to answer one question: should we scale. Most pilots in small businesses do not answer it, because they ran on the easy cases, with the builder sitting alongside, and with nothing measured before the start. They end in a good feeling and zero evidence, and then the rollout stalls on exactly what the pilot never tested.

What to choose for a pilot

Not the easy case and not the hard one, but the representative one. If 80% of cases are of one type, that is what runs in the pilot - including a similar proportion of the exceptions that actually appear in it.

Three reasonable ways to bound it:

• By time - every case in two weeks, unfiltered.
• By type - one case type, including its exceptions.
• By person - one user does everything in the new system.

The first is the cleanest, because it allows no selection. Manually filtering what enters the pilot - even in good faith - removes precisely the cases that would have taught you something.

What to measure, and when

The fourth row is the most important and the easiest to miss. An automation that ran successfully but needed somebody to fix three cases a day is not ready to scale, and that number will appear nowhere unless it is counted in real time.

How long to run it

Two weeks is a reasonable floor for a daily process, and a month for a weekly one. The simple rule: the pilot has to include at least one full cycle of everything the process does - including month end, if month end behaves differently.

What happens when you shorten it: seasonal exceptions never appear, the rollout meets them for the first time at full scale, and the conclusion is that the automation "does not work". It did work - it simply was never tested against what happens in the last week of the month.

The mistake that makes a pilot worthless

The builder sits alongside and fixes things in real time. It is natural and damaging: the result reflects an automation with a person attached, which is not the state it will be in at scale.

The fix is simple - split it in two. The first week with support, the second without, recording every time somebody had to step in. The difference between the two weeks is exactly the information the pilot exists to produce, and the logic of running without support is in an automation runbook.

What to record during the pilot

Not a report - one table with a row per case: date, case type, passed or not, and whether somebody had to step in. Four columns filled in as you go, not at the end.

The reason it is recorded live rather than reconstructed is simple: after two weeks nobody remembers how many times exactly they went in to fix something, and an estimate from memory is always lower than the real number. That is precisely the number meant to decide, so it is the one that must not be estimated.

It is worth adding a fifth column for short free text: what looked odd. Those notes are what produce the exception list for scaling, and they are almost always more useful than the numerical summary.

The decision criterion, set in advance

Before starting, write one sentence: "we scale if X". For example - if more than 90% of cases went through without intervention, and if time per case fell by at least a third.

The reason it is written in advance is that judging afterwards is hard. Somebody who has invested two weeks will lean towards scaling even on a mediocre result, and somebody who hit two faults will lean towards rejecting even when everything else worked. A sentence written beforehand removes both biases from the equation.

What do you do when a pilot fails?

First, separate three kinds of failure that look alike: a wrong build, a process that was not ready, or the wrong task choice. The first is fixed in days, the second requires redefining the process, and the third says this task was never a candidate.

The way to tell them apart is to ask what exactly failed. If the system did something wrong - that is the build. If it did exactly what was defined but the outcome did not fit - the definition was incomplete. And if it turned out every case needs different judgement - that is the task, covered in automating the wrong things.

A pilot that failed and was properly scoped is not waste. It cost two weeks and prevented a full project, which is the reason for running a pilot in the first place.

Who needs to know a pilot is running

Inside the business - everyone who touches the process, even those not participating. The reason is practical: when somebody sees a record created differently from usual and does not know why, they usually "fix" it. That correction falsifies the pilot results without anyone knowing.

What to say is short: which process, on which cases, until when, and what to do if something looks odd. That last sentence converts everyone outside the pilot from a source of interference into a source of information.

Outside the business, in most cases there is no need. The exception is when the customer sees something different - a different message, a document in a new format - and then it is better to say so, because a customer noticing an unannounced change assumes something went wrong.

How to scale after a successful pilot

Not all at once. Sensible scaling goes in stages: another case type, another user, another branch - each stage with the same measurement and the same criterion. What changes at scale is usually the volume, and that is what exposes problems the small run never showed.

It is also worth deciding what happens to the old method at each stage. Leaving it open "just in case" extends the period where two methods coexist, which is precisely the state the pilot was meant to end.

Sources

• Business automation for small businesses in Israel
• Make - error handling in scenarios