AI adoption and team training
How to measure an AI pilot before expanding it
An AI pilot should end with a business decision, not a collection of impressive prompts. To learn whether the workflow is worth keeping, compare the whole job before and during the pilot: the time to a usable result, the quality of that result, correction and review effort, exceptions, employee use and actual operating cost. Decide the measures before results are visible so enthusiasm does not move the goalposts.
By SVS Digital · Published
Start here
Choose one recurring workflow, one owner and a decision date. Record a baseline with the same source and method you will use during the pilot. Then track a small set of measures: end-to-end cycle time, accepted quality, rework, completion or error rate, review burden, meaningful employee use, incidents and actual cost. Use representative work, record limitations and have someone outside the implementation team review a sample. Expand only what the evidence supports; otherwise narrow, redesign or stop the workflow.
1. Start with the decision the pilot must support
Write one sentence that names the workflow, the team and the change you expect. “Test AI in sales” is too broad. “Help account managers prepare a checked meeting brief from approved CRM fields and reduce preparation time without increasing factual corrections” is measurable. Name the process owner, the people in the pilot and the date on which the evidence will be reviewed.
Microsoft's adoption guidance recommends connecting the business outcome, department, workflows and metrics before a pilot begins. It also separates usage from business results: an active-user count can show that people opened a tool, but not that the work improved. Keep the first pilot narrow enough that the team can observe the old and new workflow under comparable conditions.
- One workflow with a clear trigger and usable final result.
- One accountable process owner and named pilot users.
- A business question the evidence can actually answer.
- A fixed review date and possible decisions: expand, narrow, redesign or stop.
2. Measure the current workflow before adding AI
Use recent ordinary work, not only the clean examples chosen for a demonstration. For each item, record when work starts, when an acceptable result is ready, how much active employee time it takes, whether it is returned for correction and which exception path is used. Include waiting and handoffs if the goal concerns total turnaround time; include only active work if that is the agreed measure. Write the definition down.
During the pilot, use the same system of record, sampling rule and calculation. Microsoft's impact guidance recommends comparing like with like and using established business systems where possible. A baseline reconstructed from memory after the pilot is weak evidence. If volume, staffing or customer mix changes, record that context instead of attributing every difference to AI.
- Baseline period and representative sample are recorded.
- Start, finish and acceptable result mean the same before and during the pilot.
- The same data source and calculation method are retained.
- Seasonality, staffing and other material changes are noted.
3. Measure the whole workflow, including review and rework
A draft produced in thirty seconds has not saved time if an employee needs fifteen minutes to find invented details, restore missing context and reformat it. Measure until the output is ready for its real next step. Useful operational measures include median cycle time, person-minutes per completed item, substantial-revision rate, correctly completed cases, unresolved exceptions and work returned by the next person in the process.
Pair speed with a quality check. Define what acceptable means for this task: required facts match the source, mandatory fields are present, the tone follows the agreed standard, calculations are verified and prohibited claims are absent. Avoid one universal “accuracy” percentage across unrelated tasks. A meeting summary, a routed enquiry and a customer-facing draft have different failure modes and consequences.
- Time includes checking, correction and the final handoff.
- Quality criteria are specific to the workflow and written in advance.
- Errors and human rescues are counted, not hidden as completed cases.
- Averages do not conceal recurring failures in important case types.
4. Check whether people can use and supervise the workflow
Observe meaningful use: how many eligible work items used the agreed AI-assisted path, where employees abandoned it and why. A low rate may indicate poor training, unclear data rules, extra clicks or a workflow that does not help. A high rate may simply reflect a manager's instruction. Ask users which step changed, which cases they still avoid and whether the tool reduces or moves work to another colleague.
The UK Government's People Factor guide says effective human oversight requires relevant expertise, enough time and authority to challenge an output. Test those conditions directly. Reviewers should have the source material, a short checklist and a safe way to reject or escalate the result. Collect feedback from non-users as well as enthusiasts so the pilot is not designed only around early adopters.
- Eligible items and actual AI-assisted items are both counted.
- Abandoned attempts and reasons are recorded.
- Reviewers have expertise, time, evidence and authority to reject.
- Feedback includes regular users, occasional users and non-users.
5. Keep an evidence log for quality, failures and limits
Save the task version, permitted input type, output, review result, correction category and final outcome for a controlled sample. Do not place confidential or personal material in an evaluation log unless the access, retention and purpose are approved. Record incidents and near misses separately from ordinary quality scores; one unauthorised send or data exposure must not disappear inside a favourable average.
NIST's Measure playbook recommends documented test sets, metrics and evaluation methods, performance checks under conditions similar to deployment, ongoing risk tracking and input from domain experts and users. It also recommends involving internal experts who were not front-line developers or independent assessors where appropriate. For a small pilot, this can mean a team lead outside the build group reviewing a fixed sample against the same rubric.
- Representative ordinary, incomplete and difficult cases are included.
- The tool, instructions and workflow version are recorded.
- Critical incidents remain visible outside aggregate scores.
- A qualified person outside the implementation team reviews a sample.
- Known limits and untested case types are stated in the decision note.
6. Make a proportionate decision and separate facts from projections
At the review, compare the agreed baseline, target and actual result. Expand only the case types, users and permissions that were tested. Narrow the workflow when value appears in ordinary cases but exceptions remain expensive. Redesign when AI makes one step faster but the next bottleneck leaves total turnaround unchanged. Stop when review burden, quality, risk or cost does not justify continued use. A stopped pilot can still be useful if it prevents a larger unsupported rollout.
List actual costs: licenses, setup, integration, training, review, support and maintenance. If you translate time into financial value, use a method your finance team already accepts and label the result as modelled. Microsoft's measurement guidance explicitly separates measured workflow change from estimated financial contribution and warns against attributing the whole business result to AI without evidence. This guide is an operational measurement framework, not a guarantee of return or a substitute for legal, security or sector-specific review.
- Measured result, assumptions and remaining uncertainty are separated.
- Actual pilot costs include employee review and support time.
- The next scope does not exceed the permissions and cases tested.
- The written decision is expand, narrow, redesign or stop—with an owner and next review date.