Copilot rollout metrics that survive the CFO meeting
Acceptance rate is a demo metric. What to measure instead when the coding assistant bill comes due, and the baseline you needed last month.
Every copilot vendor will hand you a dashboard, and every dashboard leads with acceptance rate: how often developers keep the suggestions. It is a fine number for a demo and a useless one for a CFO, because accepting a suggestion and shipping better software are different events, sometimes opposite ones. I have watched a team celebrate 38% acceptance while their change failure rate crept up, which is a sentence that should end any acceptance-rate romance.
measure the system, not the keystrokes
The delivery outcomes you already believed in are the right lens; the DORA metrics exist precisely because "activity went up" and "delivery improved" are independent claims. Lead time for changes, deployment frequency, change failure rate, recovery time: if the copilot is working, some of these move within two quarters, on the teams using it, relative to the teams not yet using it. That relative comparison is your honest experiment, and you get it free by rolling out in waves instead of all at once.
Add two AI-specific lines. Time-to-merge on AI-assisted pull requests versus not (your review process will tell you more about your bottleneck than the model will). And rework: how often AI-assisted commits get reverted or re-edited within two weeks. Speed that boomerangs is not speed.
what the numbers looked like in the wild
Without naming clients, the pattern across rollouts I have reviewed is consistent enough to sketch. Acceptance rates start high and drift down as novelty fades, then stabilize somewhere unglamorous; the stable number is the real one, so do not celebrate week two. Time-to-merge improves for small, well-scoped changes and barely moves for the gnarly ones, which surprises nobody who writes code and disappoints everyone who bought seats expecting uniform lift. And the spread between heavy users and non-users within the same team is usually wider than the spread between teams, which means the interesting question is rarely "does the tool work" and usually "what are the heavy users doing differently", answerable by asking them, which is somehow the least-used research method in the enterprise.
Expect a J-curve in the first weeks and say so in advance. People learning where the tool helps are slower before they are faster, and a dashboard read naively at week three will kill a rollout that week nine would have vindicated. Publishing the expected shape, dip included, is the difference between "this is failing" and "this is on schedule".
the meeting where the metrics earn their keep
Bring the panel to the monthly budget conversation as one page: the three engineering numbers, the two AI lines, the baseline deltas, and one sentence of interpretation per line. Resist the dashboard with forty tiles; executives fund trends they can retell at their own next meeting. The page format also disciplines the program itself, because a metric that cannot earn a line on the page is instrumentation, not measurement, and instrumentation without a decision attached is a hobby.
the part everyone skips, then regrets
All of this requires a baseline taken before the rollout, and almost nobody takes one, because the pilot enthusiasm arrives before the measurement discipline does. Take the baseline first. The copilot conversation in month nine is either "here is the delta" or it is interpretive dance, and the CFO has seen the dance before.