Adoption telemetry for AI tools: what a trial found

In July 2025, METR published a randomised trial. Sixteen experienced open-source developers worked on 246 issues. Before the trial they expected AI to make them 24% faster. Afterwards they believed it had made them 20% faster. Their tasks took 19% longer with AI allowed. METR now marks that result as out of date.

That gap between what the developers believed and what the trial found is the subject of this article. In my reading, a dashboard built on self-report inherits it, and no cited source shows usage counts predicting a result. Because METR marks the result as out of date, take the lesson as one about method. It is not a lesson about the size of any one effect.

Executives report little effect, and that is also self-report. An NBER working paper issued in February 2026 surveyed nearly 6,000 senior executives in the US, UK, Germany and Australia. It found that 69% of firms use AI. Nine in ten executives reported no impact on employment or productivity over three years.

How far apart can belief and evidence be?

The table sorts six sources by how each one got its result: a trial or study, work-system data, or what people said.

Reported figures disagree. Executives report little impact, while technical workers report large gains. They were asked different questions. Technical workers were asked about their own speed and the value of their work. Executives were asked about employment and productivity over three years.

The first three rows disagree in size and sign, and METR flags both of its own results as limited. In February 2026 METR said it believes developers are likely more sped up in early 2026. It called its own data “only very weak evidence” of the size of the gain. Any sentence that says AI makes people faster or slower ages quickly.

A survey of your own staff belongs in the reported group. METR says survey estimates tend to be larger than field-experiment estimates.

What do the usual adoption panels count?

The panels below are the usual ones in the sources cited here. That is an inference, not a count.

Each tool defines an active user its own way. Figures from two tools do not compare.

Each panel below has a reason it misleads.

DORA’s 2025 report says: “The conversation must shift from adoption to effective use.” DORA’s page on measurement frameworks calls it a “common misconception that logs-based metrics are objective”.

None of the sources cited here shows a usage count predicting a business result. The research behind this article found no independent study that tests whether a usage-based adoption metric predicts outcomes.

What should adoption telemetry measure instead?

This set is a synthesis of the evidence above. No source states it in these words.

Sort every number by where it came from. Logs show what people did. Surveys show what people said. The system where the work is recorded shows what happened there, and a randomised trial compares like against like. No one of these is the whole truth on its own. Six measures hold up better, and each has a limit.

The first two rows are diagnostics. Coverage is a governance question about where AI is used. Paid versus active use and retention show where a tool sits unused, which wastes money and points to a workflow problem. In this set, only the third row can show value. Put the weight on the outcome, delivery health, cost and survey rows, and read the survey as opinion.

The approach page says I check who is still using the system a month after launch. The rule is people active in week four over people active in week one, per team, from gateway logs. Active means two different active days within four weeks. I count people, not accounts.

When a team stops using a tool, I treat it as a workflow question, not a people question.

Delivery health is on the list because DORA’s 2025 report, published by Google Cloud, found AI adoption going with higher delivery throughput and still with higher instability. DORA describes that as a comparison in a snapshot, not an effect.

For the survey, I reuse the items DORA publishes on trust, reliance and AI stance. I use them word for word and credit DORA. I ask once a quarter, and it takes five minutes to answer.

This agrees with the approach page. There, evals in CI, tracing, cost per outcome and a rollback path come before a workflow touches a customer. The measures here add one line. Telemetry shows whether use lasted. An outcome measure shows whether it worked.

These are different jobs. The evals in CI article covers testing a change before it ships. Adoption telemetry asks whether people keep using the system.

How do you tell whether it worked?

Agree the outcome before rollout. Read it from the system that already records it, such as ticketing, CI or your CRM, and not from the AI tool’s own logs. The pilot article covers how to pick an outcome you can measure.

Each use case gets one outcome metric. The client’s owner chooses it and signs it before the start.

The support-agent study in the table read its outcome, issues resolved per hour, from work-system data and not from tool logs. It used a staggered rollout.

Then compare against something. A clean comparison is hard, even for researchers. METR’s 2026 follow-up produced what it called an unreliable signal. METR’s stated reasons include developers choosing not to participate rather than work without AI. It also lists developers becoming more selective about which tasks they submitted.

I roll a tool out in waves by team. The teams still waiting are the comparison group.

What should you check before collecting data about people?

The rules below are general rules for monitoring staff, applied here by analogy. The sources cited here include no regulator text or court decision on AI usage logs.

Start with the risk that is easy to miss. A team count built to plan training can turn into a person-level view used to judge performance. The ICO, the UK regulator, gives a clear example. Data collected to manage workspace capacity should not be re-used for performance management without a new lawful basis and a test of necessity and proportionality. That is UK law, not EU law.

Treat a log that names a person as personal data about staff. The GDPR asks for a stated purpose, no more data than needed and no longer retention than needed (Article 5). Screen the plan for a DPIA first, and read the DPIA entry for when monitoring staff triggers one.

A client’s data protection officer or counsel runs the DPIA screening. I hand them the question set.

Do not rely on staff consent. The Article 29 Working Party’s 2017 opinion said consent is highly unlikely to be a legal basis for data processing at work. The exception is where employees can refuse without adverse consequence. That opinion was written under the old Data Protection Directive, so treat it as persuasive guidance, not current law.

Where a works council or staff body exists, check its rights before you switch anything on.

I take clients with works councils. Engaging the council before switch-on is a standard step.

Keep three things off any dashboard: rankings of named people, prompt content and re-use for performance reviews. The reasons are the ICO example above, the GDPR rule on data minimisation and a warning from DX, a vendor of measurement software. DX warns against using these metrics to judge individuals and says code volume is “particularly susceptible to gaming”.

Claude Code’s documentation says prompt text is off by default. It includes the signed-in user’s email, and the documentation tells operators to filter it at their own backend if that is a concern. That is an inference, not something the documentation says: the choice between named and team data sits at your own collector.

Set a smallest group you will show, so that a count cannot point at one person. Microsoft’s Viva Insights asks administrators to set a minimum group size of at least five. That is a product setting, not a legal figure, and the sources cited here give no legal threshold.

The same page says the end-user opt-out does not apply to Copilot usage data, which stays available at the row level. That suggests a group-size floor limits what a report shows, not what is stored. That is an inference.

Tell staff before the first log is collected. The CNIL says staff must be told beforehand. The ICO says to seek and document workers’ views unless there is a good reason not to. The acceptable use policy article covers the policy itself, not the telemetry behind it.

I show breakdowns by team only, and a group needs at least five people. I collect no prompt content. I tell staff before telemetry starts.

A named-person view is never the default. It exists only for a narrow stated purpose, recorded, approved by two people and kept 30 days.

Where does the dashboard template fit?

The Adoption Telemetry Dashboard Template turns these measures into eight panels. Paid versus active and retention become two panels, and the survey splits into trust and policy clarity. Each panel names the question it answers, the data source and the decision it supports. It is a page to copy into your own tool, not a live dashboard. Version 0.1 covers coding assistants and chat assistants.

The coverage panel reads gateway logs, sign-in app lists, expense lines, API bills and the AI register. The register article shows how to build one. The unapproved tools answer covers what to do about tools staff already use without approval. The gateway config already sets rate limits per team, not per user.

The 90-Day Production Sprint builds a dashboard that shows tracing and cost per outcome side by side. Its page promises no other panels. The fractional CAIO retainer includes a review of every AI initiative, shadow tools included.

What does this article leave out?

It reports no client results and no numbers from my own telemetry. The evidence it cites comes from public studies and vendor documents. Most of the experimental evidence predates agentic tools, so treat each figure as a dated finding.

It does not say which baseline method suits your company or which person-level views to allow. Those depend on your company, your country and your staff representatives. It does not say whether a plain count dashboard is an AI system under the EU AI Act, and no source cited here settles that.

It is a summary of public texts, not legal advice. Take the legal questions to your data protection officer or counsel.