Fractional Chief AI Officer for 30–300 person companies: AI governance that passes procurement and agentic automation shipped to production.https://meshin.co/Fri, 02 Oct 2026 00:00:00 GMThttps://meshin.co/blog/adoption-telemetry-measuring-if-ai-is-actually-usedhttps://meshin.co/blog/adoption-telemetry-measuring-if-ai-is-actually-usedAdoption telemetry for AI tools should show whether use lasted and whether an agreed outcome moved. In METR's 2025 trial, developers believed AI made them 20% faster, while their tasks took 19% longer. METR now marks that result out of date. Count use by team, then read one outcome per use case against a baseline or comparison group.Fri, 02 Oct 2026 00:00:00 GMTIn July 2025, METR published a randomised trial. Sixteen experienced open-source developers worked on 246 issues. Before the trial they expected AI to make them 24% faster. Afterwards they believed it had made them 20% faster. Their tasks took 19% longer with AI allowed. METR now marks that result as out of date. That gap between what the developers believed and what the trial found is the subject of this article. In my reading, a dashboard built on self-report inherits it, and no cited source shows usage counts predicting a result. Because METR marks the result as out of date, take the lesson as one about method. It is not a lesson about the size of any one effect. Executives report little effect, and that is also self-report. An NBER working paper issued in February 2026 surveyed nearly 6,000 senior executives in the US, UK, Germany and Australia. It found that 69% of firms use AI. Nine in ten executives reported no impact on employment or productivity over three years. How far apart can belief and evidence be? The table sorts six sources by how each one got its result: a trial or study, work-system data, or what people said. | Source and date | Kind | Result | How far to trust it | | ------------------------------------------------------------------ | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | METR randomised trial, 10 July 2025 | Randomised trial | Tasks took 19% longer with AI allowed. Developers had expected a 24% speed-up and afterwards believed 20%. 16 developers, 246 issues. | Small, with early-2025 tools. METR now marks the result out of date. | | METR follow-up, 24 February 2026 | Follow-up study | Raw estimates: returning developers 18% faster, new developers 4% faster. Both intervals include a slowdown. 57 developers, 800 or more tasks. | METR calls it "an unreliable signal". Many developers declined to include tasks they did not want to do without AI, which likely biases the estimate downward. METR calls its data "only very weak evidence" of the size of the gain. | | Brynjolfsson, Li and Raymond, Quarterly Journal of Economics, 2025 | Measured from work-system data | Issues resolved per hour rose 15% on average across 5,172 support agents. Less experienced agents improved on speed and quality. Gains for the most experienced agents were small. | Journal article with a staggered rollout. One setting, customer support. Figures are from the abstract. | | METR survey, 11 May 2026 | Reported by people | 349 technical workers. Median self-reported speed-up of three times, and 1.4 to 2 times in the value of their work. | Convenience sample, with a response rate near 2% for emailed invitations. METR says survey estimates tend to be larger than field-experiment estimates. | | NBER working paper 34836, February 2026 (revised March) | Reported by executives | Nearly 6,000 senior executives in the US, UK, Germany and Australia. 69% of firms use AI. Nine in ten report no impact on employment or productivity over three years. | Executive self-report. Figures are from the abstract. | | DORA report, September 2025 | Reported by people | Survey of nearly 5,000 people. More than 80% say AI raised their productivity. | A snapshot. DORA describes comparisons, not effects. Published by a vendor (Google Cloud). | Reported figures disagree. Executives report little impact, while technical workers report large gains. They were asked different questions. Technical workers were asked about their own speed and the value of their work. Executives were asked about employment and productivity over three years. The first three rows disagree in size and sign, and METR flags both of its own results as limited. In February 2026 METR said it believes developers are likely more sped up in early 2026. It called its own data "only very weak evidence" of the size of the gain. Any sentence that says AI makes people faster or slower ages quickly. A survey of your own staff belongs in the reported group. METR says survey estimates tend to be larger than field-experiment estimates. What do the usual adoption panels count? The panels below are the usual ones in the sources cited here. That is an inference, not a count. Each tool defines an active user its own way. Figures from two tools do not compare. | Source | What counts as active | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | GitHub Copilot usage metrics | At least two active days in the trailing 28 to be placed in a usage phase. GitHub says "Passive" is not a measure of inactivity. | | Microsoft 365 Copilot Dashboard | Usage levels over a rolling 12 weeks. A returning user has at least one action in both the current and the previous period. | | DORA 2025 report | No log rule. Adoption is a survey factor made of reliance, trust and reflexive use. | Each panel below has a reason it misleads. | Measure | What it counts | Why it misleads | | -------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Seats or licences | Money spent. | In GitHub's reports, seat data is not included. It sits in a separate user-management API. | | Daily or weekly active users | One event in a window. | The window and the event differ by vendor, as the table above shows. | | Usage frequency, prompt counts | How often. | DORA's 2024 report, also published by Google Cloud, said measuring usage frequency is "likely not as meaningful as measuring reliance". | | Lines of code accepted, pull requests merged | Output volume. | GitHub calls its lines figure directional. A preprint on Cursor, covering open-source projects, found a short jump in lines added and lasting rises in static-analysis warnings and complexity. Faros, a vendor, found that teams with high adoption completed 21% more tasks and merged 98% more pull requests, with 91% longer review time and no significant link to company-level improvement. Its data is observational. | | Acceptance rate | Suggestions taken. | In a 2022 preprint on Copilot users, it tracked how productive users said they felt, rather than how long the accepted code lasted. | | Vendor hours saved and ROI | Action counts times fixed minutes times a wage. | Microsoft builds assisted hours from action counts, fixed minutes per action, and meeting recaps counted by meeting length. A default hourly rate then values them. GitHub says its ROI figures are estimates rather than exact financial results. | | Self-reported speed-up | Opinion. | METR says that in its 2025 trial people overestimated the effect on their task time by 40 percentage points. METR now marks that result out of date. | | Vendor resolution or deflection | Conversations that ended. | Intercom counts a customer who leaves without asking again as resolved, and charges for it, even if the customer is unhappy, unless its agent escalates. | | Adoption multiplier | Engaged users against passive users. | GitHub divides the average merged pull requests of engaged users by that of passive users. That compares people who chose the tool with people who did not, so it cannot isolate the tool's effect. That is an inference, not a GitHub statement. | DORA's 2025 report says: "The conversation must shift from adoption to effective use." DORA's page on measurement frameworks calls it a "common misconception that logs-based metrics are objective". None of the sources cited here shows a usage count predicting a business result. The research behind this article found no independent study that tests whether a usage-based adoption metric predicts outcomes. What should adoption telemetry measure instead? This set is a synthesis of the evidence above. No source states it in these words. Sort every number by where it came from. Logs show what people did. Surveys show what people said. The system where the work is recorded shows what happened there, and a randomised trial compares like against like. No one of these is the whole truth on its own. Six measures hold up better, and each has a limit. | Measure | Question it answers | What it cannot tell you | | ----------------------------------------- | ------------------------------------------------------------------------------------- | -------------------------------------------- | | Coverage | How much AI use at work runs through approved tools? | Whether those tools help. | | Paid versus active, and retention by team | Which seats sit unused, and do people who tried a tool keep using it? | Whether the tool works. | | One outcome per use case | Did the outcome agreed before rollout move, against a baseline or a comparison group? | Why it moved, if there is no comparison. | | Delivery health (engineering use cases) | Did throughput or instability get worse as use grew? | Whether the use case pays off. | | Cost per outcome | What does one unit of the outcome cost? | Whether it is the right outcome. | | A short repeated survey | Do people trust the output, rely on it, and say where it gets in the way? | Whether work got faster. Read it as opinion. | The first two rows are diagnostics. Coverage is a governance question about where AI is used. Paid versus active use and retention show where a tool sits unused, which wastes money and points to a workflow problem. In this set, only the third row can show value. Put the weight on the outcome, delivery health, cost and survey rows, and read the survey as opinion. The approach page says I check who is still using the system a month after launch. The rule is people active in week four over people active in week one, per team, from gateway logs. Active means two different active days within four weeks. I count people, not accounts. When a team stops using a tool, I treat it as a workflow question, not a people question. Delivery health is on the list because DORA's 2025 report, published by Google Cloud, found AI adoption going with higher delivery throughput and still with higher instability. DORA describes that as a comparison in a snapshot, not an effect. For the survey, I reuse the items DORA publishes on trust, reliance and AI stance. I use them word for word and credit DORA. I ask once a quarter, and it takes five minutes to answer. This agrees with the approach page. There, evals in CI, tracing, cost per outcome and a rollback path come before a workflow touches a customer. The measures here add one line. Telemetry shows whether use lasted. An outcome measure shows whether it worked. These are different jobs. The evals in CI article covers testing a change before it ships. Adoption telemetry asks whether people keep using the system. How do you tell whether it worked? Agree the outcome before rollout. Read it from the system that already records it, such as ticketing, CI or your CRM, and not from the AI tool's own logs. The pilot article covers how to pick an outcome you can measure. Each use case gets one outcome metric. The client's owner chooses it and signs it before the start. The support-agent study in the table read its outcome, issues resolved per hour, from work-system data and not from tool logs. It used a staggered rollout. Then compare against something. A clean comparison is hard, even for researchers. METR's 2026 follow-up produced what it called an unreliable signal. METR's stated reasons include developers choosing not to participate rather than work without AI. It also lists developers becoming more selective about which tasks they submitted. I roll a tool out in waves by team. The teams still waiting are the comparison group. What should you check before collecting data about people? The rules below are general rules for monitoring staff, applied here by analogy. The sources cited here include no regulator text or court decision on AI usage logs. Start with the risk that is easy to miss. A team count built to plan training can turn into a person-level view used to judge performance. The ICO, the UK regulator, gives a clear example. Data collected to manage workspace capacity should not be re-used for performance management without a new lawful basis and a test of necessity and proportionality. That is UK law, not EU law. Treat a log that names a person as personal data about staff. The GDPR asks for a stated purpose, no more data than needed and no longer retention than needed (Article 5). Screen the plan for a DPIA first, and read the DPIA entry for when monitoring staff triggers one. A client's data protection officer or counsel runs the DPIA screening. I hand them the question set. Do not rely on staff consent. The Article 29 Working Party's 2017 opinion said consent is highly unlikely to be a legal basis for data processing at work. The exception is where employees can refuse without adverse consequence. That opinion was written under the old Data Protection Directive, so treat it as persuasive guidance, not current law. Where a works council or staff body exists, check its rights before you switch anything on. | Country | What the text says | How far to trust it | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Germany | Works Constitution Act, section 87(1) no. 6: the works council has a say over technical devices meant to monitor behaviour or performance. The Federal Labour Court (1 ABR 20/21, 8 March 2022) reads this as objective suitability: a device counts if it can collect such data, whatever the employer intends. That case concerned Microsoft Office 365, where the parties did not dispute that the data could be used to check behaviour or performance. Applying it to AI usage logs is an inference. | Statute and court decision, both read in the original. | | Netherlands | Works Councils Act, article 27(1): the works council must consent to rules on the processing and protection of staff personal data. It must also consent to provisions aimed at or suitable for checking presence, behaviour or performance, where they cover all or a group of staff. | Secondary copy of the statute. The official text was not read. | | France | CNIL: a monitoring device must pass tests of justification and proportionality, go to staff representatives (the CSE, at employers with 50 or more staff) and be announced to staff beforehand. | Regulator guidance, dated 9 July 2026, read in the original French and paraphrased here. | I take clients with works councils. Engaging the council before switch-on is a standard step. Keep three things off any dashboard: rankings of named people, prompt content and re-use for performance reviews. The reasons are the ICO example above, the GDPR rule on data minimisation and a warning from DX, a vendor of measurement software. DX warns against using these metrics to judge individuals and says code volume is "particularly susceptible to gaming". Claude Code's documentation says prompt text is off by default. It includes the signed-in user's email, and the documentation tells operators to filter it at their own backend if that is a concern. That is an inference, not something the documentation says: the choice between named and team data sits at your own collector. Set a smallest group you will show, so that a count cannot point at one person. Microsoft's Viva Insights asks administrators to set a minimum group size of at least five. That is a product setting, not a legal figure, and the sources cited here give no legal threshold. The same page says the end-user opt-out does not apply to Copilot usage data, which stays available at the row level. That suggests a group-size floor limits what a report shows, not what is stored. That is an inference. Tell staff before the first log is collected. The CNIL says staff must be told beforehand. The ICO says to seek and document workers' views unless there is a good reason not to. The acceptable use policy article covers the policy itself, not the telemetry behind it. I show breakdowns by team only, and a group needs at least five people. I collect no prompt content. I tell staff before telemetry starts. A named-person view is never the default. It exists only for a narrow stated purpose, recorded, approved by two people and kept 30 days. Where does the dashboard template fit? The Adoption Telemetry Dashboard Template turns these measures into eight panels. Paid versus active and retention become two panels, and the survey splits into trust and policy clarity. Each panel names the question it answers, the data source and the decision it supports. It is a page to copy into your own tool, not a live dashboard. Version 0.1 covers coding assistants and chat assistants. The coverage panel reads gateway logs, sign-in app lists, expense lines, API bills and the AI register. The register article shows how to build one. The unapproved tools answer covers what to do about tools staff already use without approval. The gateway config already sets rate limits per team, not per user. The 90-Day Production Sprint builds a dashboard that shows tracing and cost per outcome side by side. Its page promises no other panels. The fractional CAIO retainer includes a review of every AI initiative, shadow tools included. What does this article leave out? It reports no client results and no numbers from my own telemetry. The evidence it cites comes from public studies and vendor documents. Most of the experimental evidence predates agentic tools, so treat each figure as a dated finding. It does not say which baseline method suits your company or which person-level views to allow. Those depend on your company, your country and your staff representatives. It does not say whether a plain count dashboard is an AI system under the EU AI Act, and no source cited here settles that. It is a summary of public texts, not legal advice. Take the legal questions to your data protection officer or counsel.posthttps://meshin.co/blog/evals-in-ci-for-llm-agentshttps://meshin.co/blog/evals-in-ci-for-llm-agentsWithout a golden answer, evals in CI score agent output with a rubric-based judge, a groundedness check, or a comparison against the last good version. Most agent tasks have no single right answer. A fixed test set runs on every change, and a regression then fails the build like a broken unit test. Track the pass rate over time.Tue, 08 Sep 2026 00:00:00 GMTAn eval suite that only runs on a laptop protects nobody. The same regression a developer catches by eye on a good day slips through on a bad one. A model provider's silent update slips through every time. Putting an eval suite in CI turns a habit into a gate. The pull request cannot merge until the numbers clear, whoever is watching. How do you build an eval set from production traffic? Pull the first cases from what the agent has actually handled, not from what a spec said it should handle. Real traces carry the phrasing, the edge cases, and the odd multi-step requests. A person writing test data from imagination never thinks to include them. A support agent's eval set should read like a week of its own transcripts, because that is exactly what it is. Sample across a period long enough that one good or bad week does not decide the result. A month of traffic spans weekdays, weekends, and spikes the business already expects, which is a sturdier base than one convenient week. Mix the set on purpose. Include three kinds of case: routine ones that should always pass, a handful known to be hard, and a few with no single acceptable answer. Most agent tasks in production fall into that last kind. Store the set next to the code it tests, in a format a reviewer can read in a pull request. Change it through the same review a code change gets. Revisit the set on a fixed schedule rather than whenever someone remembers. Retire a case once the product has genuinely outgrown it. Add one the day a new failure reaches production. That keeps the set tracking how people actually use the agent. How do you score an agent when there is no golden answer? Most agent tasks do not reduce to one correct string, so do not force them into that shape. A support reply, a summary, or a multi-step tool sequence can be right in more than one way. A scorer built for exact match will reject good answers as often as it catches bad ones. Score in layers instead, cheapest and most reliable checks first. Start with deterministic checks on anything mechanical. Check whether the agent called the right tool, with a valid argument set, in a shape that downstream code can actually parse. These checks are cheap, and they never disagree with themselves. In most agentic systems, they catch a large share of regressions before a judge ever runs. OWASP's list of risks in large language model applications names excessive agency as one of the standard failure modes worth testing for. Excessive agency means an agent taking an action its task never asked for. Write a case for every tool the agent can call. Each case should check that the tool stays quiet when not needed, and that it fires correctly when needed. For the answer's content, add a reference-free judge. Look at the question and whatever the agent actually retrieved. Then check whether every claim in the answer traces back to a document it had in hand. This catches a fluent, confident answer that is not supported by anything the system looked at. A plain accuracy score misses that failure mode entirely. Where no rubric fits cleanly, fall back to a comparison against the last known-good version. Show a judge both outputs, blind to which one is new, and ask it to pick a winner. A judge finds picking a winner easier than defending an absolute score out of some invented scale. A win rate against a fixed baseline is also a steadier number to watch than a score with no anchor. NIST's AI RMF frames this as a selection problem under its Measure function. Which method fits depends on what is actually being measured. A team documents why it chose a metric, and which ones it rejected. That team ends up with a sturdier suite than one that picked whatever was easiest to compute. What happens when the judge itself is flaky? A judge model can score the same input differently on two runs, especially at a higher temperature. That alone will make a pipeline look unstable, even when the agent under test has not changed. Set the judge's own temperature as low as the provider allows. Treat any remaining run-to-run disagreement as a property of the judge to manage. Calibrate the judge on a fixed schedule against a small set of cases a human has already scored. The number that matters is agreement between the judge and the human reviewers. A judge that is consistent but consistently wrong is worse than one that is honestly uncertain. Consistent wrongness hides the problem instead of showing it. Let the judge decline a case it cannot confidently assess. A skipped case should fall out of the pass-rate calculation, rather than count as either a pass or a fail. A case the judge skips often is telling you the rubric does not cover that situation yet. When one specific case flips between pass and fail on identical re-runs, treat that case as broken, not the whole gate. Quarantine it. Keep it running so you can see it, stop it from blocking a merge, and put a name against fixing its rubric or its sampling. Only then should it rejoin the set that actually gates a build. What should the pipeline check, and what does a failing check do to the build? Not every check should carry the same weight. A wrong tool call and a slow response are both real problems, but only one of them should stop a release. The table below is the shape of a working gate. It shows what each row measures, how it gets a number, and what that number does to the build when it fails. | What to score | How to score it | What a failing score does to the build | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | Task success on labelled cases | Exact match or a deterministic rule against a known-good answer. | Blocks the merge immediately, with no override. | | Task success with no golden answer | A rubric-based judge scores the answer against stated criteria, not against a reference. | Blocks the merge once the pass rate drops below the floor set for that task. | | Faithfulness to retrieved context | A reference-free judge checks whether each claim traces back to a document the agent actually retrieved. | Blocks the merge; an unsupported claim counts as a defect, not a style note. | | Preference against the last good version | Pairwise comparison: the judge sees both outputs, blind to which is new, and picks a winner. | Blocks the merge if the new version loses on more cases than the agreed margin allows. | | Tool-call correctness | Deterministic check that the agent called the right tool with a valid, schema-checked argument set. | Blocks the merge; a malformed call fails loudly instead of degrading quietly. | | Judge-human agreement | Re-run the rubric against a human-labelled sample on a fixed schedule and compare agreement. | Does not block a single pull request; a drop pages the eval owner to fix the rubric. | | Cost per completed run | Sum of model and tool cost across the eval set, divided by cases completed. | Blocks the merge once the run exceeds the ceiling set in the pipeline config. | | Response time under load | Wall-clock time per case, measured in the same job as the scores. | Warns on the pull request; blocks only if it breaches the budget agreed with the product owner. | Where does the cost ceiling sit? Put a cost ceiling in the same job as the quality score. Do not put it on a separate dashboard someone checks once a week. Agent systems fail commercially long before they fail technically. Consider a tool call that retries a few times on a bad response: harmless in a demo, ruinous once real traffic hits it. Measure cost per completed run, across the whole eval set, the way you measure the pass rate. A change that makes the agent slower or more expensive without a real gain is a regression, even if every other score stays green. A 90-Day Production Sprint puts this gate in place before a workflow takes its first real user. That is a cheaper time to find the problem than after the first invoice. What does the pipeline do when a check fails? A failing check blocks the merge by default. The pull request gets a summary next to the diff that caused it. That summary lists which cases failed, what the agent returned, and what the check expected. Nobody reconstructs the failure from a log. Not every failure deserves the same response. A wrong tool call or an unsupported claim should stop the release outright. A latency or cost warning close to its budget, or a drop in judge-human agreement, can page an owner instead, without stopping every other change in the queue. Keep that distinction explicit in the pipeline config, not in someone's memory of which checks are actually load-bearing. Only a named reviewer, not the author of the change, can override a hard block. That override has to write itself into the same log the eval results live in. An override that leaves no trace is indistinguishable from no gate at all. It gets used that way the next time someone is in a hurry.posthttps://meshin.co/blog/handing-over-ai-leadership-to-an-internal-ownerhttps://meshin.co/blog/handing-over-ai-leadership-to-an-internal-ownerHand over AI leadership in steps that end in a test. My method takes three months and ends with three checks: the new owner writes an eval case, runs a rollback rehearsal and handles one incident without me. No engagement has yet ended in a full handover by this method. Searches in October 2026 found no AI-handover standard.Fri, 02 Oct 2026 00:00:00 GMTAn outside AI lead sets up governance or ships a first system. Sooner or later the role has to sit with someone on the payroll. Four questions run through a handover: what moves, who takes the role, who signs what, and how the company knows it worked. Is there a standard way to hand over AI leadership? None turned up. In October 2026, searches of standards bodies, regulators and journals found no standard, law or study on this exact move. What exists sits next door. Interim and fractional executives have exit practices. Engineering teams have ways to transfer ownership of a service. A UK financial regulator has rules for handing over a senior role. The NIST AI RMF and the EU AI Act texts checked in October 2026 never mention a change of person. The evidence is thin. One academic model of interim assignments says the evidence base on interim effectiveness lacks empirical studies. A company linked to two of its authors funded its literature search. Most checklists come from firms that sell interim or fractional staff. Read what follows as practice that others found sensible, plus some points that have no outside source. None of it is proven. What has to move in a handover? Documents alone can fall short. Esya, a UK studio that builds AI inside client companies, writes that a folder of documents can be complete while the handover is not. It says the receiving team also needs the route to deploy and controlled access to systems. It needs monitoring data, the evals and the runbook. It needs authority over incidents and practised knowledge. Esya sells related work, so read that as a peer's view. The table shows what each source asks for, and how much weight it can bear. Six rows come from sources about handovers, exits or onboarding. The three NIST rows are marked. The NIST text checked in October 2026 never mentions a change of person, so applying it to a handover is this page's reading. | What moves | What a source asks for | The source and its weight | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Decisions and reasons. | Handover material should include judgement and opinion, not just facts and figures. | UK Financial Conduct Authority handbook. Regulator guidance for UK financial firms, so an analogy only. | | Work in progress. | A status report on the goals set at the start: what got done, what remains and what new issues emerged. | CompassPoint. A practitioner's guide to interim directors of US nonprofits. | | The list of AI systems. | Mechanisms to inventory AI systems. | NIST AI RMF, Govern 1.6. A voluntary framework that NIST says it is revising. Not about handovers: this row is the page's reading. | | Roles and authority. | Roles, responsibilities and lines of communication for mapping, measuring and managing AI risks are documented and clear to individuals and teams throughout the organisation. | NIST AI RMF, Govern 2.1. Voluntary. Not about handovers: this row is the page's reading. | | How to run and change the system. | The route to deploy, monitoring data, the evals, a runbook and authority over incidents. | Esya. A studio that sells related work. | | Access. | Revoke access in one coordinated step on or before the last day, and confirm it in writing. | Fractionus. A marketplace that sells fractional executives, so one vendor's opinion. | | Monitoring and incident duty. | Monitoring plans that cover incident response, recovery and change management. | NIST AI RMF, Manage 4.1. Voluntary. Not about handovers: this row is the page's reading. | | Someone to ask. | The development team must be available to back up and advise the receiving team while it settles. | Google's SRE book. The operator's own account of its engineering teams. | | A report on how it went. | An exit review that names the issues the client needs to address. | Institute of Interim Management code. A professional body's code for UK interim managers. | Nine rows are a lot to move on one day. Google's book describes a progressive transfer of responsibilities and ownership, including parts of operations, the change management process and access rights. This page reads that as support for moving a handover in steps. My access list has six categories: the AI register, the LLM gateway, CI/CD, the cloud consoles, the vendor panels and the SSO admin. The client's internal owner or IT lead revokes each one. Revocation is confirmed in writing the same day. A revoke-all line sits on the last-day checklist. Who should take the role? The company side needs one named person with the authority to pause the system. The pilot article says the same. It adds that the person should have written at least one eval case and watched a rollback rehearsal. Someone who was not part of the build inherits a document, not an understanding. The usual successor is the tech lead or the head of data. A COO takes the role when AI sits deep in company processes. One person holds the role. This site's own pages point to two jobs inside the role. One is answering for risk, including the authority to stop a launch. The other is keeping the system working, including the eval case and the rollback rehearsal. One person holds both jobs. NIST's playbook suggests managing development apart from testing, so that someone can correct a system independently of its builders. That is about testing, not owners, but it shows why a company might split the jobs. A US federal memo from April 2025 makes a different choice. The memo from the Office of Management and Budget lets an agency head name an existing officer as chief AI officer. The chief information officer is one example, if that person has significant expertise in AI. The memo binds US federal agencies only, so for a private company it is an analogy. It also requires agencies to tell the Office of Management and Budget within thirty days when the officer changes or the post is vacant. Later revisions were not checked. Sometimes nobody inside can take the role yet. The academic paper cited earlier notes that recruiting a senior executive can take six months or more. That figure is not specific to AI roles. No competence framework for a chief AI officer or head of AI from a standards body or regulator turned up in the search. The test the new owner must pass has to be written locally. How do you know the handover worked? Test behaviour, not paperwork. Google's SRE book says a production readiness review is considered a prerequisite for a site reliability team to accept responsibility for the production side of a service. So the receiving side sets a bar before it takes ownership. That is a reading of the text. A specialist team can refuse a service that fails the review. A single internal owner may lack that power, so agree the check with the outside lead in advance. Esya's test is short: the receiving team has practised the response and can make the next change without waiting for the builders to return. Some checks already appear on this site. The pilot article says the new owner should have written at least one eval case and watched a rollback rehearsal. The approach page lists a related measure under engineering capability. It tests whether your engineers can already roll back a bad model without waiting on the outside lead. Three checks close my handover, and I judge each one. The new owner writes an eval case. The new owner runs a rollback rehearsal. The new owner handles one incident without me in the room. Write the full list of checks with the outside lead before the handover starts. For each check, name who judges it and what counts as a pass. CE Interim, which supplies interim managers, lists three causes of failed handovers. The first is making changes without enough involvement of the internal team. The second is treating knowledge transfer as a final step. The third is relying too heavily on the interim as the central decision maker. It adds that a handover is a confirmation, not a critical moment, when the organisation already runs on its own before the interim leaves. Read it as a provider's marketing page. Esya also lists promises that are not enough on their own, such as "contact the builder", "documentation has been shared" and "support remains available". One more check comes later. Fractionus suggests asking the successor at the thirty-day mark what they wish they had known on day one. It is one vendor's suggestion. Who signs what after the handover date? Every decision that carries a signature needs a name on the handover date. Examples are risk acceptances, policy exceptions, vendor approvals, changes to eval thresholds, rollback calls and answers to customer security questionnaires. Write the old signer and the new signer next to each one, with the date the change takes effect. Customer paperwork that names the outside lead needs a new name, and someone has to tell the customer. None of the sources in the table above lists who may sign what, so this part is the page's own advice. Until the handover date, I sign risk acceptances and the customer security questionnaires that carry my name. From that date, the internal owner signs. The client tells its own customers, and I provide the letter template. The FAQ on how AI risk is recorded and signed off explains how AI risk is recorded and how the Retainer signs each entry while it is running. Accountability stays with the company. The NIST AI RMF says executive leadership takes responsibility for decisions about AI risk. The EU AI Act puts the duties of a deployer on the body that uses an AI system under its authority. On that reading, an adviser's departure moves no deployer duty, because the duty never sat with the adviser. It is a reading of the text and not legal advice, so ask counsel before relying on it. Do the AI rules say who must take over? None of the AI rules checked names a successor. The Commission was asked whether an AI officer is necessary. For the AI literacy duty, its guidance says no specific governance structure is mandated and no certificate is needed. That guidance is not binding law. Article 26 covers high-risk systems. It asks deployers to give human oversight to people who have the competence, training and authority for it, and the support they need. None of the articles checked asks for an AI officer. The duties in Article 26 start on 2 December 2027 for systems that are high-risk under Annex III and on 2 August 2028 for those under Annex I. Regulation (EU) 2026/1744 set those dates in July 2026. Most high-risk systems already on the market before those dates come under the duties only if their design changes significantly. Those intended for use by public authorities must comply by 2 August 2030 in any case. Whether a system counts as high-risk is a question for counsel. The NIST AI RMF is voluntary. Its Govern function asks for clear, documented roles and for training for the people in them. It also says executive leadership takes responsibility for risk decisions. NIST says it is revising the framework, so check the current text. The contents of ISO/IEC 42001 list roles, responsibilities and authorities, and competence. Its full text is paid, so nothing from it is quoted here. None of the NIST or EU texts checked says what to do when the person changes, and the ISO text was not read in full. A handover is where role clauses like these get tested. That is a reading, not a rule. How long does a handover take? The sources that say how long a handover takes disagree, and none is about AI. Lengths run from a few weeks to a few quarters, depending on the setting. CompassPoint's guide for nonprofits describes a short overlap, then a few weeks of drop-in help. Fractionus suggests starting weeks before the last day. Google describes quarters for engineering teams taking over a service. My handover takes three months, in three phases. In months one and two, the new owner and I work side by side. In month three, the new owner leads and I review. At the end of month three come the final test and the signatures. The end date is fixed in the month-one plan, not the contract. A fixed date and a ready receiver can pull against each other. Write down both, and agree who may move the date. The Institute of Interim Management code of conduct asks members to agree formal termination terms with the client. These include a handover period and a period of notice. Sources agree that some reachable help after the last day is useful, and differ on how much. Google's book says the development team must stay available to advise the receiving team while it settles. CompassPoint's guide suggests drop-in days, then availability by phone as needed. Fractionus suggests optional advisory time after the exit, though its own page gives two different windows. Whatever is agreed, write down the length, the scope and who may call. For thirty days after the last day, I answer questions by email within one working day. I hold one review call on day 30. Anything beyond that is a separate agreement. What this does not solve The sources are neighbouring fields, vendor advice and voluntary frameworks. No outcome study of AI-leadership handovers turned up, so nothing here shows that a practice leads to better results later. No engagement has yet ended in a full handover by this method. It is not legal advice. The readings of the EU AI Act here are plain-language summaries and need counsel before anyone relies on them. It does not cover the case where nobody inside can take the role. That calls for a hiring or reassignment decision, and no handover method replaces it. The Fractional CAIO Retainer is built to end with a named internal owner and written decisions. The FAQ on how the Retainer ends and the approach page say more. The Handover Playbook turns these questions into six tables to fill in.posthttps://meshin.co/blog/how-to-choose-an-ai-operating-model-for-a-mid-size-companyhttps://meshin.co/blog/how-to-choose-an-ai-operating-model-for-a-mid-size-companyNo law or standard reviewed here requires a mid-size company to use a committee, a hub-and-spoke model or a Chief AI Officer. For high-risk systems the duties point to roles and records: overseers with competence, training and authority, an accountability framework, and logs. Choose the smallest structure that covers them, and change it on evidence from your own company.Fri, 02 Oct 2026 00:00:00 GMTThis page starts from what is required, then asks what the evidence supports. The AI operating model here is the one the glossary defines. What the law requires about structure is narrow, and the dates are still moving. What the evidence supports is long and weak. Does any law require an AI committee or a Chief AI Officer? For a private company, no source reviewed for this page requires any of four things. They are a committee, a centre of excellence, a hub-and-spoke model and a Chief AI Officer. The European Commission's AI literacy Q&A asks whether an AI officer is necessary, as under the GDPR. It also asks whether an organisation must set up an AI governance board. The answer is "No, no specific governance structure is mandated to comply with article 4 of the AI Act." That answer covers Article 4 only. The one binding rule found for this page that does require such structures is American. OMB's memo on federal AI use, dated 3 April 2025, tells each federal agency to name a Chief AI Officer. Each agency covered by the CFO Act must also convene an AI governance board. Even there, agencies "are permitted to rely on existing governance bodies." The memo binds US federal agencies and nobody else. The standards read the same way. The NIST AI RMF is voluntary. Its published text calls it "use-case agnostic, providing flexibility to organizations of all sizes." It prescribes no committee or team shape. ISO/IEC 42001 applies to "any organization, regardless of size." Its contents list roles, responsibilities and authorities. The standard is sold, not published, so its clause text was not read for this page. What does the law require instead? In the Articles reviewed here, the EU AI Act asks for duties, not structures. Which duties apply depends on what a company does with AI. It also depends on whether the company is the provider or the deployer of each system. The table lists the duties of the Articles reviewed, by trigger, using the consolidated text. The Digital Omnibus, Regulation (EU) 2026/1744, rewrote some of the wording and moved the dates in July 2026. The last column is a reading of what each duty needs in practice. It is not law text. | If your company… | The duty | Applies from | What it needs in practice | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Provides or deploys any AI system | Article 4: take measures to support the AI literacy of staff and others who use AI for you. No specific level is required. | In application since 2 February 2025. National authorities enforce from 2 August 2026. | Someone to run the measures. The Commission says there is no need for a certificate and that organisations can keep an internal record of trainings. | | Runs a chatbot or generates synthetic content | Article 50: transparency duties for providers and deployers of chatbots, generative systems and deepfake content. Providers of generative systems must mark outputs in a machine-readable format. | 2 August 2026. Generative systems placed on the market before 2 August 2026 have until 2 December 2026 to mark. | An owner for disclosure wording, and for marking if you provide the system. | | Provides a high-risk AI system | Article 17: a documented quality management system with an accountability framework. It sets out the responsibilities of management and other staff. It must fit the size of the provider. | 2 December 2027 for Annex III systems. 2 August 2028 for Annex I systems. | A written accountability framework. | | Deploys a high-risk AI system | Article 26: give human oversight to people with "the necessary competence, training and authority". Also monitor the system, keep logs for six months or more, and report risks. | The same two dates. | Named overseers. The Act leaves the deployer "freedom to organise its own resources and activities". | | Changes a system someone else built | Article 25: a deployer becomes the provider if it puts its own name on a high-risk system or changes it substantially. The same happens if it changes a system's purpose so that it becomes high-risk. | No start date for Article 25 in the sources reviewed. | A change-control step that decides when the role flips. | Look at what these duties name. They name an accountability framework (Article 17) and people with the competence, training and authority for human oversight (Article 26). None names a committee, a centre of excellence or a Chief AI Officer. In practice, Article 4 needs someone to run the literacy measures and Article 50 needs an owner for disclosure wording; that is a reading, not law text. The Commission adds that, for deployers of high-risk systems, the duty to train staff for human oversight "remains in place" alongside Article 4. I deliver the Article 4 literacy measures for clients. Each role gets a short session built for that role and a one-page list of dos and don'ts. The client signs the internal record. How firm are the EU dates? The omnibus was signed on 8 July 2026 and is in force from 27 July. It moved the high-risk dates to 2 December 2027 (Annex III systems) and 2 August 2028 (Annex I systems). Its preamble gives the reason: "the delayed availability of standards, common specifications, and alternative guidance and the delayed establishment of national competent authorities." It also rewrote Article 4 and Article 17(2). Treat the new dates with care. The omnibus dates from July 2026, and several Commission guidelines are still pending, so the practical detail is still to come. I plan client work against the dates recorded on this page: Annex III from 2 December 2027 and Annex I from 2 August 2028. I do not put dates in contracts. Company size also changes the exposure to fines. Small and medium-sized companies pay the lower of the two fine limits. The omnibus extends that rule to small mid-cap companies, a new size class just above them, for all but the highest tier. Who decides, and who answers? This page treats five decisions as ones that need a named owner; that is the writer's reading, not a finding of any source. The table sets what law and standards say about each against what surveys report. The last table in the next section gives the sample behind each survey. | Decision | What law and standards say | What organisations report | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Approve a system for production | NIST asks executive leadership to take responsibility for decisions about AI risk (GOVERN 2.3). It prescribes no committee or team shape. | The Bank of England and FCA surveyed 118 firms in 2024. 72% give accountability for AI use cases to executive leadership. Most name three or more accountable people or bodies. | | Own the evaluation | NIST MEASURE 1.3: internal experts who did not build the system, or independent assessors, join regular assessments. | No source reviewed says who should own an evaluation set after handover at a company of this size. | | Sign risk acceptance | NIST asks for residual risk to be documented. It names no signer, and neither do the AI Act articles reviewed. | The US federal memo names one: a waiver of its requirements for high-impact AI needs a written determination by the agency's Chief AI Officer. | | Keep the inventory | NIST GOVERN 1.6 asks for mechanisms to inventory AI systems. | ASIC found some licensees with no AI inventory and others with several dispersed model registers. | | Run it after launch | AI Act Article 26: deployers of high-risk systems monitor them, keep logs for six months or more and report risks. | A 2025 survey sponsored by Pacific AI found that 48% of organisations monitor production AI systems. It gives no clear figure for companies of this size. | In the table, NIST speaks to four of these decisions and AI Act Article 26 to the fifth. Of the texts reviewed, no NIST subcategory names an approver, NIST names no risk-acceptance signer and the AI Act articles reviewed name none. Practice varies too. The IAPP and Credo AI report says: "There is no clear best practice for how to build and organize an AI governance team." ASIC describes better and poorer practice for a committee where one exists. It describes better practice as a cross-functional, executive-level committee with clear responsibility and decision-making authority. Poorer practice was a committee that met infrequently and had a poorly defined mandate. Fewer than a third of the licensees it reviewed had a committee, or were setting one up. The AI register is where the owner and risk tier of each system live, and a short review checklist can be attached to each entry. The register post shows how to keep one. Which structures do companies use, and how good is the evidence? Each structure in this table carries a grade in plain words. Required means a law or binding rule says it. Observed means a regulator or a survey reports what organisations do, with the sample named. Recommended means a research centre, a consultancy or a vendor says organisations should. Popular means many vendors repeat it and no data was found behind it. Described, not evaluated means an arrangement is described and no effect is reported. | Structure | What it means | What the evidence is | Grade | | -------------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Centralised | One team or centre of excellence decides, and often builds. | Gartner reports that almost 60% of high-maturity organisations centralise AI strategy, governance, data and infrastructure. | Observed. Gartner scores maturity from a seven-question survey; its press release does not list the questions. | | Hub-and-spoke or federated | A central group sets standards. Units run the work. | McKinsey's guide describes three archetypes: centralised, federated and decentralised, with a trade-off for each, and says the decentralised one needs very high domain maturity and sufficient scale. IBM surveyed Chief AI Officers. Those leading hub-and-spoke or centralised models report twice as many pilots moved into production and 36% higher ROI. Every respondent was a Chief AI Officer, so each already had a central AI leader. IBM adds that organisations generally move from decentralised models to centralised or hub-and-spoke models as they scale. Microsoft describes its own hub and spokes. It is the one long-running named case, and it is a very large company describing itself. | Observed and recommended. The ROI is self-reported. No cost or failure data. | | Decentralised | Each unit owns its own AI work. | ASIC placed its "leveraged, decentralised" licensees in the middle of its maturity scale. MIT NANDA found that decentralised implementation works when accountability stays put: "companies succeed when they decentralize implementation authority but retain accountability." | Observed. The sources point different ways. | | Embedded | AI people sit inside business units. | One case story from MIT CISR: Italgas placed business translators in each unit and added a group AI office. No source reviewed measures this against the others or separates it from decentralised. | Popular. One case story. | | Champions network | Named people in teams who spot issues. | Microsoft runs Responsible AI Champs. The European Commission lists an AI Champions network among its own staff measures. Neither reports an effect. | Described, not evaluated. | The next table gives the sample behind eight of these sources and the main limit of each. None of the large studies reports results for companies of 30 to 300 people. | Source | Sample | What it found | Main limit | | -------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Bank of England and FCA | 118 financial firms, 2024 | 72% give accountability for AI use cases to executive leadership. | Self-reported. Financial services only. | | ASIC Report 798 | 23 licensees and 624 use cases, data from December 2023 | Seven had, or were setting up, a committee. Governance lagged adoption. | ASIC says the sample is not representative. | | IBM Institute for Business Value | More than 600 Chief AI Officers, 2025 | Hub-and-spoke and centralised models link to more pilots in production and higher ROI. | Self-reported. Every respondent already had a central AI leader. | | Gartner | 432 respondents, fourth quarter of 2024 | Almost 60% of high-maturity organisations centralise AI strategy, governance, data and infrastructure. | Gartner scores maturity from a seven-question survey. Its press release does not list the questions. | | MIT NANDA | 52 interviews and 153 survey responses, 2025 | Companies succeed when they decentralise implementation authority but keep accountability. | Preliminary. Selection bias, by its own account. A third-party copy was read. | | IAPP and Credo AI | More than 670 respondents in 45 countries (report published April 2025) | Privacy, and legal and compliance, each have primary responsibility for AI governance in 22% of organisations. | Self-reported. | | Gradient Flow | 351 respondents, 2025 | A role or office for AI governance existed at 36% of small companies and at 62% to 64% of larger ones. | Pacific AI sponsored it. Small means 500 staff or fewer. | | UTS Human Technology Institute | Two papers, October 2025 and May 2026. Recommendations, not outcome data. | Recommends one accountable executive with a small cross-functional working group. | A university research centre with industry partners. It shows what advisers agree on, not outcomes. | Most surveys are self-reported and lean toward large companies. ASIC's data is from December 2023. The sources also disagree about where authority should sit. IBM and Gartner lean central. MIT NANDA leans local for implementation. They may describe different layers: central for policy and risk, local for finding use cases. No source tests that reading. I take no public side on centralised versus decentralised. Where do the switch-point numbers come from? Vendor pages tell readers to change structure at a set number of AI projects or business units. The numbers differ from page to page. No data was found behind any of them: the one vendor page read in full cites no data for its numbers, and the other two were seen only as search snippets. One page also puts a figure on rework. That figure has no data behind it either. This page does not repeat those numbers. Printing a number gives it weight the evidence does not give it. The three vendor thresholds found have no data behind them. No source reviewed reports a cost for any structure in headcount, budget or time. My own switch point has no number. The structure grows when the queue of decisions outruns what one named owner can review. One ASIC finding is about the order of events. Among the licensees ASIC reviewed that planned to use more AI, nearly all were only planning or starting a governance uplift. Only one appeared to have done it before the rise in use it expected. In that sample, most governance uplifts were still being planned or started when the licensees expected their use of AI to rise. What fits a company of 30 to 300 people? The sources say little about a company this size. Gradient Flow ran the only size-split governance survey found. Small companies were far less likely than larger ones to have a role or office for AI governance. Pacific AI sponsored it. Small there includes companies under 30 people. The UTS Human Technology Institute describes a lean design: "a single accountable executive supported by a small cross-functional working group." It says the CEO may take that role in a smaller organisation. It also says "There is not a single correct answer to these questions." Its papers show what advisers agree on, not what has been shown to work. Parts of the law scale with size too. Article 17(2) ties a provider's quality management system to the size of its organisation. The Australian government's AI adoption guidance tells organisations to adapt each practice to their size. It also asks for a senior leader as the overall owner and a specific person accountable for every system. How do you choose? The evidence cannot pick a structure for you. It can narrow the choice. What follows is a reading of the sources, not a finding from any of them. 1. Start from the duties you have. Decide for each AI system whether your company is the provider or the deployer, and whether the system is high-risk. That decides which dates apply. 2. Name an owner for each of the five decisions. The law and the standards reviewed ask for named people with clear roles. None requires a committee. 3. Choose the smallest structure that covers them. The lean design above is one accountable executive and a small working group. Add a standing group only when a decision needs more than one function. Give any group a clear mandate and regular meetings, because ASIC's poorer practice was a committee that met infrequently and had a poorly defined mandate. 4. Pick triggers from your own events, not from a vendor number. Review the structure when a duty date arrives, when you add a high-risk system, after an incident, or when use grows faster than the controls. 5. Test the structure on your own records at a fixed date. None of the large studies reports results for a company your size. Your register does. The post on why pilots stall shows what happens when step two is skipped. Write the five names down first. Then see whether any duty above needs more than that. What do these sources leave out? They cannot say which structure produces better results at a company this size. None of the large studies reports results for companies of 30 to 300 people. They give no cost for any structure. Most are dated 2023 to 2026; Microsoft's account of its own programme is from 2021, and two pages carry no date. Some of the data is older than the omnibus. Agentic systems are a gap. No regulator text or standard reviewed assigns decision rights to an AI agent. The Australian guidance still says to name a person accountable for every system. Standards are moving too. NIST says its framework "is being updated", so every subcategory cited here comes from the January 2023 text.posthttps://meshin.co/blog/multi-agent-reference-architecture-in-productionhttps://meshin.co/blog/multi-agent-reference-architecture-in-productionThe sources reviewed here include no multi-agent reference architecture from a standards body. Microsoft, Google Cloud and Anthropic advise starting small. In my reading, Anthropic and Cognition mark out the two sides of one agent versus several, and the guidance says least about limits, rollback and ownership. Little of it is measured.Fri, 02 Oct 2026 00:00:00 GMTMicrosoft, Google Cloud and AWS each publish guidance on multi-agent systems. The guidance is useful, and it agrees on more than you might expect. It is also vendor guidance with little measured evidence behind it. In my reading it says least about limits, rollback and ownership. This page grades the main claims and says plainly where the research found nothing. A multi-agent system, in these sources, puts several agents on one task. Each agent is a model with its own instructions and tools. One agent usually coordinates the rest. The sources call it an orchestrator, a supervisor, a coordinator, a manager agent or a lead agent. This page says orchestrator. The agentic workflow entry in the glossary covers the wider idea. Outside claims name their source, and a date where the page has one. I read each page on 2 October 2026, except the Gartner release. Gartner blocks scripted reads, so I checked its sentences on a reproduction. Several of the pages change often. Where I write that I found no source, I mean none among the sources I read. A miss does not prove nobody says it elsewhere. How far can each claim be trusted? Treat most of it as advice. Almost none of it was measured in a production workload. The table grades the points that matter most, and the sections after it give the detail. | Point | What the sources show | How far to trust it | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | One agent or several | Anthropic (13 June 2025) reports gains from its multi-agent research system. Cognition (12 June 2025) argued against agents working together. On 22 April 2026 it said its observations still hold for parallel writers. | Each is one vendor's own work. I found no source that compares both in a production workload. | | Test failure rates | A NeurIPS 2025 paper (revised 26 October 2025) found failure rates from 41% to 86.7% across seven open-source frameworks. | Peer reviewed. It tested open-source frameworks, not production systems with an eval gate. | | Cancelled projects | Gartner (25 June 2025) forecasts that over 40% of agentic AI projects will be cancelled by the end of 2027. | A forecast. The release leans on a poll of 3,412 webinar attendees. | | Runaway loops | Anthropic says early versions spawned 50 subagents for simple queries. A preprint (2 July 2026) found loop-prone code in 47 projects. | Not found: a dated, checkable post-mortem from production. The only accounts found repeat one set of figures with no primary record. | | Human approval | Anthropic (25 May 2026) says users approved roughly 93% of permission prompts. Microsoft (4 June 2026) calls bypass of the human step the most consistently exploited failure mode. | Each is the vendor's own finding. Anthropic's post gives no method for its figure. | | Evals in CI | Microsoft's Cloud Adoption Framework (3 December 2025) and Anthropic's evals post (9 January 2026) say to run evaluations in CI/CD pipelines. | Two primary sources among the pages searched. A practice, not a standard. | | Rollback speed | Microsoft says "immediate". Temporal says "instant". | Not found: a target time in any source reviewed. | Do the big vendors agree on a shape? My reading is mostly yes, on four ideas. A coordinating agent routes work to others. Tool access goes through one controlled boundary. State lives outside the model. Every call is logged against a named identity. The guides do not all say all four, but each idea recurs. Microsoft's reference architecture, last updated on 18 August 2025, has an orchestrator. It consults a classifier to route a request and a registry to find agents. It connects tools through a Model Context Protocol server and keeps state in persistent storage. The multi-agent guide from Google Cloud, last reviewed on 16 September 2025, also centres on a coordinator agent. Microsoft, Google Cloud and Anthropic each advise starting small. Microsoft (12 February 2026) says to use the lowest level of complexity that reliably meets your requirements. Google Cloud (reviewed 28 May 2026) recommends starting with a single agent if you are early in agent development. Anthropic (19 December 2024, updated 10 August 2026) advises finding the simplest solution possible and adding complexity only when needed. AWS (10 June 2026) warns of coordination overhead, handoff complexity and distributed failure modes. An undated AWS Marketplace learning module, written in the first person, also recommends the simplest pattern that fits your workflow. These are companies that sell the platforms. The Microsoft and Google Cloud pages quoted above report no measured data. Their agreement tells you what the vendors advise. Should the default be one agent or several? Start with one. Microsoft, Google Cloud and Anthropic each advise starting small, and the sources still disagree about what comes next. Two of the sources landed a day apart, and my reading is that they disagree on where the line sits. Cognition, which makes the Devin coding agent, published "Don't Build Multi-Agents" on 12 June 2025. Anthropic described its multi-agent research system on 13 June 2025. Cognition wrote that "in 2025, running multiple agents in collaboration only results in fragile systems." One of its principles was to share context. Anthropic says its research system "uses a multi-agent architecture with an orchestrator-worker pattern". A lead agent coordinates specialised subagents that run in parallel. Anthropic's own evaluations show that such systems "excel especially for breadth-first queries". I read that as a read-heavy task, graded by the vendor itself. Cognition came back to the question on 22 April 2026. Its original observations, it wrote, "still hold today for parallel-writer swarms". It also described a narrower pattern that works, in which several agents add intelligence to a task while "writes stay single-threaded". The same post gives two rules that pull apart. One says to "share as much context as possible between the agents". For review agents, it says the technique works best when the coding and review agents "do not share any context beforehand". My default puts the line between reading and writing. Many independent reads suit several agents. Writes that must agree suit one. I found no source that tested this in a production workload. The closest measurement found is a Google Research study of agent designs on benchmark tasks. Its paper, posted on 9 December 2025 and revised on 8 April 2026, compared several designs with a single agent. Results ran from a gain of 80.8% to a loss of 70.0%, depending on the task. The Google Research blog post about the paper, of 28 January 2026, reports that independent agents amplified errors 17.2 times, and centralised ones 4.4 times. The blog counts fewer test configurations than the paper's latest abstract, so the paper changed after the blog. I cannot say the multipliers hold for the paper. Anthropic's own post says multi-agent systems "work mainly because they help spend enough tokens to solve the problem". One reading is that more tokens help, however they are spent. I found no source that separates the gain from structure from the gain from extra tokens. What do the failure studies show? Multi-agent systems fail often in tests, and the studies below do not show how often they fail in production. One is MAST, from Cemri and colleagues, accepted to the NeurIPS 2025 Datasets and Benchmarks track. It read the traces of seven open-source multi-agent frameworks and found failure rates from 41% to 86.7%. Repeated steps and agents that miss a finished task were among the common failures. The paper is peer reviewed. It tested open-source frameworks, not production systems with an eval gate in front of them. A runaway loop is a different case, because it costs money. The research found no dated, checkable post-mortem of one in production. The only accounts I found were blog posts that repeat one set of figures with no primary record. What I found is weaker. Anthropic says early versions of its research system made errors such as spawning 50 subagents for simple queries. A preprint of 2 July 2026, "When Agents Do Not Stop", scanned public agent repositories for code that can start an infinite loop. It counts code patterns and does not show a loop that ran in production. My own case is the one first-hand example here: a runaway loop in client work, with no client named. An automatic stop ended it: a step limit and a cost ceiling. The account is anonymised. Hijacking is another documented failure. NIST's CAISI tested agent hijacking on a public benchmark. The strongest new attack succeeded 81% of the time, against 11% for the strongest baseline attack. NIST published the result on 17 January 2025 and updated it on 19 December 2025. That is a benchmark, so it shows what is possible. It gives no rate for real systems. NIST's wording is that many agents that read untrusted text are open to prompt injection. Where do limits and cost controls belong? On every run, set by a person, with a defined result when the limit is reached. The defaults in popular frameworks differ, and they count different things. The table shows five frameworks as read on 2 October 2026. The OpenAI default comes from its source code on GitHub, and the other rows from documentation pages. | Framework | Default loop limit | What it counts | | ---------------------------- | ------------------- | ------------------------------------------------------------------- | | OpenAI Agents SDK | 10 | Turns of the agent loop in one run. Setting none removes the limit. | | LangGraph | 1,000 | Steps, since version 1.0.6. | | Microsoft Agent Framework | 100 | Iterations of a workflow run toward convergence. | | Google Agent Development Kit | None stated | A loop agent ends on a stop signal or on a maximum you set. | | Claude Agent SDK | No limit unless set | The docs advise setting a budget for production agents. | A turn, a step and an iteration are different units, so in my reading the three numbers cannot be compared. Step limits and cost limits are also separate controls. I give every run a step limit. A run that reaches it stops, an alert goes out, and a person decides whether it resumes. I print no numbers. The Claude Agent SDK documentation says the loop runs until Claude finishes if you set no limits. It calls a budget a good default for production agents. The FAQ on cost per outcome gives this site's definition of the measure, and I would set a ceiling from your own traces. Does an agent need durable state and traces? Yes, the sources say both. State has to survive a crash, and every call needs a trace. Anthropic's managed agents post of 8 April 2026 keeps the session log outside the process that runs the agent. Nothing in that process has to survive a crash. Temporal, which sells durable execution, says its service hands the work to another process after a crash and rebuilds the state. LangGraph's documentation says in-memory checkpoints are lost when the process restarts, and it recommends a durable checkpointer. The principle is well supported. I found no source that says every production agent needs a separate engine, and none that measures the benefit. Microsoft's Cloud Adoption Framework (3 December 2025) says to attribute every action to a specific agent instance and conversation ID. The OpenTelemetry conventions for agent spans were still marked as in development in the repository file read on 2 October 2026. Whether the core traces signal is stable is unconfirmed, so this page makes no claim about it. OpenAI's Agents SDK turns tracing on by default, warns that traces may capture sensitive data, and does not offer tracing under zero data retention. A full trace is kept for 30 days, then only metrics are kept. Personal data is masked before it enters the trace. Anthropic (9 January 2026, updated 18 March 2026) says an eval suite makes problems visible before they reach users, and that it needs clear ownership. Running evals in the pipeline that ships changes is a narrower claim. Two primary sources say it: Microsoft's Cloud Adoption Framework and Anthropic's evals post, which calls automated evals especially useful pre-launch and in CI/CD. The Microsoft Cloud Adoption Framework says to integrate evaluations into CI/CD pipelines before production. The post on evals in CI is about running them without a golden answer. In my work, a failing eval blocks the merge. I set the threshold on the first run against historical data, and it only rises, never falls. The owner can override a block, in writing in the pull request. Who approves what, and who owns the system? A person should approve high-impact actions, and one named person should own the system. The sources agree on the first and say less about the second. The Model Context Protocol specification (revision 28 July 2026) says there should always be a human in the loop who can deny tool invocations. The word is "should", and the specification says the protocol cannot enforce its own security principles. AWS sorts actions into three tiers, autonomous, notify and approve, by impact and reversibility, so that not every action waits for a person. Approval is also a control that can fail. Anthropic reports that its telemetry showed users approved roughly 93% of permission prompts (25 May 2026). It adds that the more approvals a user sees, the less attention each one gets. Microsoft's AI Red Team calls bypass of the human step the most consistently exploited failure mode (4 June 2026). Both are the vendors' own findings. Anthropic's post gives no method for its figure. This page cannot say how many prompts is too many. In my work, two things always need a person: money, and anything that goes outside the company. Everything else runs within limits. Access follows the same logic. Microsoft's Cloud Adoption Framework (page updated 26 June 2026) says agent actions must be attributable and enforceable to a unique identity. The MCP authorization specification says authorization is optional for implementations. I read that as follows: a claim of least privilege through MCP describes how someone configured it, and the protocol does not guarantee it. On ownership, Microsoft's framework says to give agent governance to the leaders who already own cloud governance, security and compliance. It also says to designate a human role accountable for approving deployments and monitoring compliance. Microsoft's reference architecture site describes a cross-functional governance committee. In my reading these fit together if the committee advises and one person decides. This site's approach page asks for one named person who can refuse a launch and is accountable for AI risk. In my work, one named owner holds the budget, the access rights and the stop button. That owner cannot change the risk policy or switch off the eval gate alone. Both need a second person. The EU AI Act names people with competence and authority too. Its human oversight duties apply to high-risk systems only. The Commission's AI Act service desk still shows the 2024 text of those articles. Can you roll back a multi-agent change? Microsoft's guidance says yes, if you version everything that shapes behaviour. I found no target time in the sources I read, so they cannot say how fast. My target is that the last deploy rolls back in minutes with one command. I publish no measured case until one happens and is written up. Microsoft's security page for its reference architecture (updated 12 May 2025) says to version every agent's logic, prompt configuration and communication contract. It also says to enable immediate rollback to a previous version when anomalies are detected. Temporal's "Why Temporal?" page promises instant rollback, with no measurement behind the word. I pin models by exact ID. A model changes only through a commit in version control, and a rollback reverts that commit. Pinning a model does not always pin behaviour. Anthropic's documentation says infrastructure updates occasionally produce minor differences in observable behaviour even when the model ID and weights have not changed. I read that as follows: rolling back the model ID does not always roll back behaviour. Evals run after a rollback are how you would see the difference. Providers also retire models on published schedules, so a pinned model has an end date. Anthropic and OpenAI each publish a deprecations page, and someone has to read it. Which claims did the research not find a source for? The research looked for each of these and did not find it. - A dated, checkable post-mortem of a production agent loop that ran away and cost real money. - Any multi-agent reference architecture from a standards body. NIST announced an AI agent standards initiative on 17 February 2026. Its page, updated on 14 August 2026, describes convenings and gap analyses, and the research found no finished NIST guidance for agents. The ISO, IEC and IEEE catalogues were not browsed, so the honest wording is "not found". - Measured results that compare one agent with several inside one company's production workload. - A target time for rollback in any source reviewed. - Support for the "four planes" model of control, execution, state and capability beyond one undated AWS Marketplace learning module, written in the first person. One figure rests on a weak source. A LangChain blog of 16 June 2025 says to start small with evals and that even about twenty examples is enough. LangChain sells agent tooling and the post is commentary, so this page does not adopt the figure. Retrieval and memory design were outside this research. The retrieval-augmented generation entry covers the basics. Does this site commit to any of it? Yes, to five controls. The approach page names evals in CI, tracing, cost per outcome, a rollback path and a named owner. In my reading, the sources above back them unevenly. Tracing recurs across the guides, and a named owner is clearest in Microsoft's Cloud Adoption Framework. Rollback appears as a practice in three of the 14 pages I searched. Evals in CI appears in the two primary sources named above. The outside sources cover spend limits and spend per reasoning cycle or per agent, which is a different measure from cost per outcome. I found no target time for rollback in the sources I read. The FAQ on what counts as in production gives the bar this site uses.posthttps://meshin.co/resources/adoption-telemetry-dashboard-templatehttps://meshin.co/resources/adoption-telemetry-dashboard-templateAn adoption telemetry dashboard template: eight panels, each with its question, data source and the decision it supports.Fri, 02 Oct 2026 00:00:00 GMTWhat this is A panel specification for tracking how AI tools are used at work. It has eight panels. Each names the question it answers, the data source and the decision it supports. It is a page, not a dashboard file. You copy the panels into whatever tool you already run. Version 0.1 covers coding assistants and chat assistants. Version 0.2 will cover SaaS tools with AI built in and internal agents. I will write it when clients who run them appear. The panels fall into three kinds. The coverage, seat and retention panels count what people did. The two survey panels count what people said. The outcome, delivery health and cost panels read what the work produced. The outcome, delivery health, cost and trust panels carry the weight, and the trust survey is read as opinion. That order is a synthesis of the sources, not a finding any of them states. The adoption telemetry article gives the evidence behind it. The eight panels | # | Panel | Question it answers | Data source | Decision it supports | | --- | --------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- | | 1 | Coverage | What share of AI use at work runs through approved tools? | LLM gateway request logs, the sign-in app list, expense lines, API bills and the AI register | Which tools to approve, replace or block, and where the approved path is hard | | 2 | Paid versus active | Which paid seats see no use? | Vendor admin APIs, such as GitHub's user-management `last_activity_at` | Reclaim or reassign seats, and aim training | | 3 | Retention by team | Do people who tried a tool keep using it at 4, 8 and 12 weeks? | Gateway or vendor logs, with one written definition of active | Change the workflow or retire the tool | | 4 | Outcome per use case | Did the outcome agreed before rollout move, against a baseline or a comparison group? | The system of record for that outcome, such as ticketing, CI or your CRM, not the AI tool's own logs | Scale, fix or retire the use case | | 5 | Delivery health (engineering use cases) | Did throughput or instability get worse as use grew? | CI/CD and version control: DORA's lead time for changes, deployment frequency, failed deployment recovery time, change fail rate and rework rate, plus review time | Slow or pause a rollout when instability rises | | 6 | Cost per outcome | What does one unit of the outcome cost? | Spend and the outcome count from the outcome panel, as set out in how cost per outcome is measured | Budget, build versus buy, and model choice | | 7 | Trust and friction pulse | Do people trust the output, rely on it, and say where it gets in the way? | A five-minute survey once a quarter, using the trust and reliance items DORA publishes, word for word and with credit. The smallest-group rule applies | Training and verification effort, and tool choice | | 8 | Policy clarity | Do staff know what is expected, what is allowed and what applies to them? | The same survey, using the four AI-stance items in DORA's 2025 report, word for word and with credit | Fix how the acceptable use policy is explained | The delivery health and cost panels apply to engineering and workflow use cases. Write the outcome panel for each use case. Fix these definitions before you build - What counts as active. GitHub needs two active days in the trailing 28 to place a user in a usage phase. Microsoft counts a returning user as one with an action in both the current and the previous period. Choose one rule, write it on the panel and keep it. Mine is two different active days within four weeks, and I count people, not accounts. - When you check retention. The retention panel uses 4, 8 and 12 weeks. Week 4 is the first check. I read it as people active in week 4 over people active in week 1, per team, from gateway logs. - The smallest group you show. Mine is five people. Microsoft's Viva Insights needs its minimum group size set to at least five. That is a product setting. The sources read here give no legal figure for a minimum group size. Write down your own. What stays off the dashboard - Seat count as adoption. A seat shows money spent, and GitHub keeps seat data apart from its usage reports. - Daily or weekly active users with no written definition. GitHub, Microsoft and DORA each define active differently. - Prompt counts. DORA's 2024 report, published by Google Cloud, said measuring usage frequency is "likely not as meaningful as measuring reliance". - Lines of code accepted as a success measure. GitHub calls its own lines figure directional. - Acceptance rate as a success measure. In a 2022 preprint on Copilot users it tracked how productive users said they felt. - Vendor hours saved or ROI as value. Microsoft builds assisted hours from action counts, fixed minutes per action, and meeting recaps counted by meeting length. A default hourly rate then values them. - Rankings of named people. DX, a vendor of measurement software, warns against using these metrics to judge individuals. - Prompt content. The GDPR asks for no more data than needed, and Claude Code's documentation says prompt text is off by default. I collect none. Who sees what Show team-level figures by default, for teams of five or more. Where you control the collector, filter or redact the person field before data reaches the dashboard. Claude Code includes the signed-in user's email by default, and its documentation says to filter it at your own backend if that is a concern. The paid versus active panel shows idle seats by team. Reclaiming a seat needs a name, and a name is person-level data. Show named views only for a narrow stated purpose, recorded, approved by two people and kept 30 days. Write down who sees the list. Then run the screening in the adoption telemetry article before the list exists. What it is not It is not a live dashboard, and it is not an export for one tool. It does not show that an AI tool works. Only the outcome panel can do that, against a baseline or a comparison group, and that is a synthesis, not a finding any source states. It is a summary of public documents, not legal advice.resourcehttps://meshin.co/resources/handover-playbookhttps://meshin.co/resources/handover-playbookThe Handover Playbook is six fill-in tables for moving AI leadership from an outside lead to a named owner, from the test to a later check.Fri, 02 Oct 2026 00:00:00 GMTWhat this is The Handover Playbook is a plan to fill in, not a finished one. It moves AI leadership from an outside lead to a named person inside your company. Most rows ask for a name, a place or a date; the rest ask for a decision. Only your company knows those. The reasons for each table, and how much weight each source carries, are in How to hand over AI leadership to an internal owner. Stage 1. The owner and the test Fill this in first. This playbook asks for a written test so that the handover can end on a result, not only on a date. The pilot article says the new owner should have written at least one eval case and watched the rollback rehearsal. The approach page has a related test of whether your engineers can roll back a bad model without waiting on the outside lead. My test has three checks, and I judge each one. The new owner writes an eval case and runs a rollback rehearsal. The new owner also handles one incident without me in the room. | Question | Your answer | | -------------------------------------------------------------------------- | ----------- | | Who takes the role? Write one name. | | | Who decides whether to stop a launch, if that is a different person? | | | What must the new owner do without the outside lead before the role moves? | | | Who judges each check? | | | What counts as a pass? Write it so a third person could judge it. | | | What happens if the new owner is not ready on the planned date? | | Stage 2. What moves My handover takes three months, in three phases. The month-one plan sets the end date, not the contract. | Phase | Months | What happens | | ----- | ------------------ | ----------------------------------------------------- | | 1 | Months one and two | The new owner and the outside lead work side by side. | | 2 | Month three | The new owner leads. The outside lead reviews. | | 3 | End of month three | The final test and the signatures. | One row per item. Fill in who holds it today, who holds it after, where it lives and the date it moved. The reason for each row comes first, and its source where it has one, so you can drop a row you disagree with. | Item | Why it is here | Held today by | Held after by | Where it lives | Done on | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------- | -------------- | ------- | | Decisions and their reasons | A UK regulator says handover material should include judgement and opinion, not just facts and figures. FCA handbook, an analogy from financial firms. | | | | | | Work in progress | Status, owner and next action for each open item. Fractionus, one vendor's advice. | | | | | | Open risks and concerns | The same regulator says the material should include unresolved or possible breaches and unresolved concerns raised by regulators. FCA handbook, an analogy. | | | | | | The list of AI systems | NIST asks for mechanisms to inventory AI systems. NIST AI RMF, Govern 1.6, voluntary. | | | | | | Roles and authority | NIST asks that roles, responsibilities and lines of communication are documented and clear. NIST AI RMF, Govern 2.1, voluntary. | | | | | | Runbook, evals and rollback | Google says training for a receiving team can include hands-on exercises in operations. Google SRE book, the operator's own account. | | | | | | Monitoring and logs | NIST asks for monitoring plans that cover incident response and change management (Manage 4.1). For high-risk systems, deployers will have to keep the logs they control for at least six months. That duty starts on 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems. Article 26 of the EU AI Act. | | | | | | Outside parties | An interim director helps announce the new leader to funders and peers. CompassPoint, a practitioner's guide for US nonprofits. | | | | | | Vendor and sub-processor contacts | Vendor and sub-processor contacts are not personal. They are a row in the AI register, and the renewal calendar moves with it. | | | | | Stage 3. Who signs what NIST AI RMF Govern 2.1 asks that roles, responsibilities and lines of communication are documented and clear. Its voluntary Playbook also suggests documenting delegated authorities. Write that down here for the handover. The rows are examples of decisions that carry a signature; no source lists them. Delete any that do not apply and add your own. For a high-risk system under the EU AI Act, ask counsel who must hold human oversight. | Decision | Signs today | Signs after the handover date | Change takes effect on | | ------------------------------------------- | ----------- | ----------------------------- | ---------------------- | | Entries in the AI register | | | | | Risk acceptances | | | | | Policy exceptions | | | | | Vendor and model approvals | | | | | Changes to eval thresholds | | | | | Rollback calls | | | | | Answers to customer security questionnaires | | | | | Serious incident calls | | | | Stage 4. Access List every account the outside lead holds. Revoke them in one coordinated step on or before the last day, and do not leave removal to the outside lead. A marketplace that sells fractional executives gives this advice and asks for written confirmation. It is one vendor's opinion. The table lists six categories of account. The client's internal owner or IT lead revokes each one, and the revocation is confirmed in writing the same day. A revoke-all line sits on the last-day checklist. | Account or system | Granted by | Revoked by | Confirmed in writing on | | ---------------------------------------- | ---------- | -------------------------------------- | ----------------------- | | The AI register | | The client's internal owner or IT lead | | | The LLM gateway | | The client's internal owner or IT lead | | | CI/CD | | The client's internal owner or IT lead | | | The cloud consoles | | The client's internal owner or IT lead | | | The vendor panels | | The client's internal owner or IT lead | | | The SSO admin | | The client's internal owner or IT lead | | Stage 5. After the last day Sources agree that some reachable help is useful and differ on how much. Agree it in writing, with its length, scope and who may call. The middle column holds my terms where I have set them. | Question | My terms | Your answer | | -------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------- | | Will the outside lead stay reachable, and for which questions? | Yes. Questions by email, answered within one working day. | | | For how long, and in what fee shape? | Thirty days after the last day, with one review call on day 30. | | | Who may call? | | | | What is out of scope after the last day? | Anything beyond the email answers and the day-30 call is a separate agreement. | | Stage 6. The check after handover Pick the date and the person who asks. The first question comes from one vendor's advice to ask the successor about a month after the exit. The other rows are prompts. Add your own. | Question | Date | Who asks | Answer | | -------------------------------------------------------------------- | ---- | -------- | ------ | | What do you wish you had known on day one? | | | | | Which decision since the handover was made without the outside lead? | | | | | Which item in the What moves table turned out to be missing? | | | | | Do customer paperwork and the AI register show the new name? | | | | What it is not It is not a standard. A search in October 2026 found no standard, law or study that sets out how to hand over AI leadership, and no outcome study of AI-leadership handovers. Nothing shows that these tables lead to better results. It is not legal advice, and it is not a compliance certificate. The FAQ on how the Retainer ends covers how the Fractional CAIO Retainer ends. No engagement has yet ended in a full handover by the method in these tables.resourcehttps://meshin.co/glossary/ai-operating-modelhttps://meshin.co/glossary/ai-operating-modelAn AI operating model names who chooses, reviews and runs a company's AI work. What it covers has no standard definition, so check which one is in use.Fri, 02 Oct 2026 00:00:00 GMTWhy it comes up No standard defines the term, so a board that asks someone to build one has no standard definition to hand them. What to do about it Answer the three questions for one real system, and write a name beside each answer on one page. Then decide what else the company needs. The article on how to choose an AI operating model for a mid-size company covers the legal duties and the evidence. It also says how to choose, and where the evidence runs out. Treat any structure sold as the answer with caution, because no source reviewed shows that one structure works best.glossaryhttps://meshin.co/glossary/dpiahttps://meshin.co/glossary/dpiaA DPIA, or data protection impact assessment, is a written GDPR check of privacy risk before high-risk personal-data processing begins.Tue, 08 Sep 2026 00:00:00 GMTWhy it comes up Almost any AI system worth building processes personal data, which makes a DPIA the default question rather than an edge case. Legal or procurement asks for it specifically once a system moves from a prototype to something touching real customer records. People often confuse it with the newer AI Act impact assessment. That is why explaining the difference is usually the more useful answer than either document alone. When monitoring staff triggers one Article 35(3) names three cases, but Article 35(1) asks for a DPIA wherever processing is likely to result in a high risk to people. The Article 29 Working Party's 2017 guidelines list criteria that point to high risk. Systematic monitoring is one. Data about vulnerable people is another, and employees may count as vulnerable because of the power imbalance. Meeting two criteria means a DPIA in most cases. Whether a dashboard of AI tool usage counts as systematic monitoring depends on how person-level it is. That is an inference, not a regulator's ruling. What to do about it Write the DPIA before the system is built, not after, since a risk found during design is cheap to change. The same risk found after launch usually is not. Start from what the system actually collects and why, not from a template's headings. Consult the data protection officer while there is time to act. The AI Governance Starter Kit lays out the same data classification a DPIA starts from.glossaryhttps://meshin.co/glossary/mcphttps://meshin.co/glossary/mcpThe Model Context Protocol (MCP) connects AI agents to tools and data, but cannot enforce consent, data privacy or tool safety at the protocol level.Fri, 02 Oct 2026 00:00:00 GMTWhy it comes up In this site's reading, it comes up when a team connects an agentic workflow to its own systems and a security reviewer asks what the agent can touch. "Open protocol" can sound like a safety claim. It is not one. A 2026 sheet from the US National Security Agency (NSA) says MCP's security posture "remains uneven and highly dependent on implementation discipline rather than protocol guarantees". The sheet cites the November 2025 revision of the specification. What to do about it The model picks which tool to call, and text it reads can steer that choice. That is prompt injection. The MCP project's security policy says a report that an LLM invoked an unexpected tool is not an MCP vulnerability. So the limits go in the host and the servers, as guardrails in code. Give each server only the access its task needs. That is least privilege. The NSA sheet recommends that tool execution be constrained and sandboxed. I start every new tool read-only. Any action that writes, spends money or touches personal data needs approval from a named person. Credentials live with the gateway or the server, never in the prompt, and they rotate every quarter. Before a third-party server connects I pin the exact version of a third-party server and review its source before it connects. The approved-server list lives in the client's AI register. Where the specification stops The specification, in its 28 July 2026 revision, names who should do the first three jobs. The protocol does not check that anyone does. - Approval before a tool runs: the host. There should always be a human in the loop who can deny a call. - Checks on inputs and access: each server. Servers must validate all tool inputs and implement proper access controls. - Logs: the host's clients. They should log tool usage, which feeds observability. MCP traffic goes through the gateway. The call and the result are logged, never the full arguments, and the log is kept for 90 days. The log is reviewed in the weekly ops check. - Local servers: a local server the client starts runs with the client's privileges. The policy says the SDK's STDIO transport, which does the starting, "is not a sandbox". Local STDIO servers run inside a container by default. Remote HTTP is used only through the gateway, with authentication. What the client receives The server inventory and the approved-server list land in the client's AI register. This work is part of the two-week AI readiness and risk assessment. When I review this entry Last reviewed: 7 October 2026. I review this entry again on every new revision of the specification and on any new class of CVE.glossaryhttps://meshin.co/blog/ai-acceptable-use-policy-for-a-50-person-companyhttps://meshin.co/blog/ai-acceptable-use-policy-for-a-50-person-companyAn AI acceptable use policy for a company of 30 to 300 people classifies which data may reach which AI vendor and names who approves a new tool. It also requires review before AI output reaches a customer. It must not try to ban AI outright or list every tool by name. It also names who decides after a breach.Tue, 08 Sep 2026 00:00:00 GMTMost companies between 30 and 300 people already have staff using AI tools, whether or not a policy says so. The policy that works is not the longest one. It states which data may go where, who approves a new tool, and what happens when someone gets it wrong. It does this in language a new hire reads in one sitting. What does a working AI acceptable use policy actually need to say? A policy that survives contact with real staff decides a short list of things, not a long one. Nine sections make up a working one. Each row here names the section, the decision it forces, and the specific failure it exists to prevent. | Section | What it decides | Failure it prevents | | --------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | Scope | Which staff, contractors and AI tools the policy covers. | A contractor or a new hire assuming the rules do not apply to them. | | Data classification | Which data tier may reach which category of AI vendor. | Client or personal data pasted into a public chatbot with no data agreement. | | Approved tools and requests | Which AI systems are pre-approved, and how staff add another one. | Staff picking an unapproved tool because asking takes too long. | | Human review | Whether a person checks AI output before it reaches a customer, a contract or a filing. | An unreviewed AI draft going out under someone's name. | | Customer disclosure | When the company must tell a customer they are talking to an AI system or seeing AI-made content. | A hidden AI interaction reaching a customer where disclosure is a legal duty. | | Vendor training use | Whether a vendor may use company or client input to train its own models. | Confidential input quietly improving a vendor's public model. | | Ownership and review | Who owns the policy, and how often it is checked against what staff actually use. | A policy that describes a tool nobody uses, and stays silent about the one everyone does. | | Incident reporting | What staff do the moment an AI system produces something wrong or exposes data. | A bad output or a leak going unreported because nobody knew who to tell. | | Enforcement | What happens after a breach, and who decides. | A rule everyone knows the company never actually enforces. | The customer disclosure row is not optional once a system talks to people outside the company. Article 50 of the EU AI Act already requires a provider or a deployer to tell a person they are interacting with an AI system. It also requires labelling content the system generated, whether or not that system counts as high-risk. A staff-facing policy has to name who checks that duty before a customer-facing AI feature ships. The policy must not leave that to whoever built the feature to remember. What should the policy refuse to try to do? A policy that tries to be the technical control fails twice. It fails once because a paragraph of text cannot stop a paste into a browser. It fails again because staff read that overreach as proof the whole document is for show. Access limits, contract terms and vendor settings do the technical work. The policy states the rule those controls exist to support. Listing every approved tool by name reads as complete on the day you publish it. It is wrong within weeks, because a new model or a new vendor account always arrives faster than a document gets updated. A policy built around named tools also implies that anything missing from the list is quietly fine, and a reviewer wants to see the opposite. An outright ban is the easiest policy to write and the least effective one to run. It does not remove the reason someone went looking for an AI tool in the first place. It just moves that search out of sight, where it is much harder to see or govern. What do you do about the tools staff are already using? By the time you draft a policy, some staff are already using AI without asking anyone first. That means drafting client emails in a personal chatbot account or running a browser extension nobody vetted. It also means building a habit around a coding assistant they picked themselves. That is shadow AI, and treating it as rare is the first mistake a policy can make. Two moves fix more of it than a warning email does. First, make the approved path faster than the workaround: a fast yes for a low-risk tool beats a slow review nobody wants to wait for. Second, check instead of asking. Checking browser extensions, expense claims and what the company actually pays AI vendors shows what people are really running. That beats trusting a self-reported survey. What happens when someone breaks the policy? Most breaches are not sabotage. Someone pastes a client's contract into a personal chatbot account to move faster, not to leak it. That is different from someone who was told a tool was blocked and used it anyway, or who hid AI involvement in work they signed off as their own. A policy that cannot tell the two apart will punish the wrong case or excuse the real one. One named role decides what happens next. That is the same person who owns the AI register and the policy itself, not a group chat and not a vote. They classify what happened and decide whether legal or an affected client needs to hear about it. They also choose the response, from a conversation to removal of access to company systems. That decision runs through the disciplinary process HR already has for any other policy breach. The AI policy needs a clear point where that existing process starts, not a punishment system of its own. Silence is its own answer too. If a policy never states what happens after a breach, staff test it once and learn from the result whether any of it is real. Who should own it, and how does it stay current? The same person who is already accountable for AI risk overall should hold this document. A shared inbox cannot hold it, because someone has to be answerable when a customer or an auditor asks a direct question. ISO/IEC 42001 treats a policy like this as one part of a management system, reviewed on a schedule and checked against a live inventory. A review interval only works if you tie it to something that actually changes. That can be a new hire, a new vendor account, or a client contract that names a data rule of its own. Quarterly is a reasonable default for a company this size. The exact interval matters less than the fact that one exists and someone is accountable for keeping it. The version I run for my own practice is public at my AI policy. It is one page, a worked example of the same shape at a much smaller scale. The Starter Kit lists the documents a security review asks for and who signs each. It suits a company with more than one person to coordinate. What doesn't a policy like this solve? A written policy does not tell you whether anyone inside the company actually owns AI risk today. If nobody does, there is no obvious person to hand the draft to. So the gap usually surfaces during an AI readiness and risk assessment, before it ever reaches the policy stage. Someone determined to break the rule will do it regardless; no policy stops that. A policy removes the excuse of not knowing, and it gives the company a fair, defensible reason to act when someone crosses the line anyway. Nobody enforces it without an owner, a review date and a stated consequence. Skip those and it is a page people skim once and never open again.posthttps://meshin.co/blog/how-to-build-an-ai-system-register-in-a-weekhttps://meshin.co/blog/how-to-build-an-ai-system-register-in-a-weekBuild a usable AI system register in a week with a spreadsheet and four interviews, no software required. The register lists every AI tool a company builds, buys or lets a team adopt. Each entry records what it does, what data it touches, who owns it, and how risky it is. A security reviewer asks for this list first.Tue, 08 Sep 2026 00:00:00 GMTA security review of an AI system usually starts with one question: what is actually running. If the honest answer takes a week of digging through chat logs and old tickets, the review stalls before it reaches architecture or data flows. An AI system register is the fix: one document listing every AI system a company runs, who owns it, and how risky it is. What is an AI system register? An AI system register is the full inventory of every AI system a company runs, regardless of how it got there. That includes a customer-facing feature and an internal copilot. It also includes a marketing tool with a model switched on inside it that nobody called an AI project when they signed up. A working entry names what the system does, what data can reach it, and which vendor or model sits behind it. It also names who owns it and a rough risk tier. Some call this same list an AI register rather than an AI system register. Most of the systems on the list never started as a formal AI project. That is why the list has to go looking for them rather than wait to be told. Why does a security reviewer ask for it before anything else? A security or procurement reviewer asks one version of the same question first: what touches our data. Without a register, answering that means paging engineering, marketing, finance and support one by one, and hoping nobody forgot a tool. The NIST AI RMF treats this as a named control, not a nice extra. Its Govern function includes a specific requirement: a mechanism to inventory AI systems, resourced according to how much risk the organisation is willing to carry. Everything a reviewer asks next, a data flow diagram, a vendor contract clause, an incident plan, gets scoped against rows already on this list. Skip the register, and the reviewer either stalls the process or signs off on a system nobody can fully describe. What should the register record? A register only earns its keep if a reviewer can answer their own questions from it, without a follow-up meeting. What follows is close to the version I hand a client. It has enough fields to answer a first review, and no field that exists only to look thorough. | What to record | Why a reviewer wants it | Who fills it in | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | Name the system and what it does. | A reviewer cannot scope a check against a system they cannot picture. | The system's product or engineering owner. | | Note whether the company built it in-house or bought it, and which vendor or model sits behind it. | A vendor's own controls become part of the risk the moment its API touches your data. | Whoever set up the account or the integration. | | List the data the system can see. | This sets the risk tier and decides which contract terms and legal duties actually apply. | The team that owns the data source, with legal signing off. | | Record who relies on the system's output. | Blast radius matters as much as data sensitivity when a reviewer decides how much scrutiny to apply. | The system owner, from usage or team headcount. | | Mark its deployment status: pilot, production or retired. | A pilot running on sample data is a different conversation from a live system holding real customer records. | The system owner, kept current as status changes. | | Name one internal owner. | A reviewer wants a person who can answer for the system, not a distribution list nobody reads. | Whoever runs the team that operates the system. | | Assign a risk tier. | This decides how much documentation, testing and sign-off the reviewer expects before anything ships. | Whoever runs the security or risk review, not the system's own owner. | | Log the date it was last checked. | A register that has not moved in a year gets treated the same as no register at all. | Whoever owns the register process, on a fixed schedule. | How do you build one in a week without buying a tool? None of this needs procurement. A spreadsheet, a shared document and four short conversations get a usable first version out the door. - Monday. Create the spreadsheet with those three columns. Seed it with every system already in use: the model inside the product, the support tool, the analytics dashboard that quietly added an AI summary. - Tuesday and Wednesday. Interview engineering, marketing, finance and support directly, not by survey. Ask each team what they actually use day to day, since a spreadsheet plug-in with a model behind it belongs on the list too. - Thursday. Fill in the vendor, data and risk-tier columns from what you heard. Flag anything nobody could describe clearly for a follow-up conversation rather than a guess. - Friday. Assign an owner to every row, pick a reviewer for the risk tiers, and put a recheck date on the calendar before anyone calls the list finished. What a register does not do Building the register is the easy part of this, and several harder parts of AI governance sit outside it. It does not replace a lawyer's read of a vendor contract. The register tells legal which vendor to check and what data reaches them. It says nothing about whether that vendor's specific contract terms are safe to sign. Those terms change often enough that last year's answer will not do. The registration duty in the EU AI Act is a separate, public step for certain high-risk systems. It means registering under Article 49, in the database the Commission set up under Article 71, before the system goes live. An internal register is how you find out fast which systems might owe that separate duty; it is not a substitute for filing it. It does not stay accurate on its own, either. A register nobody owns drifts out of date within a quarter, and a stale register reads to a reviewer as no register at all. If you would rather someone else run the discovery and the first risk pass, an AI Readiness & Risk Assessment does the same work in two weeks. It also hands you a finished register. The starter kit lists the register with the other documents a security review asks for.posthttps://meshin.co/blog/why-your-ai-pilot-never-reached-productionhttps://meshin.co/blog/why-your-ai-pilot-never-reached-productionMost AI pilots stall because no single person owns the risk, legal has never decided which data may reach a vendor and success has no number attached to it. A pilot reaches production once someone is accountable for it, the data path is classified, an outcome is measured and a regression triggers an automatic rollback.Tue, 08 Sep 2026 00:00:00 GMTA pilot stalls in the space between working once and working every time. Most proof-of-concepts skip four things production always needs. Two of them are a named owner who still answers a year after launch and a decision about which data may leave the building. The other two are a number that says whether the system is working, and an automatic check that catches a regression before a customer does. Where do pilots actually stop? Nobody owns the risk Pilots are usually built by whoever was curious enough to try. That person is rarely still around, or still in charge, once the project needs a decision about risk. Nobody has the standing to say the system is ready, and nobody has to explain a bad output to a customer or a regulator. Ownership means one name attached to the outcome, with the budget and the authority to pause the system if it misbehaves. A shared inbox or a Slack channel does not count. Without that name, every stall becomes permanent, because nobody is accountable for unsticking it. Legal has not said which data may leave Most pilots reach a vendor's API before anyone has classified the data going into the prompt. That is fine for a demo running on sample records. It stops a rollout the moment a security review asks what customer data the system touches and where it goes. The NIST AI RMF treats this as a governance decision under its Govern function, not a technical detail to settle after launch. Without a data classification written down in advance, every new use case reopens the same argument with legal. The pilot waits on a decision nobody has been asked to make yet. There is no measurable outcome A pilot that nobody can prove is working also cannot be proven to be failing, which is worse. Teams show accuracy on a curated test set, or a handful of good transcripts. Nobody has picked the business number the system is meant to move. Examples are tickets resolved, hours saved or the error rate on a defined task. Without that number, a debate about whether to invest further has no evidence on either side. The safest move for everyone involved is to let the pilot run quietly forever, rather than kill it or fund it properly. Nobody would notice a regression Prompts change, the underlying model changes on a provider's own schedule, and traffic drifts. Pilots rarely have a way to detect when any of that makes the output worse. There is no test suite running against real examples before a change ships. So a regression appears first as a customer complaint or a bad downstream decision, not as a failed build. That is the opposite of how the rest of the engineering organisation ships software. This gap is usually the reason a platform or security review declines to sign off. What has to be true to ship? Six conditions turn a pilot into something a platform team will actually run. None of them are exotic. Most engineering teams already do the equivalent for every other production system they own. | Condition | What it means in practice | What its absence looks like | | ----------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | An owner. | One named person, on the team that runs the system, with authority to pause it. | No one can explain a bad output, so the stall becomes permanent. | | A data classification. | A written answer, agreed with whoever owns data risk, on what a vendor may store. | Every new use case reopens the same argument with legal. | | A measurable outcome. | One number the business already tracks, with a floor the system must clear. | Nobody can prove the pilot works, so it runs quietly forever instead of shipping or dying. | | Evals in CI. | Real test cases run on every prompt or model change; OWASP treats this as baseline hygiene. | A regression reaches a customer before anyone on the team sees it. | | Tracing and cost per outcome. | Every run logged well enough to reconstruct, with a cost a finance team can read. | Nobody can say what a run costs, or which step produced a bad answer. | | Rollback. | A rehearsed way to revert to the last prompt, model or version within minutes. | The team improvises during the first real incident instead of using a tested path. | For a system that touches decisions about people, the EU AI Act sets a similar bar for higher-risk uses. It requires a documented risk-management process, records that make a decision traceable and a human who can step in. Meeting the six conditions above gets most of the way there even for teams the Act does not yet cover. How would I sequence it? I run this as one pass, not six parallel workstreams, because the six conditions above depend on each other. You cannot set a rollback threshold before you have a measured outcome, and you cannot classify data before someone owns the decision to do it. Pick one process Choose a single workflow with enough volume to matter and an outcome you can measure without a survey, not five pilots running at once. An AI readiness and risk assessment is the fastest way to get that choice onto paper. Two weeks of reading the existing systems and vendor list end in a memo. The memo names an owner candidate, a first data classification and the number the process should move. Build it in ninety days A 90-Day Production Sprint turns that memo into a running system. Evals get written alongside the first working version, rather than bolted on at the end. Tracing and cost per outcome go live on a dashboard from week one. The rollback path gets rehearsed before real traffic arrives, not during the first incident. Hand it to someone who was in the room The internal owner should have written at least one eval case and watched the rollback rehearsal, so the runbook describes a system they already understand. A handover document given to someone who was not part of the build is a liability with a due date, not a transfer of knowledge. What this does not solve This sequence gets one process into production with the six conditions in place. Several harder governance questions stay outside it. Your risk appetite is still your call, not this sequence's. Two companies with the same pilot and the same data can reasonably choose different answers about what a vendor may see. That decision belongs to whoever owns risk at your company, not something a framework or an outside advisor can make for you. It does not replace a legal review of your contracts with model vendors. Classifying data tells your lawyers what question to ask. It does not answer whether a specific vendor's terms are acceptable, and that answer changes as vendors change their terms. The underlying model is exactly as reliable as it was before this sequence started. Evals catch regressions against the cases you thought to test. They do not catch every way a general-purpose model can fail on a case nobody wrote down. That is why the rollback path matters as much as the evals do. It does not scale to five processes at once. The sequence above is built for one workflow with one owner. Running it for a second or third process repeats the same ninety days. It does not shortcut them.posthttps://meshin.co/services/90-day-production-sprinthttps://meshin.co/services/90-day-production-sprintA 90-day production sprint exists to get one process running in production and keep it there. I build it with your engineers rather than beside them, so the team can change it after I leave. You get evals in continuous integration, tracing, a cost per outcome your finance team can read and a rollback path.Tue, 08 Sep 2026 00:00:00 GMTWhat this is Ninety days, one process, in production. The measure of success is written into the statement of work before the first day. What you keep The workflow, the eval suite, the traces, and an engineer on your team who has built it with me and can change it without me. Where this maps to a standard Evals in CI apply the MEASURE function (one of the framework's four functions) inside the NIST AI Risk Management Framework (AI RMF). The scope is one production workflow, not a whole AI programme. A dashboard with cost per outcome is evidence a customer's security team can check, not a claim they have to trust.servicehttps://meshin.co/services/ai-readiness-and-risk-assessmenthttps://meshin.co/services/ai-readiness-and-risk-assessmentThis AI readiness assessment takes two weeks of interviews and reading your systems, ending in a memo your board can act on. You learn where you stand against the frameworks your customers ask about, what a customer audit would find today and which three processes are worth automating first. The fee is credited if a Sprint follows within three months.Tue, 08 Sep 2026 00:00:00 GMTWhat this is Two weeks. I interview the people doing the work and read your systems and your vendor list. I write one memo you can hand to a board or a customer. What you keep You keep one document, whether or not you continue. It holds the readiness score, the gap list, and the three processes ranked by what automating them is worth this quarter. ISO/IEC 42001 gap assessment The gap list compares your company with ISO/IEC 42001, the standard for an AI management system that an accredited body can certify. It shows where you fall short of what the standard asks. This is a gap assessment, not a certification. An accredited body runs the independent audit that grants certification, and the certification FAQ covers the difference between a working system and a certificate. Why it is two weeks and not two days A readiness claim you cannot show evidence for is worth nothing in a security review. I read the code, talk to the people who use the tools, and check what your vendors actually do with your data. Two weeks is the shortest span in which I can do all three.servicehttps://meshin.co/services/fractional-caio-retainerhttps://meshin.co/services/fractional-caio-retainerThis fractional CAIO retainer buys one or two days a week as the AI officer your customers can ask for by name. The week is procurement answers, design reviews and a board update, in that order of urgency. The engagement is built to end: there is a handover plan from the first month.Tue, 08 Sep 2026 00:00:00 GMTWhat this is One or two days a week, on the record, as your AI officer. Every week follows the same order: procurement, design, the board. How it ends With an internal owner and written decisions. The handover plan exists from the first month, not the last. What the register maps to The register, policy and classification are the GOVERN function (one of the framework's four functions) of the NIST AI Risk Management Framework (AI RMF). The scope is one company, not a whole sector. ISO/IEC 42001 asks for the same three documents inside a certified management system. This retainer keeps them current without pursuing the certification.servicehttps://meshin.co/resources/agent-evals-ci-starterhttps://meshin.co/resources/agent-evals-ci-starterA starter CI pipeline for agent evals. It runs on every pull request and blocks a bad merge before it ships, using a plain YAML case format.Tue, 08 Sep 2026 00:00:00 GMTWhat this is A CI workflow and an eval case format for one agentic workflow. The same discipline sits inside the 90-Day Production Sprint. A prompt or model change should not merge without passing a test set first. The eval case format | Field | Purpose | Example | | -------- | -------------------------------------------------------------------------- | ------------------------------------------------- | | id | A stable name, so a failure is traceable across runs | refund-double-charge | | input | The exact prompt or tool call the agent receives | Customer says they were charged twice | | expected | A pass rule: an exact match, a check, or a second model acting as a grader | Refund issued, ticket closed, no invented apology | | tags | The failure mode a case guards against | prompt-injection, tool-misuse, refusal | The CI gate The workflow runs the eval set on every pull request, scores it, and blocks the merge under your floor. Nobody has to remember to run it by hand, which is the part that fails first once a team is busy. The tags borrow their shape from OWASP's Top 10 for Large Language Model Applications. Prompt injection and excessive agency (the agent doing something nobody asked it to) appear in a support agent as often as in a public chatbot. A tagged case catches the regression either way. What it is not It is not a considered eval set on its own. A handful of cases pulled from real conversations tests more than a long list invented at a desk. The starter takes no position on which cases you pick. Choosing the cases still maps to the MEASURE function (one of the framework's four functions) in NIST's AI Risk Management Framework (AI RMF).resourcehttps://meshin.co/resources/ai-governance-starter-kithttps://meshin.co/resources/ai-governance-starter-kitAI governance starter kit: the paperwork a security review asks for — register, policy, data classification and a vendor questionnaire.Tue, 08 Sep 2026 00:00:00 GMTWhat this is The four documents that answer most of a corporate security review. They are deliberately short: a register nobody updates is worse than no register. The kit also sets out the order in which to call people when an AI system fails. What it is not It is not a compliance certificate, and it is not legal advice. It is the paperwork a reviewer expects to see before they look at your architecture. Who signs what | Document | What it's for | Who signs it | | ------------------------- | ---------------------------------------------------------- | ------------------------------------------------------- | | AI register | Every AI system in use, who owns it, what data reaches it. | The named AI risk owner keeps it current. | | Acceptable-use policy | What you may paste into a tool, and what you may not. | Every employee who uses an AI tool for work. | | Data classification table | Which vendors may see which tier of data. | The AI risk owner, together with whoever owns the data. | | Vendor questionnaire | One answer set procurement can reuse for every new vendor. | Procurement, before anyone signs a vendor contract. | It maps loosely to what ISO/IEC 42001 and the NIST AI Risk Management Framework (AI RMF) both expect a company to have on file. It does not require either. Vendor questionnaire The other three documents describe how you handle AI. This one asks each vendor how it handles your data. Send the same questions to every AI vendor, and file the answers with the contract. The questions cover these topics: - Whether the vendor trains models on your data, and how you opt out. - Where your data is processed and stored, who at the vendor can read it, and which sub-processors touch it. - How the vendor announces a model change, and whether you can stay on a fixed version. - How the vendor tells you about an incident, and who answers when you call. - Whether the system counts as high-risk under the EU AI Act, and which duties the contract leaves with you. - What happens to your data and prompts when the contract ends. Incident response plan When an AI system fails, its owner should not have to work out who to call. Write this call order down beside the register, with a name and a backup against each role. Inside the company, in this order: 1. The system owner named in the register, who limits or stops the system and keeps its logs and model version as they were. 2. The AI risk owner, who decides whether the incident is serious. 3. An executive who can approve a report and a message to customers. 4. Legal counsel, and whoever handles data breaches if personal data is involved. Outside the company, in this order: 5. The provider of the system, which is often the vendor. 6. The importer or distributor, if the system has one. 7. The market surveillance authority. 8. Affected customers, as your contracts require. For a high-risk system you deploy, Article 26 of the EU AI Act asks the deployer to give notice at once when it identifies a serious incident. The order is the provider first, then the importer or distributor and the market surveillance authority. Log the time each person learned of the incident. The page on serious incident reporting says which moment starts the clock for a report and sets out the deadlines, so this plan does not repeat them.resourcehttps://meshin.co/resources/ai-readiness-scorecardhttps://meshin.co/resources/ai-readiness-scorecardA five-axis AI readiness scorecard you score yourself in an afternoon: strategy, data, delivery, people, and governance.Tue, 08 Sep 2026 00:00:00 GMTWhat this is Five axes, scored from zero to three, that give you a rough position before you pay anyone to check it properly. The axes match the ones in the AI Readiness & Risk Assessment. This is the self-administered version, without the evidence behind it. The five axes | Axis | 0 — not started | 1 — informal | 2 — documented | 3 — evidenced | | ---------- | ---------------------------------------------------------- | -------------------------------------------- | ------------------------------------------ | ------------------------------------------------------- | | Strategy | No one has named which processes to automate first | Someone has an opinion, nothing written down | A ranked shortlist exists | The shortlist is reviewed by whoever owns the budget | | Data | No one knows what a vendor tool can see | A rough sense exists, not written down | A data classification table exists | The table is checked before a new tool is approved | | Delivery | Every pilot has stalled before production | One pilot is close to live | One system is live, unmonitored | A system is live, monitored, with a rollback path | | People | Only the original builder understands any AI system in use | One other person has read the code | A second person has changed it and shipped | Ownership is written down and handed over on a schedule | | Governance | No AI register exists | A register exists, out of date | A register exists, reviewed occasionally | The register is reviewed on a fixed schedule | Scoring it Score each axis from zero to three using the table above, then add the five scores for a total out of fifteen. A low total early on is common. It usually means nobody has written anything down yet, not that the engineering is behind. The five axes are not a substitute for the NIST AI Risk Management Framework (AI RMF) or ISO/IEC 42001. They are a rougher pass at the same territory, built to take an afternoon rather than an audit. What it is not It is not the Assessment, and it carries no evidence a reviewer would accept. A self-given three is a claim until someone outside your team checks what you have actually written down.resourcehttps://meshin.co/resources/build-vs-buy-ai-kithttps://meshin.co/resources/build-vs-buy-ai-kitThe build vs buy AI kit: a one-page worksheet of the six criteria I work through before recommending either path, and how to score them.Tue, 08 Sep 2026 00:00:00 GMTWhat this is A worksheet for one recurring decision: build the workflow yourself, or buy a vendor's version of it. It will not choose for you. It gives the choice a written reason, which is what a board or a buyer asks for later. The six criteria | Criterion | Build tends to win when | Buy tends to win when | | ------------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------- | | Data residency | Customer data cannot leave your infrastructure by contract or by law | A vendor already holds a data-processing agreement you trust | | Customisation depth | The workflow is core to what makes your product different | The workflow looks like everyone else's version of the same task | | Time to a working version | You already have the engineers and the model access | Procurement moves faster than a hiring plan would | | Total cost over years | Usage is high enough that a per-call fee compounds | Usage is uncertain, and a fixed fee caps the downside | | Exit cost | You can rebuild the model calls without rebuilding the product | A second vendor could take over most of the same job | | Ownership when it breaks | An internal team already owns the surrounding process | A vendor's contract has a real answer to who to call | Scoring it Weight the six criteria for your situation, score build and buy against each one, and let the total argue with your gut feeling. A bought tool still needs an owner and a place in your AI register. NIST's AI Risk Management Framework (AI RMF) treats a vendor's model the same as one you trained. That sits under its MAP function (one of the framework's four functions). The risk to your customer does not care who wrote the code. Total cost of ownership The Total cost over years criterion needs a total cost of ownership (TCO) estimate, and a vendor quote covers only part of it. Fill in the categories in the table below, for build and for buy. Use the same horizon for both sides, long enough to cover the longest contract you would sign. For each side, add what you pay once to what you pay in every period of that horizon. | Side | Cost category | What to count | | ----- | ---------------- | --------------------------------------------------------------------------------------------- | | Build | Engineering time | Engineer hours for the first version, and the work those engineers would otherwise ship | | Build | Maintenance | Upkeep as the model API, your libraries and your product change | | Build | Drift monitoring | The evals and review that catch quality slipping as models and data change | | Buy | Licence | The vendor's fee in the unit they meter it, and how it grows with your usage | | Buy | Integration | Connecting the tool to your data, login and workflows, plus the security review | | Buy | Lock-in | The cost of leaving: moving data out, redoing prompts and workflows, retraining staff | Take every amount from your own quotes and payroll, because this page lists no amounts. Use the same usage forecast on both sides, since the criteria table above shows that usage often decides which side costs less. Lock-in is the cost behind the Exit cost criterion, so fill in both together. What it is not It is not a procurement process, and it will not pick the vendor for you. It decides one thing: whether the job belongs inside your codebase or inside someone else's product. This is the call the Fractional CAIO Retainer reviews every quarter, alongside the shadow AI a team already bought without asking.resourcehttps://meshin.co/resources/llm-gateway-governance-confighttps://meshin.co/resources/llm-gateway-governance-configA governance config for an LLM gateway. It decides who may call which model, what gets redacted, how long logs are kept and the rate and fallback rules.Tue, 08 Sep 2026 00:00:00 GMTWhat this is A policy file for whichever LLM gateway you run, hosted or self-built. It is the config, not the gateway itself. The decisions live in a file your team reviews, not in a script only one engineer remembers. The policy schema | Key | What it controls | Example | | -------------- | ------------------------------------------------------------------ | ------------------------------------------------- | | allowed_models | Which providers and model versions a team may call | two named models, not an open list | | redact | Fields stripped before a prompt or response is logged or forwarded | email, national_id, card_number | | rate_limit | Calls per minute before the gateway queues or refuses | a ceiling per team, not per user | | log_retention | How long a request and response pair is kept | matches your data retention policy, not a default | | fallback | What happens if the primary provider is down | a named second provider, or a hard failure | Logging and retention The logging schema borrows its shape from the EU AI Act's Article 12, which requires automatic event logs for high-risk systems. Not every call through a gateway is high-risk. The habit costs little to build in now. It is expensive to retrofit once a regulator or a customer asks first. The `allowed_models` and `log_retention` keys map to two NIST SP 800-53 control families: Access Control and Audit and Accountability. Procurement questionnaires often draw on this catalogue, even when nobody names it. What it is not It is not a gateway, and it will not stop a determined engineer from calling a model directly. It makes an unrouted call a shadow AI finding. That is the layer the 90-Day Production Sprint puts in front of a workflow before it counts as production.resourcehttps://meshin.co/glossary/agentic-workflowhttps://meshin.co/glossary/agentic-workflowAn agentic workflow is a task where one or more AI agents plan steps, call tools and choose the next move without a person's approval.Tue, 08 Sep 2026 00:00:00 GMTWhy it comes up It comes up the moment a pitch or a product description says "agent" instead of "assistant". A security reviewer's next question is almost always what the agent can do without asking first. The reviewer asks because that is where the real exposure sits, not the model choice. A board hears "agentic" and correctly hears "this thing takes actions on its own." That is a different risk conversation than a chatbot that only produces text. What to do about it List every action an agent can take, not just what it can say, and sign off on each as a separate risk. Set a real limit on what it can do without a person confirming first, especially anything that spends money, touches a customer record, or sends something external. Log every tool call, so a bad outcome traces to a specific decision. Evals belong here: test the whole action chain, not just the model's replies.glossaryhttps://meshin.co/glossary/ai-governancehttps://meshin.co/glossary/ai-governanceAI governance is the decisions, roles and controls a company uses to manage the risk of the AI systems it builds or buys.Tue, 08 Sep 2026 00:00:00 GMTWhy it comes up An insurer asks who owns AI risk before writing a policy. A board member reads about an AI failure elsewhere and asks what would happen here. An investor's due-diligence checklist now has a line for it too. Write down who is responsible for each system, and who would notice if it started producing worse answers. What to do about it Start with a one-page register: every AI system in use, who owns it, what data it touches, and whether customer data leaves the building. Assign one named owner per system, not a team. Add a single sign-off step before a new system reaches production. Map what you already do against the NIST AI RMF or ISO/IEC 42001. Do this so you have a recognised structure to point at, not to get certified immediately.glossaryhttps://meshin.co/glossary/ai-impact-assessmenthttps://meshin.co/glossary/ai-impact-assessmentAn AI impact assessment is a written check, before a high-risk system goes live, of who it affects and whether they can challenge its decision.Tue, 08 Sep 2026 00:00:00 GMTWhy it comes up It comes up wherever an AI system's output changes the offer a real person gets. That includes a loan, a job interview, an insurance price or a public benefit. A board hears about it once a pilot nears launch and legal asks who reviewed fairness before customers saw it. Procurement teams increasingly expect a written answer rather than a verbal assurance, since the assessment itself is what a regulator can ask to see. What to do about it Name the specific groups the system could disadvantage before launch. Use the actual decision it makes and who it makes it about. Write down the mitigation for each risk identified, and who checked that the mitigation actually works rather than just exists on paper. Before a customer's legal team asks, I flag which systems need this specific assessment inside the AI readiness and risk assessment.glossaryhttps://meshin.co/glossary/ai-management-systemhttps://meshin.co/glossary/ai-management-systemAn AI management system is the recurring set of policies, roles and reviews a company runs to govern its AI, not a one-off risk assessment.Tue, 08 Sep 2026 00:00:00 GMTWhy it comes up A procurement questionnaire increasingly asks for an AI management system by name, not a risk assessment for one product. This happens because a buyer wants proof that governance survives a change of staff. A board hears the phrase once a competitor announces ISO/IEC 42001 certification and asks whether the company should follow. The honest answer is usually that the certificate matters less than the structure behind it, since an unread policy is worth little more than none. What to do about it Write the policy in one page an engineer would actually follow, not a document written for an auditor. Name an owner for the system as a whole, not just for individual products. Set a fixed date to review the register rather than waiting for a customer to ask. I score exactly this structure against ISO/IEC 42001 in the two-week AI readiness and risk assessment. So the gap list is evidence rather than a guess.glossaryhttps://meshin.co/glossary/ai-registerhttps://meshin.co/glossary/ai-registerAn AI register is a company's own inventory of every AI system it builds, buys or uses, naming its owner and how risky it is.Tue, 08 Sep 2026 00:00:00 GMTWhy it comes up It is usually the first document a procurement team or an investor's checklist asks for, and the first one a company discovers it lacks. A security review cannot get past "what AI systems touch our data" without one. A board asking about AI risk gets a shorter conversation once a real list sits in front of it. For high-risk systems, the EU AI Act goes further: Article 71 requires joining a public EU database. What to do about it Build a spreadsheet before anything more elaborate. Use one row per system, with columns for name, what it does, vendor or model, data it can see, owner and risk tier. Interview engineering, marketing, finance and support instead of relying on a self-reported survey. That is how tools added outside a formal process get missed. Review the list on a fixed schedule, since a stale register looks worse in review than admitting you lack one. Quarterly is reasonable.glossaryhttps://meshin.co/glossary/conformity-assessmenthttps://meshin.co/glossary/conformity-assessmentA conformity assessment proves a high-risk AI system meets the EU AI Act before sale. Either the provider or an outside body runs it.Tue, 08 Sep 2026 00:00:00 GMTWhy it comes up An investor or a board asks how a company knows its AI system is compliant. A vague assurance is not the same as pointing to a completed assessment and a signed declaration. A procurement team increasingly asks to see evidence the assessment happened at all, rather than taking a compliance claim on trust. It also surfaces at the worst time, during a late-stage system change, when nobody planned for a second assessment. What to do about it Work out early, before the system is built, which assessment route applies. That matters because a notified-body review takes far longer to arrange than an internal one, and you cannot rush it at launch. Keep the assessment evidence alongside the technical documentation it depends on, not in a file only one person can find. Mapping each system to its likely route is part of the AI readiness and risk assessment done before a deadline forces the decision.glossaryhttps://meshin.co/faq/are-models-trained-on-our-customer-datahttps://meshin.co/faq/are-models-trained-on-our-customer-dataNot by default.Sat, 12 Sep 2026 00:00:00 GMTNot by default. I use vendor terms that exclude your data from model training, the same enterprise or business-tier terms a compliance team would ask about. I check that clause before any tool touches your data. If a vendor will not commit to that in writing, that tool does not get used on your systems.faqhttps://meshin.co/faq/can-you-get-us-iso-iec-42001-certified-or-just-a-working-systemhttps://meshin.co/faq/can-you-get-us-iso-iec-42001-certified-or-just-a-working-systemI build a working system, not the certification itself.Sat, 12 Sep 2026 00:00:00 GMTI build a working system, not the certification itself. I work to get your management system, risk register, and evidence trail to the point an auditor would recognise as ready. An accredited body runs the independent audit that grants certification. Whether you then pursue ISO/IEC 42001 itself is a separate decision, best made once you can see the system actually running.faqhttps://meshin.co/faq/can-you-work-with-our-existing-ai-vendor-or-platformhttps://meshin.co/faq/can-you-work-with-our-existing-ai-vendor-or-platformI work with what you already run.Sat, 12 Sep 2026 00:00:00 GMTI work with what you already run. OpenAI, Anthropic, Azure, or whatever your team picked before I arrived — the Retainer does not require a vendor switch. What changes is the governance layer around the tools you keep. That means a data flow review, contract terms checked for training exclusions, and evals wrapped around workflows already running. It is not a rebuild from scratch.faqhttps://meshin.co/faq/how-does-ai-risk-get-recorded-and-who-signs-it-offhttps://meshin.co/faq/how-does-ai-risk-get-recorded-and-who-signs-it-offIn an AI register.Sat, 12 Sep 2026 00:00:00 GMTIn an AI register. One line per initiative: its data flows, its risk level, and its owner. Under the Retainer, I review every entry and sign it as the named officer. Your organisation still holds final accountability for anything that reaches customers or the board, as it does for any other risk register you keep.faqhttps://meshin.co/faq/how-is-a-prompt-or-model-change-stopped-from-shipping-unnoticedhttps://meshin.co/faq/how-is-a-prompt-or-model-change-stopped-from-shipping-unnoticedEvals in continuous integration.Sat, 12 Sep 2026 00:00:00 GMTEvals in continuous integration. A test set runs on every change and fails the build when quality drops, the same gate your engineers already trust for code. A prompt edit, a model swap, a vendor's silent update: all pass the same evals as the original release before reaching production. Tracing shows what changed.faq