Fractional Chief AI Officer for 30–300 person companies: AI governance that passes procurement and agentic automation shipped to production.https://meshin.co/Fri, 02 Oct 2026 00:00:00 GMThttps://meshin.co/resources/adoption-telemetry-dashboard-templatehttps://meshin.co/resources/adoption-telemetry-dashboard-templateAn adoption telemetry dashboard template: eight panels, each with its question, data source and the decision it supports.Fri, 02 Oct 2026 00:00:00 GMTWhat this is A panel specification for tracking how AI tools are used at work. It has eight panels. Each names the question it answers, the data source and the decision it supports. It is a page, not a dashboard file. You copy the panels into whatever tool you already run. Version 0.1 covers coding assistants and chat assistants. Version 0.2 will cover SaaS tools with AI built in and internal agents. I will write it when clients who run them appear. The panels fall into three kinds. The coverage, seat and retention panels count what people did. The two survey panels count what people said. The outcome, delivery health and cost panels read what the work produced. The outcome, delivery health, cost and trust panels carry the weight, and the trust survey is read as opinion. That order is a synthesis of the sources, not a finding any of them states. The adoption telemetry article gives the evidence behind it. The eight panels | # | Panel | Question it answers | Data source | Decision it supports | | --- | --------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- | | 1 | Coverage | What share of AI use at work runs through approved tools? | LLM gateway request logs, the sign-in app list, expense lines, API bills and the AI register | Which tools to approve, replace or block, and where the approved path is hard | | 2 | Paid versus active | Which paid seats see no use? | Vendor admin APIs, such as GitHub's user-management `last_activity_at` | Reclaim or reassign seats, and aim training | | 3 | Retention by team | Do people who tried a tool keep using it at 4, 8 and 12 weeks? | Gateway or vendor logs, with one written definition of active | Change the workflow or retire the tool | | 4 | Outcome per use case | Did the outcome agreed before rollout move, against a baseline or a comparison group? | The system of record for that outcome, such as ticketing, CI or your CRM, not the AI tool's own logs | Scale, fix or retire the use case | | 5 | Delivery health (engineering use cases) | Did throughput or instability get worse as use grew? | CI/CD and version control: DORA's lead time for changes, deployment frequency, failed deployment recovery time, change fail rate and rework rate, plus review time | Slow or pause a rollout when instability rises | | 6 | Cost per outcome | What does one unit of the outcome cost? | Spend and the outcome count from the outcome panel, as set out in how cost per outcome is measured | Budget, build versus buy, and model choice | | 7 | Trust and friction pulse | Do people trust the output, rely on it, and say where it gets in the way? | A five-minute survey once a quarter, using the trust and reliance items DORA publishes, word for word and with credit. The smallest-group rule applies | Training and verification effort, and tool choice | | 8 | Policy clarity | Do staff know what is expected, what is allowed and what applies to them? | The same survey, using the four AI-stance items in DORA's 2025 report, word for word and with credit | Fix how the acceptable use policy is explained | The delivery health and cost panels apply to engineering and workflow use cases. Write the outcome panel for each use case. Fix these definitions before you build - What counts as active. GitHub needs two active days in the trailing 28 to place a user in a usage phase. Microsoft counts a returning user as one with an action in both the current and the previous period. Choose one rule, write it on the panel and keep it. Mine is two different active days within four weeks, and I count people, not accounts. - When you check retention. The retention panel uses 4, 8 and 12 weeks. Week 4 is the first check. I read it as people active in week 4 over people active in week 1, per team, from gateway logs. - The smallest group you show. Mine is five people. Microsoft's Viva Insights needs its minimum group size set to at least five. That is a product setting. The sources read here give no legal figure for a minimum group size. Write down your own. What stays off the dashboard - Seat count as adoption. A seat shows money spent, and GitHub keeps seat data apart from its usage reports. - Daily or weekly active users with no written definition. GitHub, Microsoft and DORA each define active differently. - Prompt counts. DORA's 2024 report, published by Google Cloud, said measuring usage frequency is "likely not as meaningful as measuring reliance". - Lines of code accepted as a success measure. GitHub calls its own lines figure directional. - Acceptance rate as a success measure. In a 2022 preprint on Copilot users it tracked how productive users said they felt. - Vendor hours saved or ROI as value. Microsoft builds assisted hours from action counts, fixed minutes per action, and meeting recaps counted by meeting length. A default hourly rate then values them. - Rankings of named people. DX, a vendor of measurement software, warns against using these metrics to judge individuals. - Prompt content. The GDPR asks for no more data than needed, and Claude Code's documentation says prompt text is off by default. I collect none. Who sees what Show team-level figures by default, for teams of five or more. Where you control the collector, filter or redact the person field before data reaches the dashboard. Claude Code includes the signed-in user's email by default, and its documentation says to filter it at your own backend if that is a concern. The paid versus active panel shows idle seats by team. Reclaiming a seat needs a name, and a name is person-level data. Show named views only for a narrow stated purpose, recorded, approved by two people and kept 30 days. Write down who sees the list. Then run the screening in the adoption telemetry article before the list exists. What it is not It is not a live dashboard, and it is not an export for one tool. It does not show that an AI tool works. Only the outcome panel can do that, against a baseline or a comparison group, and that is a synthesis, not a finding any source states. It is a summary of public documents, not legal advice.resourcehttps://meshin.co/resources/handover-playbookhttps://meshin.co/resources/handover-playbookThe Handover Playbook is six fill-in tables for moving AI leadership from an outside lead to a named owner, from the test to a later check.Fri, 02 Oct 2026 00:00:00 GMTWhat this is The Handover Playbook is a plan to fill in, not a finished one. It moves AI leadership from an outside lead to a named person inside your company. Most rows ask for a name, a place or a date; the rest ask for a decision. Only your company knows those. The reasons for each table, and how much weight each source carries, are in How to hand over AI leadership to an internal owner. Stage 1. The owner and the test Fill this in first. This playbook asks for a written test so that the handover can end on a result, not only on a date. The pilot article says the new owner should have written at least one eval case and watched the rollback rehearsal. The approach page has a related test of whether your engineers can roll back a bad model without waiting on the outside lead. My test has three checks, and I judge each one. The new owner writes an eval case and runs a rollback rehearsal. The new owner also handles one incident without me in the room. | Question | Your answer | | -------------------------------------------------------------------------- | ----------- | | Who takes the role? Write one name. | | | Who decides whether to stop a launch, if that is a different person? | | | What must the new owner do without the outside lead before the role moves? | | | Who judges each check? | | | What counts as a pass? Write it so a third person could judge it. | | | What happens if the new owner is not ready on the planned date? | | Stage 2. What moves My handover takes three months, in three phases. The month-one plan sets the end date, not the contract. | Phase | Months | What happens | | ----- | ------------------ | ----------------------------------------------------- | | 1 | Months one and two | The new owner and the outside lead work side by side. | | 2 | Month three | The new owner leads. The outside lead reviews. | | 3 | End of month three | The final test and the signatures. | One row per item. Fill in who holds it today, who holds it after, where it lives and the date it moved. The reason for each row comes first, and its source where it has one, so you can drop a row you disagree with. | Item | Why it is here | Held today by | Held after by | Where it lives | Done on | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------- | -------------- | ------- | | Decisions and their reasons | A UK regulator says handover material should include judgement and opinion, not just facts and figures. FCA handbook, an analogy from financial firms. | | | | | | Work in progress | Status, owner and next action for each open item. Fractionus, one vendor's advice. | | | | | | Open risks and concerns | The same regulator says the material should include unresolved or possible breaches and unresolved concerns raised by regulators. FCA handbook, an analogy. | | | | | | The list of AI systems | NIST asks for mechanisms to inventory AI systems. NIST AI RMF, Govern 1.6, voluntary. | | | | | | Roles and authority | NIST asks that roles, responsibilities and lines of communication are documented and clear. NIST AI RMF, Govern 2.1, voluntary. | | | | | | Runbook, evals and rollback | Google says training for a receiving team can include hands-on exercises in operations. Google SRE book, the operator's own account. | | | | | | Monitoring and logs | NIST asks for monitoring plans that cover incident response and change management (Manage 4.1). For high-risk systems, deployers will have to keep the logs they control for at least six months. That duty starts on 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems. Article 26 of the EU AI Act. | | | | | | Outside parties | An interim director helps announce the new leader to funders and peers. CompassPoint, a practitioner's guide for US nonprofits. | | | | | | Vendor and sub-processor contacts | Vendor and sub-processor contacts are not personal. They are a row in the AI register, and the renewal calendar moves with it. | | | | | Stage 3. Who signs what NIST AI RMF Govern 2.1 asks that roles, responsibilities and lines of communication are documented and clear. Its voluntary Playbook also suggests documenting delegated authorities. Write that down here for the handover. The rows are examples of decisions that carry a signature; no source lists them. Delete any that do not apply and add your own. For a high-risk system under the EU AI Act, ask counsel who must hold human oversight. | Decision | Signs today | Signs after the handover date | Change takes effect on | | ------------------------------------------- | ----------- | ----------------------------- | ---------------------- | | Entries in the AI register | | | | | Risk acceptances | | | | | Policy exceptions | | | | | Vendor and model approvals | | | | | Changes to eval thresholds | | | | | Rollback calls | | | | | Answers to customer security questionnaires | | | | | Serious incident calls | | | | Stage 4. Access List every account the outside lead holds. Revoke them in one coordinated step on or before the last day, and do not leave removal to the outside lead. A marketplace that sells fractional executives gives this advice and asks for written confirmation. It is one vendor's opinion. The table lists six categories of account. The client's internal owner or IT lead revokes each one, and the revocation is confirmed in writing the same day. A revoke-all line sits on the last-day checklist. | Account or system | Granted by | Revoked by | Confirmed in writing on | | ---------------------------------------- | ---------- | -------------------------------------- | ----------------------- | | The AI register | | The client's internal owner or IT lead | | | The LLM gateway | | The client's internal owner or IT lead | | | CI/CD | | The client's internal owner or IT lead | | | The cloud consoles | | The client's internal owner or IT lead | | | The vendor panels | | The client's internal owner or IT lead | | | The SSO admin | | The client's internal owner or IT lead | | Stage 5. After the last day Sources agree that some reachable help is useful and differ on how much. Agree it in writing, with its length, scope and who may call. The middle column holds my terms where I have set them. | Question | My terms | Your answer | | -------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------- | | Will the outside lead stay reachable, and for which questions? | Yes. Questions by email, answered within one working day. | | | For how long, and in what fee shape? | Thirty days after the last day, with one review call on day 30. | | | Who may call? | | | | What is out of scope after the last day? | Anything beyond the email answers and the day-30 call is a separate agreement. | | Stage 6. The check after handover Pick the date and the person who asks. The first question comes from one vendor's advice to ask the successor about a month after the exit. The other rows are prompts. Add your own. | Question | Date | Who asks | Answer | | -------------------------------------------------------------------- | ---- | -------- | ------ | | What do you wish you had known on day one? | | | | | Which decision since the handover was made without the outside lead? | | | | | Which item in the What moves table turned out to be missing? | | | | | Do customer paperwork and the AI register show the new name? | | | | What it is not It is not a standard. A search in October 2026 found no standard, law or study that sets out how to hand over AI leadership, and no outcome study of AI-leadership handovers. Nothing shows that these tables lead to better results. It is not legal advice, and it is not a compliance certificate. The FAQ on how the Retainer ends covers how the Fractional CAIO Retainer ends. No engagement has yet ended in a full handover by the method in these tables.resourcehttps://meshin.co/resources/agent-evals-ci-starterhttps://meshin.co/resources/agent-evals-ci-starterA starter CI pipeline for agent evals. It runs on every pull request and blocks a bad merge before it ships, using a plain YAML case format.Tue, 08 Sep 2026 00:00:00 GMTWhat this is A CI workflow and an eval case format for one agentic workflow. The same discipline sits inside the 90-Day Production Sprint. A prompt or model change should not merge without passing a test set first. The eval case format | Field | Purpose | Example | | -------- | -------------------------------------------------------------------------- | ------------------------------------------------- | | id | A stable name, so a failure is traceable across runs | refund-double-charge | | input | The exact prompt or tool call the agent receives | Customer says they were charged twice | | expected | A pass rule: an exact match, a check, or a second model acting as a grader | Refund issued, ticket closed, no invented apology | | tags | The failure mode a case guards against | prompt-injection, tool-misuse, refusal | The CI gate The workflow runs the eval set on every pull request, scores it, and blocks the merge under your floor. Nobody has to remember to run it by hand, which is the part that fails first once a team is busy. The tags borrow their shape from OWASP's Top 10 for Large Language Model Applications. Prompt injection and excessive agency (the agent doing something nobody asked it to) appear in a support agent as often as in a public chatbot. A tagged case catches the regression either way. What it is not It is not a considered eval set on its own. A handful of cases pulled from real conversations tests more than a long list invented at a desk. The starter takes no position on which cases you pick. Choosing the cases still maps to the MEASURE function (one of the framework's four functions) in NIST's AI Risk Management Framework (AI RMF).resourcehttps://meshin.co/resources/ai-governance-starter-kithttps://meshin.co/resources/ai-governance-starter-kitAI governance starter kit: the paperwork a security review asks for — register, policy, data classification and a vendor questionnaire.Tue, 08 Sep 2026 00:00:00 GMTWhat this is The four documents that answer most of a corporate security review. They are deliberately short: a register nobody updates is worse than no register. The kit also sets out the order in which to call people when an AI system fails. What it is not It is not a compliance certificate, and it is not legal advice. It is the paperwork a reviewer expects to see before they look at your architecture. Who signs what | Document | What it's for | Who signs it | | ------------------------- | ---------------------------------------------------------- | ------------------------------------------------------- | | AI register | Every AI system in use, who owns it, what data reaches it. | The named AI risk owner keeps it current. | | Acceptable-use policy | What you may paste into a tool, and what you may not. | Every employee who uses an AI tool for work. | | Data classification table | Which vendors may see which tier of data. | The AI risk owner, together with whoever owns the data. | | Vendor questionnaire | One answer set procurement can reuse for every new vendor. | Procurement, before anyone signs a vendor contract. | It maps loosely to what ISO/IEC 42001 and the NIST AI Risk Management Framework (AI RMF) both expect a company to have on file. It does not require either. Vendor questionnaire The other three documents describe how you handle AI. This one asks each vendor how it handles your data. Send the same questions to every AI vendor, and file the answers with the contract. The questions cover these topics: - Whether the vendor trains models on your data, and how you opt out. - Where your data is processed and stored, who at the vendor can read it, and which sub-processors touch it. - How the vendor announces a model change, and whether you can stay on a fixed version. - How the vendor tells you about an incident, and who answers when you call. - Whether the system counts as high-risk under the EU AI Act, and which duties the contract leaves with you. - What happens to your data and prompts when the contract ends. Incident response plan When an AI system fails, its owner should not have to work out who to call. Write this call order down beside the register, with a name and a backup against each role. Inside the company, in this order: 1. The system owner named in the register, who limits or stops the system and keeps its logs and model version as they were. 2. The AI risk owner, who decides whether the incident is serious. 3. An executive who can approve a report and a message to customers. 4. Legal counsel, and whoever handles data breaches if personal data is involved. Outside the company, in this order: 5. The provider of the system, which is often the vendor. 6. The importer or distributor, if the system has one. 7. The market surveillance authority. 8. Affected customers, as your contracts require. For a high-risk system you deploy, Article 26 of the EU AI Act asks the deployer to give notice at once when it identifies a serious incident. The order is the provider first, then the importer or distributor and the market surveillance authority. Log the time each person learned of the incident. The page on serious incident reporting says which moment starts the clock for a report and sets out the deadlines, so this plan does not repeat them.resourcehttps://meshin.co/resources/ai-readiness-scorecardhttps://meshin.co/resources/ai-readiness-scorecardA five-axis AI readiness scorecard you score yourself in an afternoon: strategy, data, delivery, people, and governance.Tue, 08 Sep 2026 00:00:00 GMTWhat this is Five axes, scored from zero to three, that give you a rough position before you pay anyone to check it properly. The axes match the ones in the AI Readiness & Risk Assessment. This is the self-administered version, without the evidence behind it. The five axes | Axis | 0 — not started | 1 — informal | 2 — documented | 3 — evidenced | | ---------- | ---------------------------------------------------------- | -------------------------------------------- | ------------------------------------------ | ------------------------------------------------------- | | Strategy | No one has named which processes to automate first | Someone has an opinion, nothing written down | A ranked shortlist exists | The shortlist is reviewed by whoever owns the budget | | Data | No one knows what a vendor tool can see | A rough sense exists, not written down | A data classification table exists | The table is checked before a new tool is approved | | Delivery | Every pilot has stalled before production | One pilot is close to live | One system is live, unmonitored | A system is live, monitored, with a rollback path | | People | Only the original builder understands any AI system in use | One other person has read the code | A second person has changed it and shipped | Ownership is written down and handed over on a schedule | | Governance | No AI register exists | A register exists, out of date | A register exists, reviewed occasionally | The register is reviewed on a fixed schedule | Scoring it Score each axis from zero to three using the table above, then add the five scores for a total out of fifteen. A low total early on is common. It usually means nobody has written anything down yet, not that the engineering is behind. The five axes are not a substitute for the NIST AI Risk Management Framework (AI RMF) or ISO/IEC 42001. They are a rougher pass at the same territory, built to take an afternoon rather than an audit. What it is not It is not the Assessment, and it carries no evidence a reviewer would accept. A self-given three is a claim until someone outside your team checks what you have actually written down.resourcehttps://meshin.co/resources/build-vs-buy-ai-kithttps://meshin.co/resources/build-vs-buy-ai-kitThe build vs buy AI kit: a one-page worksheet of the six criteria I work through before recommending either path, and how to score them.Tue, 08 Sep 2026 00:00:00 GMTWhat this is A worksheet for one recurring decision: build the workflow yourself, or buy a vendor's version of it. It will not choose for you. It gives the choice a written reason, which is what a board or a buyer asks for later. The six criteria | Criterion | Build tends to win when | Buy tends to win when | | ------------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------- | | Data residency | Customer data cannot leave your infrastructure by contract or by law | A vendor already holds a data-processing agreement you trust | | Customisation depth | The workflow is core to what makes your product different | The workflow looks like everyone else's version of the same task | | Time to a working version | You already have the engineers and the model access | Procurement moves faster than a hiring plan would | | Total cost over years | Usage is high enough that a per-call fee compounds | Usage is uncertain, and a fixed fee caps the downside | | Exit cost | You can rebuild the model calls without rebuilding the product | A second vendor could take over most of the same job | | Ownership when it breaks | An internal team already owns the surrounding process | A vendor's contract has a real answer to who to call | Scoring it Weight the six criteria for your situation, score build and buy against each one, and let the total argue with your gut feeling. A bought tool still needs an owner and a place in your AI register. NIST's AI Risk Management Framework (AI RMF) treats a vendor's model the same as one you trained. That sits under its MAP function (one of the framework's four functions). The risk to your customer does not care who wrote the code. Total cost of ownership The Total cost over years criterion needs a total cost of ownership (TCO) estimate, and a vendor quote covers only part of it. Fill in the categories in the table below, for build and for buy. Use the same horizon for both sides, long enough to cover the longest contract you would sign. For each side, add what you pay once to what you pay in every period of that horizon. | Side | Cost category | What to count | | ----- | ---------------- | --------------------------------------------------------------------------------------------- | | Build | Engineering time | Engineer hours for the first version, and the work those engineers would otherwise ship | | Build | Maintenance | Upkeep as the model API, your libraries and your product change | | Build | Drift monitoring | The evals and review that catch quality slipping as models and data change | | Buy | Licence | The vendor's fee in the unit they meter it, and how it grows with your usage | | Buy | Integration | Connecting the tool to your data, login and workflows, plus the security review | | Buy | Lock-in | The cost of leaving: moving data out, redoing prompts and workflows, retraining staff | Take every amount from your own quotes and payroll, because this page lists no amounts. Use the same usage forecast on both sides, since the criteria table above shows that usage often decides which side costs less. Lock-in is the cost behind the Exit cost criterion, so fill in both together. What it is not It is not a procurement process, and it will not pick the vendor for you. It decides one thing: whether the job belongs inside your codebase or inside someone else's product. This is the call the Fractional CAIO Retainer reviews every quarter, alongside the shadow AI a team already bought without asking.resourcehttps://meshin.co/resources/llm-gateway-governance-confighttps://meshin.co/resources/llm-gateway-governance-configA governance config for an LLM gateway. It decides who may call which model, what gets redacted, how long logs are kept and the rate and fallback rules.Tue, 08 Sep 2026 00:00:00 GMTWhat this is A policy file for whichever LLM gateway you run, hosted or self-built. It is the config, not the gateway itself. The decisions live in a file your team reviews, not in a script only one engineer remembers. The policy schema | Key | What it controls | Example | | -------------- | ------------------------------------------------------------------ | ------------------------------------------------- | | allowed_models | Which providers and model versions a team may call | two named models, not an open list | | redact | Fields stripped before a prompt or response is logged or forwarded | email, national_id, card_number | | rate_limit | Calls per minute before the gateway queues or refuses | a ceiling per team, not per user | | log_retention | How long a request and response pair is kept | matches your data retention policy, not a default | | fallback | What happens if the primary provider is down | a named second provider, or a hard failure | Logging and retention The logging schema borrows its shape from the EU AI Act's Article 12, which requires automatic event logs for high-risk systems. Not every call through a gateway is high-risk. The habit costs little to build in now. It is expensive to retrofit once a regulator or a customer asks first. The `allowed_models` and `log_retention` keys map to two NIST SP 800-53 control families: Access Control and Audit and Accountability. Procurement questionnaires often draw on this catalogue, even when nobody names it. What it is not It is not a gateway, and it will not stop a determined engineer from calling a model directly. It makes an unrouted call a shadow AI finding. That is the layer the 90-Day Production Sprint puts in front of a workflow before it counts as production.resource