LLM Gateway Governance Config
A governance config for an LLM gateway. It decides who may call which model, what gets redacted, how long logs are kept and the rate and fallback rules.
Available as a page on this site. A repository copy is not published yet.
Published · Updated
What is inside
How to use it
Use the schema on this page to build your own policy file, with one row per team or workflow that calls a model. Start with whatever is already in production. Route every model call through the gateway before a second application starts talking to a model directly. Review your file on the same schedule as your AI register.
In my words
What this is
A policy file for whichever LLM gateway you run, hosted or self-built. It is the config, not the gateway itself. The decisions live in a file your team reviews, not in a script only one engineer remembers.
The policy schema
| Key | What it controls | Example |
|---|---|---|
| allowed_models | Which providers and model versions a team may call | two named models, not an open list |
| redact | Fields stripped before a prompt or response is logged or forwarded | email, national_id, card_number |
| rate_limit | Calls per minute before the gateway queues or refuses | a ceiling per team, not per user |
| log_retention | How long a request and response pair is kept | matches your data retention policy, not a default |
| fallback | What happens if the primary provider is down | a named second provider, or a hard failure |
Logging and retention
The logging schema borrows its shape from the EU AI Act’s Article 12, which requires automatic event logs for high-risk systems. Not every call through a gateway is high-risk. The habit costs little to build in now. It is expensive to retrofit once a regulator or a customer asks first.
The allowed_models and log_retention keys map to two NIST SP 800-53 control families: Access Control and Audit and Accountability. Procurement questionnaires often draw on this catalogue, even when nobody names it.
What it is not
It is not a gateway, and it will not stop a determined engineer from calling a model directly. It makes an unrouted call a shadow AI finding. That is the layer the 90-Day Production Sprint puts in front of a workflow before it counts as production.
Sources and neighbours
Sources
- Regulation (EU) 2024/1689 (Artificial Intelligence Act) — Article 12, record-keeping
EUR-Lex · retrieved 2026-09-08
- NIST SP 800-53 Rev. 5 — Security and Privacy Controls for Information Systems and Organizations
NIST · retrieved 2026-09-08
Related terms