Giving Your Teams a Real AI Budget with LiteLLM

The previous article argued that AI governance by policy fails because there is no point of passage where a rule could be applied, and that the way in is to build the accounting layer first: measure, then show back, then govern. This article builds that layer. At the end of it, every call made by a person or a script in your organization is attributed to a named identity and a team, each team has a spending ceiling that resets on a schedule, a daily bound sits underneath the monthly one so a single runaway process cannot consume the month, alerts fire before anything blocks, and the finance team can be handed a per team breakdown that reconciles against a provider invoice.

The tool is LiteLLM, deployed as a proxy. Everything below has been arranged so you can follow it in order on a single machine and end with something you could actually put in front of a team.

What we are building

The shape is a reverse proxy for inference. Your applications, your developer tools and your internal scripts stop holding provider credentials and instead point at one internal endpoint. That endpoint speaks the OpenAI API, which matters more than it sounds: almost every SDK, agent framework and editor integration already speaks it, so adoption costs a base URL and a key rather than a rewrite.

Behind the endpoint, the proxy holds the real provider credentials, decides which upstream a request goes to, applies the limits attached to the caller, writes a spend record to Postgres, and forwards logs to wherever you keep them. The path of a request is: caller presents a virtual key, proxy resolves that key to a user and a team, checks the model is allowed and the budget is not exhausted, forwards to the provider, computes the cost of the response from its price table, writes a spend row, returns the answer.

Postgres is not optional for what we are doing. Without a database the proxy is a routing convenience; with one it becomes the ledger, and the ledger is the entire point.

Why LiteLLM

There are several products in this category and the honest summary is that they differ mostly on hosting model and on which primitives are free.

Hosted routers like OpenRouter are the fastest path to multi-provider access, but they place a third party between you and your prompts and give you no say in where the ledger lives, which is disqualifying for anyone with a data residency obligation. Observability-first tools like Langfuse or Helicone are excellent at showing you what happened and are not, primarily, enforcement points. General purpose API gateways can be made to do this with enough configuration, but you will be modelling budgets and per model pricing yourself.

LiteLLM is chosen here because it is self-hostable, open source, exposes an OpenAI compatible surface so nothing upstream needs to change, and ships the specific primitives this problem needs already built: virtual keys, teams, budgets with reset windows, rate limits, model allowlists, request tags and a spend ledger. One caveat to check before you plan around it: a subset of features, including parts of the SSO and enterprise administration surface, sit behind a commercial license. Verify the current licensing for the specific features you intend to depend on rather than discovering it during rollout.

Deploying the proxy

Start with the official compose file, which brings up the proxy and its database together. The litellm-database image bundles the Prisma toolchain and runs the schema migration on start, which is what you want for a single node deployment.

services:
  litellm:
    image: docker.litellm.ai/berriai/litellm-database:v1.81.9-stable
    ports:
      - "4000:4000"
    environment:
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
      DATABASE_URL: postgresql://litellm:${POSTGRES_PASSWORD}@db:5432/litellm
      STORE_MODEL_IN_DB: "True"
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      GEMINI_API_KEY: ${GEMINI_API_KEY}
    volumes:
      - ./config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml"]
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: litellm
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Two of those variables deserve more attention than they usually get. LITELLM_MASTER_KEY is the administrative credential for the whole proxy: it creates keys, reads everyone's spend, and changes every limit described in this article. It is not an application key and it should never be handed to a team. LITELLM_SALT_KEY encrypts provider credentials stored in the database, and it cannot be rotated after models have been added without making those stored credentials unreadable. Generate both from a real random source, put them in your secret manager on day one, and pin the image to a version tag rather than latest so that a restart is never a surprise upgrade.

The configuration file declares which models exist and under what internal names.

model_list:
  - model_name: chat-default
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: chat-frontier
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: chat-cheap
    litellm_params:
      model: gemini/gemini-2.5-flash
      api_key: os.environ/GEMINI_API_KEY

  - model_name: embeddings
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Name the models by role, not by vendor. chat-default and chat-frontier are internal contracts you can repoint at a different provider later without touching a single caller, which is the same reason you put a hostname in front of a database. If you expose claude-sonnet-5 as the public name, you have leaked a vendor decision into every application in the company and you will pay for it the day you want to change it.

Bring it up and confirm it answers:

docker compose up -d
curl http://localhost:4000/health/liveliness

Identity before budgets

The single most common way this deployment fails is to skip straight to generating keys. Create the teams first, because the team is where the money lives and a key without a team is spend you cannot roll up to anyone.

curl -X POST 'http://localhost:4000/team/new' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "team_alias": "support",
    "max_budget": 400,
    "budget_duration": "30d",
    "models": ["chat-default", "chat-cheap"]
  }'

That returns a team_id, which every key you issue to that department will carry. The team also carries its own ceiling and its own model allowlist, and both are inherited as an outer bound: a key inside the team can be more restricted, never less. This is the structure that makes the finance conversation possible, because a department head can be shown a number that corresponds exactly to a cost center they already own.

Now issue a key to a person.

curl -X POST 'http://localhost:4000/key/generate' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "key_alias": "aminata.support",
    "user_id": "aminata@example.com",
    "team_id": "<team_id from above>",
    "models": ["chat-default", "chat-cheap"],
    "max_budget": 40,
    "budget_duration": "30d",
    "rpm_limit": 60
  }'

One key per person. The temptation to issue a single key per application and let five engineers share it will destroy the attribution you are building, and it will do so silently: the numbers keep coming, they are just no longer about anybody. If several services need access, give each service its own key with a service account identity, so that "the nightly enrichment job" is as attributable as a human.

At real headcount, provisioning keys by hand stops scaling and you want identity to come from your identity provider. LiteLLM supports SSO and can map JWT claims onto users and teams, so that group membership in the IdP becomes team membership on the proxy and a departure handled in the IdP is a revocation on the proxy. Defaults for anyone arriving through SSO are set in configuration:

litellm_settings:
  default_internal_user_params:
    user_role: "internal_user"
    models: ["chat-default", "chat-cheap"]
  default_team_params:
    max_budget: 100
    budget_duration: 30d
    team_member_permissions:
      - "/team/daily/activity"

That last permission is small and worth setting early: it lets ordinary team members see their own team's consumption without an administrator in the loop. Visibility is the mechanism from the first article, and it does not work if the only person who can see the numbers is the platform engineer.

Budgets that reset, and the daily bound

Here is the part that answers the objection every finance director raises, which is that token billing makes a fixed commitment impossible.

A budget with a duration is a recurring ceiling. It is not a prepayment and not a forecast, it is a maximum loss per period, and it resets on its own:

curl -X POST 'http://localhost:4000/key/generate' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "key_alias": "marketing.batch", "max_budget": 50, "budget_duration": "30d" }'

That alone leaves one hole, and it is the hole that actually bites: nothing stops the entire monthly allowance from being consumed in twenty minutes by a misconfigured loop on a Tuesday morning. The month is protected. The month is also over. Stacked budget windows close it:

curl -X POST 'http://localhost:4000/key/generate' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "key_alias": "eng.agent-runner",
    "budget_limits": [
      {"budget_duration": "24h", "max_budget": 20},
      {"budget_duration": "30d", "max_budget": 200}
    ]
  }'

Both windows are enforced. Normal work never approaches the daily bound, so nobody notices it exists; a runaway agent hits it within the hour and stops, having cost you twenty dollars and one afternoon instead of the department's quarter. This is the concrete answer to "we cannot commit a fixed amount": you commit a maximum, per day and per month, and you size it from measurement rather than from anxiety.

Then set the soft budget, which is the part that keeps humans in the loop:

curl -X POST 'http://localhost:4000/key/generate' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "key_alias": "eng.agent-runner", "soft_budget": 150, "max_budget": 200, "budget_duration": "30d" }'

A soft budget triggers an alert rather than a rejection. Wire it to the channel the team lead actually reads, and put real distance between soft and hard so the alert arrives while there is still time to decide something. An alert that fires at ninety-eight percent of the ceiling is not a warning, it is a notification of an outage that has already started.

Rate limits, and why agents need the token one

Budgets bound the money over a period. Rate limits bound the rate, and the two failure modes are different enough that you want both.

curl -X POST 'http://localhost:4000/key/generate' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "key_alias": "eng.agent-runner", "rpm_limit": 120, "tpm_limit": 400000 }'

For conversational traffic, a requests-per-minute limit is the natural guard: a human generates few requests and each is small. For agentic traffic it is close to useless, because the pathological case is not a high request count, it is a modest number of requests each dragging an enormous accumulated context. An agent on its fortieth step may be sending a hundred thousand tokens per call while making one call every few seconds, which sails under any reasonable rpm limit while burning money at a rate no rpm number can express. Cap tokens per minute for anything automated. Cap requests per minute for anything with a person on the other end.

Model tiering

The largest single cost lever is which model serves the request, and the gap between the cheapest capable model and the frontier one is a multiple rather than a margin. Tiering is how you exploit that without holding a review meeting about it.

Default everyone to a fast, inexpensive model that handles the large majority of requests acceptably. Grant the frontier tier at the team level to the roles that demonstrably need it. Since keys inherit the team's allowlist as an outer bound, moving someone up a tier is a change to their team membership or their key, not a negotiation:

curl -X POST 'http://localhost:4000/key/update' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "key": "sk-...", "models": ["chat-default", "chat-cheap", "chat-frontier"] }'

Make this a group membership a manager can grant, not a ticket. The moment the upgrade path involves a queue, people go back to their personal accounts and you lose the traffic, which is the only thing you were ever really protecting.

Fallbacks belong in the same conversation, because they are what makes the internal model names honest. If chat-default is unavailable, the proxy can route to an equivalent rather than returning an error to a user who has no idea what a provider outage is:

router_settings:
  fallbacks:
    - chat-default: ["chat-cheap"]
    - chat-frontier: ["chat-default"]

This is also the mechanism that makes a provider migration a configuration change instead of a project, which matters a great deal in the sovereignty scenario the next article takes up.

Attribution beyond the key

Keys tell you who spent. They do not tell you what for, and the moment someone asks you to bill a client or justify a project's cost, that distinction becomes the whole question. Tags carry the second dimension.

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $VIRTUAL_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "chat-default",
    "messages": [{"role": "user", "content": "summarize this ticket"}],
    "metadata": {
      "tags": ["project:helpdesk-triage", "customer:acme"]
    }
  }'

Applications you control can set tags directly. For everything else, promote a header into a spend tag so that callers you cannot modify still produce attributable traffic:

litellm_settings:
  extra_spend_tag_headers:
    - "x-project-id"
    - "x-customer-id"

Now the same spend can be sliced by person, by team, by project and by end customer, from one ledger. This is what turns the platform from a cost control into something the business asks for, because per customer cost is the input to pricing decisions that were previously made by feel.

Reading the money

The ledger is queryable. The aggregated view is what you send to a department head:

curl -G "http://localhost:4000/spend/logs" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  --data-urlencode "start_date=2026-08-01" \
  --data-urlencode "end_date=2026-08-31"

Passing summarize=false returns individual transactions instead, each carrying the key, the user, the team and any tags. That per row view is what you need for an audit question, for reconciling an anomaly, and for the first month of the rollout when the interesting output is not a total but a distribution.

For continuous reporting, send the events somewhere that does dashboards properly rather than building them on the proxy. LiteLLM emits to the usual destinations through its callback system, including OpenTelemetry, so LLM spend can land in the same Grafana the rest of your platform already uses. That is the cheapest possible integration for most teams, because the pipeline already exists and cost simply becomes one more signal next to latency and error rate.

The daily activity endpoints are the ones to expose to teams themselves. Self service visibility is what makes the optimization phase happen without you.

Guardrails, and what to log

Once every prompt passes through one place, that place can inspect them, and you have to decide deliberately what it does with that power.

Presidio-based guardrails detect and act on personal data before the request leaves your infrastructure:

guardrails:
  - guardrail_name: "pii-mask"
    litellm_params:
      guardrail: presidio
      mode: "pre_call"
      presidio_filter_scope: both
      presidio_score_thresholds:
        ALL: 0.7
        CREDIT_CARD: 0.8
      pii_entities_config:
        CREDIT_CARD: "MASK"
        EMAIL_ADDRESS: "MASK"
        IBAN_CODE: "BLOCK"

Masking replaces the detected entity before the call goes upstream; blocking rejects the request outright. Use blocking sparingly and only for categories where the correct answer is genuinely "never", because a guardrail that blocks legitimate work teaches people to route around the proxy, and detection is statistical rather than exact. Treat it as risk reduction, not as a compliance guarantee, and be careful about how you describe it internally, because "we mask PII" is heard as "we cannot leak PII" by everyone who is not in the room.

The logging decision is the more consequential one. By default, logging integrations can record message content, which means your observability stack becomes a second copy of every prompt in the company, including the ones containing credentials someone pasted by accident. For most organizations the right default is to log metadata and keep the content out:

litellm_settings:
  turn_off_message_logging: true

You lose the ability to debug a bad response by reading it, which is a real cost. You avoid building a searchable archive of everything your employees have ever asked, which is a real liability and, under some regimes, a processing activity in its own right. If you need content logging for a specific workflow, scope it to that workflow deliberately, with a retention period and a named owner, rather than leaving it on globally because it was the default.

Day two

Three things will surprise you after this is running, and they are worth planning for rather than discovering.

The numbers will not match the invoice. The proxy computes cost from a price table maintained in software, and it will drift from what the provider actually bills: cached prefixes discounted, batch tiers, enterprise rates you negotiated, a price change shipped on a Tuesday. The gateway's figures are excellent for relative comparison and allocation between teams, which is what you use them for. They are not accounting truth. Assign someone the monthly reconciliation against the real invoice and treat a widening gap as a signal that a price table or a routing assumption has gone stale.

The proxy is now a single point of failure and an extremely interesting target. Every AI-dependent workflow in the company fails when it does, so it needs more than one replica, a health check that means something, and a latency budget, since you have inserted a hop into every inference call. It also holds every provider credential and sees every prompt, which makes it a higher value target than most internal services. Its threat model deserves the treatment you would give an identity provider, not the treatment you would give an internal dashboard.

Keys need a lifecycle. They leak into shell history, CI configuration and laptops. Set expiry at creation, rotate on a schedule, revoke on departure, and prefer SSO-derived keys precisely because they inherit a lifecycle you already operate. A key that outlives its owner is the exact failure the whole exercise was meant to prevent.

Recap

The order matters more than any individual setting, and it mirrors the sequence from the first article. Deploy the proxy with its database and pin the version. Create teams before keys, and one key per identity, human or service. Set generous limits and measure for a month without blocking anything. Publish the per team numbers and let the obvious waste get fixed without instruction. Then set budgets from what you measured, with a daily bound stacked under the monthly one, a soft alert far enough below the hard ceiling to be actionable, token-per-minute limits on anything automated, and a default model tier that most people never need to leave.

What you have at that point is the attribution layer, which was the tractable one of the three unknowns. The next article takes up the second one, exposure, in a setting where it stops being a matter of internal policy: what a gateway does and does not change when the prompts leaving your building are personal data and the law asking about them is Togo's 2019-014.

Sources

• LiteLLM documentation, Proxy deployment: images, environment variables, LITELLM_SALT_KEY semantics and version pinning. docs.litellm.ai/docs/proxy/deploy
• LiteLLM documentation, Cost tracking and spend logs: /spend/logs, custom metadata, request tags and extra_spend_tag_headers. docs.litellm.ai/docs/proxy/cost_tracking
• LiteLLM documentation, Users, teams and budgets: /team/new, /key/generate, budget_duration, stacked budget_limits, soft_budget, rpm and tpm limits. docs.litellm.ai/docs/proxy/users
• LiteLLM documentation, Model access control and JWT auth: team based model access and SSO defaults. docs.litellm.ai/docs/proxy/model_access
• LiteLLM documentation, Guardrails: PII masking with Presidio. docs.litellm.ai/docs/proxy/guardrails/pii_masking_v2