Connecting to LiteLLM

Connect your LiteLLM gateway to CloudZero to bring your AI inference usage into a unified view of all your cloud and AI spend. The integration streams model usage, token counts, and cost per request from your gateway so you can see AI activity in near real time and organize it into the categories (called Dimensions) that matter most to your business: business unit, team, product, feature, environment, customer, or virtually anything else. Once connected and organized, you can answer questions about your AI spend that matter to your stakeholders.

This guide covers configuring a LiteLLM gateway to send telemetry to CloudZero. The integration has two components you install together:

• The CloudZero telemetry plugin runs inside your gateway. It reads each request and derives business context for cost attribution (project, task, repo, branch, ticket) without sending raw prompt content.
• The OTEL exporter is LiteLLM's built-in otel callback, pointed at CloudZero. It ships the plugin's telemetry to CloudZero.

You register both as callbacks (cz_telemetry.cz_telemetry_instance and otel) and point the exporter at CloudZero. The plugin produces the enriched telemetry, and the exporter delivers it. Both are required: the plugin does not deliver anything on its own.

What you need

• CloudZero user with permission to create API keys
• A LiteLLM gateway you control, running v1.81.0 or later. The examples use the litellm-database image at v1.88.0.
• Python 3.10 or later (included in the LiteLLM Docker images)
• The CloudZero plugin wheel (cz_litellm_telemetry_plugin-<version>-py3-none-any.whl), provided by your CloudZero account team
• Outbound HTTPS connectivity from your gateway host to telemetry.cloudzero.com

The plugin has no runtime dependencies beyond what LiteLLM already provides (OpenTelemetry and litellm).

Step 1: Create a CloudZero API key

1. In CloudZero, go to Settings > API Keys.
2. Select Create API Key and give it a descriptive name (for example, "LiteLLM Telemetry").
3. Under Scopes, open the Ai-Telemetry-Ingest category and select ingest-litellm-v1 (the full scope is ai-telemetry-ingest:ingest-litellm-v1). You can type litellm in the scope search box to find it.
4. Copy the key value and store it securely. You set it as an environment variable in a later step. Treat it like any other secret.

Step 2: Install the plugin and register callbacks

The plugin installs into the Python environment that runs LiteLLM and registers as a callback alongside otel. Installation has three parts:

1. Install the .whl into the Python environment that runs LiteLLM.
2. Create a callback shim, a small .py file that re-exports the plugin's callback instance.
3. Register both callbacks in your LiteLLM config.

LiteLLM's proxy loads custom callbacks by looking for a .py file relative to the config directory. It does not resolve installed Python packages directly, so the shim bridges that gap.

Create the callback shim

cz_telemetry.py is the same for every deployment type:

"""Shim: re-exports the plugin's callback instance for LiteLLM's file-based loader."""

from cz_litellm_telemetry_plugin import cz_telemetry_instance  # noqa: F401

Register the callbacks

In your litellm_config.yaml, register both the plugin and otel:

litellm_settings:
  callbacks:
    - cz_telemetry.cz_telemetry_instance
    - otel
  # Keeps raw prompt/response content out of LiteLLM's native OTEL spans.
  turn_off_message_logging: true

cz_telemetry.cz_telemetry_instance is the plugin (it enriches the telemetry), and otel is the exporter (it ships the telemetry to the CloudZero endpoint you set in Step 3). The plugin will not deliver anything without otel.

Step 3: Point the exporter at CloudZero

Configure CloudZero as the OTLP traces destination on your gateway process. These six settings are the same regardless of how your gateway is deployed. What changes is where you put them (Step 4 shows each option).

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://telemetry.cloudzero.com/v2/telemetry/litellm
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS="Authorization=<your-cloudzero-api-key>"
USE_OTEL_LITELLM_REQUEST_SPAN=true
OTEL_SERVICE_NAME=<friendly-gateway-name>
OTEL_RESOURCE_ATTRIBUTES=cloud.account.id=<your-account-id>

Step 4: Install for your deployment

Each option below covers the wheel install, shim placement, and where to set the environment variables.

Option 1: Pip (bare metal or VM)

The simplest path when you run LiteLLM directly on a host or VM.

# Install the plugin and OpenTelemetry SDK (required for span export)
pip install cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl \
    opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Place cz_telemetry.py in the same directory as your litellm_config.yaml. Set the environment variables in your shell or process manager, then start LiteLLM:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://telemetry.cloudzero.com/v2/telemetry/litellm
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=<your-cloudzero-api-key>"
export USE_OTEL_LITELLM_REQUEST_SPAN=true
export OTEL_SERVICE_NAME=<friendly-gateway-name>
export OTEL_RESOURCE_ATTRIBUTES=cloud.account.id=<your-account-id>
litellm --config litellm_config.yaml

Option 2: Docker

Build a custom image that bundles the wheel and the shim. Place the .whl and cz_telemetry.py next to your Dockerfile:

FROM ghcr.io/berriai/litellm-database:v1.88.0

COPY litellm_config.yaml /app/config.yaml

# Install the CloudZero telemetry plugin (the litellm image has no pip,
# so extract the wheel directly into the virtualenv's site-packages)
COPY cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl /tmp/
RUN python3 -m zipfile -e /tmp/cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl \
        /app/.venv/lib/python3.13/site-packages/ \
    && rm /tmp/cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl

# Callback shim for LiteLLM's file-based loader
COPY cz_telemetry.py /app/cz_telemetry.py

docker build -t litellm-with-cz-telemetry .
docker run -p 4000:4000 \
  -e OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://telemetry.cloudzero.com/v2/telemetry/litellm \
  -e OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
  -e "OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-cloudzero-api-key>" \
  -e USE_OTEL_LITELLM_REQUEST_SPAN=true \
  -e OTEL_SERVICE_NAME=<friendly-gateway-name> \
  -e "OTEL_RESOURCE_ATTRIBUTES=cloud.account.id=<your-account-id>" \
  litellm-with-cz-telemetry --config /app/config.yaml --port 4000

Option 3: Docker Compose

Same approach as Docker: build a custom image that includes the plugin.

Directory layout:

litellm/
├── Dockerfile
├── litellm_config.yaml
├── docker-compose.yaml
├── cz_telemetry.py
└── cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl

Use the same Dockerfile as Option 2. The Step 3 export variables go in the environment block of docker-compose.yaml:

services:
  litellm:
    build: .
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"
    environment:
      - LITELLM_MASTER_KEY=sk-your-master-key
      - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://telemetry.cloudzero.com/v2/telemetry/litellm
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-cloudzero-api-key>
      - USE_OTEL_LITELLM_REQUEST_SPAN=true
      - OTEL_SERVICE_NAME=<friendly-gateway-name>
      - OTEL_RESOURCE_ATTRIBUTES=cloud.account.id=<your-account-id>
      # Optional plugin tuning, see Optional: Tune the plugin below
      # - CZ_CONTEXT_EXTRACTION=false

docker compose up --build

Option 4: Kubernetes and Helm

LiteLLM's official Helm chart uses the ghcr.io/berriai/litellm-database image. Build a custom image (as above) and point the chart at it. The Helm chart mounts config at /etc/litellm/, so place the shim there too:

FROM ghcr.io/berriai/litellm-database:v1.88.0

COPY cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl /tmp/
RUN python3 -m zipfile -e /tmp/cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl \
        /app/.venv/lib/python3.13/site-packages/ \
    && rm /tmp/cz_litellm_telemetry_plugin-0.1.0-py3-none-any.whl

# The Helm chart mounts config at /etc/litellm/, so LiteLLM looks for the shim there
COPY cz_telemetry.py /app/cz_telemetry.py
COPY cz_telemetry.py /etc/litellm/cz_telemetry.py

docker build -t your-registry/litellm-with-cz-telemetry:v1.88.0 .
docker push your-registry/litellm-with-cz-telemetry:v1.88.0

Override the image in your Helm values and register the callbacks. Set the export variables on the pod. A Kubernetes Secret is recommended for OTEL_EXPORTER_OTLP_HEADERS:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://telemetry.cloudzero.com/v2/telemetry/litellm
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-cloudzero-api-key>
USE_OTEL_LITELLM_REQUEST_SPAN=true
OTEL_SERVICE_NAME=<friendly-gateway-name>
OTEL_RESOURCE_ATTRIBUTES=cloud.account.id=<your-account-id>

# values.yaml
image:
  repository: your-registry/litellm-with-cz-telemetry
  tag: v1.88.0

masterkey: sk-your-master-key

proxy_config:
  litellm_settings:
    callbacks:
      - cz_telemetry.cz_telemetry_instance
      - otel
    turn_off_message_logging: true

helm install litellm oci://docker.litellm.ai/berriai/litellm-helm -f values.yaml

Step 5: Restart and verify

1. Restart your LiteLLM gateway so the new config and environment variables take effect.

2. Confirm the plugin loaded by checking the gateway startup logs for the callback registration:

   Callback: cz_telemetry.cz_telemetry_instance

3. Make sure normal request traffic is hitting the gateway. To verify quickly, send a test request:

   Bash

   curl -s http://localhost:4000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer sk-your-master-key" \
     -d '{
       "model": "gpt-4o",
       "messages": [{"role": "user", "content": "Hello"}]
     }'

4. Open AI Signals in CloudZero and watch for activity from your OTEL_SERVICE_NAME.

You should see activity within a few minutes of the gateway receiving requests. If nothing appears after 5 minutes of confirmed gateway traffic, see Troubleshooting.

What to expect

After you connect, your LiteLLM usage appears in AI Signals within a few minutes, and reconciled costs settle within about 24 hours as they are matched against your provider billing.

From there, your LiteLLM costs work like any other cost data in CloudZero. For example, you can explore them in the Explorer, organize them by team or product using Dimensions, track trends in Dashboards, set Budgets, or ask questions in the AI Hub.

Optional: Tune the plugin

All plugin configuration is done through environment variables. None are required, since the plugin works with defaults.

Context extraction is on by default, so project, task, department, and topic attributes are populated automatically. Activity classification is always on: model-classified by default, or keyword-matched when CZ_CONTEXT_EXTRACTION=false. Combine the activity and department attributes to break down spend by both. Requests that cannot be classified are labeled unknown.

When extraction is on, the raw <cz-context> marker is always removed from responses and replaced with a short, human-readable telemetry note (for example, _CloudZero usage telemetry · project: … · activity: … · ticket: …_), so the recorded tag stays visible to the end user rather than silently disappearing. When no displayable field was reported, the marker is simply removed.

Recommended: Tag virtual keys for per-user and per-team attribution

If your team uses LiteLLM virtual keys (one key per user, application, team, or tenant), set the right attribution fields when you mint each key. CloudZero uses these fields to break down usage. Without them, all traffic from a given model collapses into a single bucket and you lose per-user, per-team, and per-application visibility.

The plugin's context extraction adds richer attribution on top of virtual-key tags. It does not replace them.

There are three attribution dimensions, set in different places:

Example: mint a key with user and tag attribution.

curl -X POST 'http://your-litellm-gateway/key/generate' \
  -H 'Authorization: Bearer sk-master-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "user_id": "[email protected]",
    "metadata": {
      "tags": ["team:platform", "project:ai-gateway", "env:prod"]
    },
    "models": ["claude-sonnet", "gpt-4o"]
  }'

For more detail, see LiteLLM's documentation:

• Virtual Keys: /key/generate reference, including user_id and metadata
• Request Tags for Spend Tracking: key-level and request-level tagging
• Customers and End-Users: how end-user identity is passed per-request

Data privacy

The configuration in this guide keeps raw prompt and response content out of CloudZero. If you followed Step 2, both protections are already in place, and none of the dimensions CloudZero uses for cost allocation or chargeback depend on message content.

• The plugin exports only metadata. With CZ_CONTENT_CAPTURE=metadata_only (the default), the plugin's telemetry never carries raw message text, only message and turn counts, lengths, SHA-256 fingerprints, client tool, repo, branch, ticket, and (by default) the derived project, task, department, and topic.
• LiteLLM's native content logging is off. turn_off_message_logging: true (set in Step 2) suppresses the prompt and completion content LiteLLM's otel integration would otherwise capture by default.

The plugin reads the un-redacted request inside your proxy to derive attribution, then exports only metadata, so keeping content out costs you no business context. Repo, branch, ticket, and conversation shape all remain.

If you want raw text for debugging in a trusted environment, CZ_CONTENT_CAPTURE also supports hashed and preview (see Optional: Tune the plugin).

What gets sent to CloudZero

For each LLM request that flows through your gateway, CloudZero receives:

• Timestamp and latency
• Model name (deployment model and requested model)
• Provider (Anthropic, OpenAI, and so on)
• Input, output, and cache token counts
• Cost as calculated by LiteLLM (CloudZero does not recompute cost)
• Virtual key alias and user_id if set
• Success or error status

The plugin adds derived business-context attributes: message and turn counts, content lengths and SHA-256 fingerprints, client tool, repo, branch, ticket, and (by default) project, task, department, and topic. No raw prompt or response text is included under the default CZ_CONTENT_CAPTURE=metadata_only. For how raw content is kept out, see Data privacy.

Network requirements

The LiteLLM gateway must be able to make HTTPS requests to telemetry.cloudzero.com on port 443. CloudZero does not publish a static IP range for this endpoint (it sits behind infrastructure with dynamic IPs), so allowlist by hostname, not by IP. No inbound connections from CloudZero are required.

Troubleshooting

If data is not appearing 5 minutes after confirmed gateway traffic, work through these in order, starting with connectivity. The OTEL exporter reports success at startup even when it cannot reach CloudZero, so a clean startup log does not prove data is leaving your network.

Confirm connectivity to CloudZero (start here)

OTLP over HTTP export failures surface only at export time, not when the exporter initializes, and depending on your logging configuration they may not appear in your gateway logs at all. Confirm connectivity directly with a probe.

Run this from inside the gateway's network context (the same host, pod, or container that runs LiteLLM) so it traverses the same egress path (firewall, forward proxy, and any service-mesh sidecar) your telemetry uses:

curl -i -X POST https://telemetry.cloudzero.com/v2/telemetry/litellm \
  -H "Authorization: <your-cloudzero-api-key>" \
  -H "Content-Type: application/x-protobuf" \
  --data-binary $'\x00'

From a Kubernetes pod:

kubectl exec -it <litellm-pod> -- \
  curl -i -X POST https://telemetry.cloudzero.com/v2/telemetry/litellm \
    -H "Authorization: <your-cloudzero-api-key>" \
    -H "Content-Type: application/x-protobuf" \
    --data-binary $'\x00'

If curl is not in the image, attach an ephemeral debug container that shares the pod's network (for example, kubectl debug -it <litellm-pod> --image=curlimages/curl --target=<container>).

Read the result:

When connectivity is blocked

If the probe times out or is refused, telemetry is being dropped before it reaches CloudZero. The firewall and the service mesh are enforced independently, so an allow rule in one does not satisfy the other. Check both.

1. Firewall and egress allowlist. The gateway needs outbound HTTPS (TCP 443) to telemetry.cloudzero.com. CloudZero publishes no static IP range, so allowlist by hostname, not IP.

2. Service mesh egress (Istio, Linkerd, Consul, and similar). A mesh with restricted egress drops traffic to any host it does not know about, regardless of your firewall allowlist. This is a common cause of "exporter starts clean, but nothing arrives." If the gateway runs in a mesh, add an egress rule for the host. For Istio (when the outbound traffic policy is REGISTRY_ONLY):

   YAML

   apiVersion: networking.istio.io/v1beta1
   kind: ServiceEntry
   metadata:
     name: cloudzero-telemetry
   spec:
     hosts:
       - telemetry.cloudzero.com
     ports:
       - number: 443
         name: https
         protocol: TLS
     resolution: DNS
     location: MESH_EXTERNAL

   If your sidecar originates or terminates TLS for egress, you may also need a matching DestinationRule. Confirm with your platform team.

3. Forward or corporate proxy. If egress goes through an HTTP or HTTPS proxy, the exporter must use it. Set HTTPS_PROXY (and NO_PROXY where appropriate) in the gateway's environment. A proxy that inspects TLS must present a certificate the gateway trusts, or the export fails with a TLS error.

4. Re-run the connectivity probe after each change until it returns HTTP 202. Until the probe passes, gateway configuration is not the problem.

Plugin not loading

If you see ImportError: Could not import cz_telemetry_instance, LiteLLM cannot find the shim or the wheel. Confirm both:

1. The cz_telemetry.py shim file exists in the same directory as your config (in Docker, this is /app/; with the Helm chart, /etc/litellm/).

2. The .whl is installed in the same Python environment. Verify with:

   Bash

   python -c "import cz_litellm_telemetry_plugin; print('OK')"
   # In Docker:
   docker exec <container> python -c "import cz_litellm_telemetry_plugin; print('OK')"

Common misconfigurations

1. Only one callback registered. Both cz_telemetry.cz_telemetry_instance and otel must be in callbacks. The plugin enriches spans, and otel exports them. With the plugin alone, nothing reaches CloudZero.
2. Wrong protocol. gRPC fails silently. OTEL_EXPORTER_OTLP_PROTOCOL must be http/protobuf.
3. Bearer prefix on the API key. The Authorization header value must be the raw key (for example, Authorization=sk-cz-...), not Authorization=Bearer sk-cz-....
4. Wrong endpoint variable that appends /v1/traces. This is the most common cause of 403 and 404 errors. Use OTEL_EXPORTER_OTLP_TRACES_ENDPOINT set to …/v2/telemetry/litellm. Do not use OTEL_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT, because LiteLLM and the OTEL SDK append /v1/traces, producing the wrong path.
5. No endpoint variable. With no OTLP endpoint set, the exporter defaults to localhost:4317 and traces ship to nowhere.
6. Gateway not restarted after a config change. Config and environment variables are read at gateway startup. Restart the gateway after any change.

If context attributes are missing, confirm CZ_CONTEXT_EXTRACTION has not been set to false. Context attributes currently populate reliably on streaming responses only. Non-streaming responses can drop extracted context.

Check your gateway logs for OTEL delivery errors

If the config looks correct and data still is not flowing, tail your LiteLLM gateway logs. OTEL exporter failures (network errors, auth rejections, bad endpoints) surface there:

# Docker or Compose
docker logs -f <litellm-container>

# Kubernetes
kubectl logs -f <litellm-pod>

# Systemd or bare process
journalctl -u <litellm-service> -f

Look for messages from the OTEL exporter around the time you sent a test request.