AI Data Plane architecture

The AI Data Plane runs as software in your own infrastructure, on the path between your AI clients and your AI inference provider. This page describes how it is shaped, what it touches, and how it behaves when something fails, so your operations team can vet the design before anyone installs it.

This is the architecture, not the install. The Helm chart is the reference for workloads, ports, values, and secrets, and your account manager provides it along with its values documentation.

For what the AI Data Plane does and why, see AI Data Plane. For every field it sends to CloudZero, see AI Data Plane telemetry schema.

Where the boundary sits

Everything that touches your inference traffic runs in your cluster. CloudZero runs the ingest that receives telemetry and the control plane the deployment registers with. Nothing else crosses the line.

CloudZero does not route your inference, resell it, or stand between you and your providers.

Two paths through the deployment

Separating these two paths is the fastest way to reason about the deployment, because they have different requirements and different failure modes.

The inference path

A client's request on its way to a model. It arrives at your load balancer, reaches the data plane's proxy, and is forwarded to the inference endpoint you configure, which serves the request exactly as it does today.

Your router decides what enters this path at all. Traffic you route to the data plane is proxied and analyzed. Traffic you do not route to it reaches your inference endpoint untouched and produces no telemetry. You can start with one team or one workload and widen from there.

TLS terminates in one of two places, and you choose which. Either your load balancer terminates it and forwards plaintext to the proxy inside the cluster, in which case no private key is mounted into a CloudZero container, or the proxy terminates it itself from a certificate you supply. The first is the simpler posture; the second suits a cluster where in-cluster plaintext is not acceptable.

The telemetry path

Outbound only. The context engine exports OpenTelemetry spans to the egress collector, which exports to CloudZero over HTTPS.

Nothing on this path needs an ingress, a certificate, or a DNS record. The collector is reachable only inside the cluster.

What runs in your cluster

Four roles, each a separate workload:

The session cache is a Dragonfly instance, deployed by the Dragonfly operator. That operator must already be present in your cluster, and it is the one cluster-level prerequisite the chart does not install for you.

Two of the four hold your CloudZero credential: the egress collector, to export telemetry, and the coordinator, to register the deployment. The proxy, the context engine, and the cache never see it.

How it scales

The context engine holds no in-process state. Turn continuity rides on the session cache, so any pod serves any round trip and the deployment scales horizontally. It runs two replicas by default, and an autoscaler is available when you have load data to set the bounds with.

Two properties follow from that design and are worth confirming against your own standards:

• No databases and no persistent volumes. The only state is in the cache, and it is losable by design. Losing it costs a broken trace, never a blocked request.
• The coordinator runs alone. It is off the telemetry path and does no work per request, so a single instance is sufficient.

How it fails

The governing rule is that no component stops your traffic when it fails. Every part fails open.

Classification runs off the forwarding path. The one step on the request itself is adding trace context, which is what lets cost be joined to attribution later, and it runs against a short budget. Exceed the budget, or lose the context engine entirely, and the request is still forwarded. The cost is a gap in attribution coverage.

Fail-open covers the classification machinery. It does not make the proxy itself redundant: the proxy is in the request path for the traffic you route to it, so give it the capacity planning, health checking, and rollout care you would give any in-path proxy. An unreachable inference endpoint, or invalid TLS and routing configuration, still prevents inference.

What leaves your network

Four outbound destinations, all initiated from inside your cluster:

The telemetry is prompt-free. AI Data Plane telemetry schema lists every field in it, and what never appears.

Running in front of an AI gateway

Everything above holds for any HTTP inference endpoint. Today that endpoint is an AI gateway you run, and the architecture does not require one: the proxy forwards to a host you name, and a provider's API is the same shape.

What a gateway adds is the second telemetry stream. The AI Data Plane knows which work each turn was doing; it does not price that work. Cost comes from the gateway, which exports its own usage telemetry, and the two join on a shared trace ID. A provider API exports nothing to your collector, so an endpoint that is not a gateway yields attribution without the cost joined to it.

A gateway qualifies when it exports OpenTelemetry usage spans and propagates W3C trace context through to them. Nothing about the AI Data Plane is specific to one gateway. See AI Data Plane telemetry schema for both streams and the join between them.

Three things change in this topology:

• The telemetry path gains a second sender. Your gateway exports its usage spans to the egress collector alongside the context engine's. The collector keeps the two streams separate all the way to CloudZero.
• The gateway enrolls as a device. The coordinator reports it to CloudZero as an attached device, so it appears alongside the deployment with its own status.
• A gateway outside the cluster needs a route in. The collector is reachable only inside the cluster, so you provide that route.

The gateway and the collector reference each other, which looks circular and is not. Each names the other in configuration, not at startup, so either can come up first. A missing peer surfaces as export errors or a failed request, clears when the other side arrives, and never crashes anything.

What you provide

• A load balancer or ingress in front of the data plane, with its certificate and DNS.
• A router that decides which traffic reaches the data plane and which bypasses it.
• Your inference endpoint. To see cost as well as attribution, that endpoint is a gateway exporting its own usage telemetry to the collector.
• A classifier endpoint and credential, pointed at your inference endpoint directly rather than at the data plane, so classification requests are not themselves captured and classified.
• The cache operator, installed in the cluster before the chart.
• A route to the collector if your gateway runs outside the cluster.

Once running, the deployment appears as a device in AI Signals, alongside your gateway when you run one, where you can confirm status, version, and last-seen time.