---
updatedAt: 2026-05-08T19:20:41.000Z
agentTools:
  projectIndex: https://docs.cloudzero.com/llms.txt
---

# Advanced Kubernetes Agent Configuration

The CloudZero Agent for Kubernetes is configured through a Helm values file. The [install page](/docs/installation-of-cloudzero-agent-for-kubernetes) covers the required parameters and optional label configuration. This page covers additional configuration for production environments and specific infrastructure requirements.

All configuration on this page is applied through your `values.yaml` file. After making changes, apply them with:

```shell
helm upgrade cloudzero-agent cloudzero/cloudzero-agent \
    --namespace cloudzero \
    -f values.yaml
```

## Annotations

The install page covers [label configuration](/docs/installation-of-cloudzero-agent-for-kubernetes#step-4-optional-configure-kubernetes-labels). Annotations use the same pattern but are disabled by default.

To collect annotations, add the following to your `values.yaml`:

```yaml
insightsController:
  annotations:
    enabled: true
    patterns:
      - '.*'
    resources:
      pods: true
      namespaces: true
```

| Parameter   | Default          | Description                                                                                                                                  |
| ----------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`   | `false`          | Annotations are not collected unless explicitly enabled.                                                                                     |
| `patterns`  | `['.*']`         | Regular expressions matched against annotation keys. `'.*'` captures all annotations.                                                        |
| `resources` | pods, namespaces | Resource types to collect annotations from. Supported types: pods, namespaces, deployments, statefulsets, nodes, jobs, cronjobs, daemonsets. |

### Label and annotation disambiguation

When CloudZero collects labels and annotations from multiple resource types, it uses prefixes to distinguish the source:

| Source                | Format in CloudZero                    | Example                            |
| --------------------- | -------------------------------------- | ---------------------------------- |
| Pod labels            | No prefix (for backward compatibility) | `team`                             |
| Other resource labels | `resource:label_key`                   | `node:team`                        |
| Annotations           | `resource:annotation:key`              | `namespace:annotation:cost-center` |

<Callout icon="ℹ️" theme="info">
  CloudZero supports a maximum of 300 labels, annotations, and tags across all Kubernetes integrations. Labels and annotations on resources other than pods require agent version 1.0.0 or later.
</Callout>

## Use a Kubernetes Secret for the API key

Storing the API key in a Kubernetes Secret keeps credentials out of your command history and Helm release metadata. For stronger security, the agent also supports [External Secrets Operator](https://github.com/Cloudzero/cloudzero-charts/tree/develop/charts/cloudzero-agent/docs/secret-store-operator-aws-example.md) for integration with external secrets managers.

1. Create the secret:

```shell
kubectl create secret -n cloudzero generic cloudzero-api-key \
    --from-literal=value=<CLOUDZERO_API_KEY>
```

2. Reference the secret in your `values.yaml`:

```yaml
existingSecretName: cloudzero-api-key
```

When `existingSecretName` is set, the `apiKey` parameter is not required.

## Sizing

The default resource requests are sufficient for most clusters. For larger clusters, the agent's memory requirements scale with node count:

| Cluster size    | Recommended memory request | Recommended memory limit |
| --------------- | -------------------------- | ------------------------ |
| Up to 100 nodes | 512Mi (default)            | 1024Mi (default)         |
| 200 nodes       | 2048Mi                     | 2560Mi                   |
| 500 nodes       | 4352Mi                     | 4864Mi                   |

The formula is: `512Mi + (nodes / 100) x 768Mi`.

<Callout icon="ℹ️" theme="info">
  High-churn workloads (frequent pod creation and deletion) or nodes running large numbers of pods may require additional memory beyond this formula.
</Callout>

To adjust resource requests, add the following to your `values.yaml`:

```yaml
components:
  agent:
    resources:
      requests:
        memory: "2048Mi"
      limits:
        memory: "2560Mi"
```

## Custom TLS certificates

By default, the agent generates a self-signed certificate for its admission webhook. To use your own certificate or integrate with cert-manager, add the following to your `values.yaml`:

```yaml
insightsController:
  tls:
    enabled: true
    crt: ""
    key: ""
    secret:
      create: true
      name: ""
    mountPath: /etc/certs
    caBundle: ""
    useCertManager: false
```

| Parameter        | Default      | Description                                                                                                                                                |
| ---------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `crt`            | `""`         | TLS certificate content. If empty, the chart generates a self-signed certificate.                                                                          |
| `key`            | `""`         | TLS private key. If empty, auto-generated with the certificate.                                                                                            |
| `secret.create`  | `true`       | Whether the chart creates a Kubernetes Secret for the certificate.                                                                                         |
| `secret.name`    | `""`         | Custom name for the Secret. If empty, a default name is generated.                                                                                         |
| `mountPath`      | `/etc/certs` | Path inside the container where the certificate is mounted.                                                                                                |
| `caBundle`       | `""`         | Base64-encoded CA bundle for the Validating Admission Webhook. If empty, uses the self-signed certificate. Set to an empty string when using cert-manager. |
| `useCertManager` | `false`      | When `true`, delegates certificate management to [cert-manager](https://cert-manager.io/docs/), which must already be installed in your cluster.           |

## Istio compatibility

The CloudZero Agent detects Istio automatically in most single-cluster deployments. No manual configuration is required.

If you encounter issues with label collection in an Istio-enabled cluster (labels not appearing in CloudZero despite the agent running), the problem is typically Istio's mutual TLS (mTLS) interfering with the agent's admission webhook. The webhook still allows pod deployments, but the mTLS conflict prevents it from capturing labels needed for cost allocation.

Use one of the following options to resolve the conflict:

### Option A: Disable sidecar injection for webhook pods

Removes the Istio sidecar from webhook-server pods only. The rest of the cluster retains normal Istio functionality.

```yaml
insightsController:
  server:
    podAnnotations:
      sidecar.istio.io/inject: "false"
```

### Option B: Exclude the webhook port from Envoy

Keeps the sidecar but bypasses Envoy for webhook traffic on port 8443. See the [Istio documentation on port exclusion](https://istio.io/latest/docs/reference/config/annotations/#SidecarTrafficExcludeInboundPorts) for details.

```yaml
insightsController:
  server:
    podAnnotations:
      traffic.sidecar.istio.io/excludeInboundPorts: "8443"
```

### Option C: Disable mTLS for the agent

Disables mTLS for the webhook-server pods only, keeping it enabled for the rest of the cluster.

Apply the following `PeerAuthentication` resource:

```yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: cloudzero-agent-mtls
  namespace: <CLOUDZERO_NAMESPACE>
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: webhook-server
  mtls:
    mode: DISABLE
```

Replace `<CLOUDZERO_NAMESPACE>` with the namespace where the agent is deployed (typically `cloudzero`), then apply:

```shell
kubectl apply -f cloudzero-agent-mtls.yaml
```

## Custom kube-state-metrics

<Callout icon="⚠️" theme="warn">
  CloudZero does not recommend using a custom kube-state-metrics instance. This can lead to silent data loss and inaccurate cost data. The agent includes a preconfigured version that is tuned and isolated to the CloudZero deployment.
</Callout>

If your organization requires a custom kube-state-metrics instance, add the following to your `values.yaml`:

```yaml
kubeStateMetrics:
  enabled: false
  prometheusScrape: false
  targetOverride: <SERVICE_NAME>.<NAMESPACE>.svc.cluster.local:<PORT>
```

| Parameter          | Default | Description                                                                            |
| ------------------ | ------- | -------------------------------------------------------------------------------------- |
| `enabled`          | `true`  | Set to `false` to disable the built-in kube-state-metrics.                             |
| `prometheusScrape` | `true`  | Set to `false` when using a custom instance, since the endpoint is defined explicitly. |
| `targetOverride`   | `""`    | The Kubernetes service address of your custom kube-state-metrics instance.             |

<Callout icon="ℹ️" theme="info">
  Have questions or feedback? Reach out to your account manager.
</Callout>