Advanced Kubernetes Agent Configuration

The CloudZero Agent for Kubernetes is configured through a Helm values file. The install page covers the required parameters and optional label configuration. This page covers additional configuration for production environments and specific infrastructure requirements.

All configuration on this page is applied through your values.yaml file. After making changes, apply them with:

helm upgrade cloudzero-agent cloudzero/cloudzero-agent \
    --namespace cloudzero \
    -f values.yaml

Annotations

The install page covers label configuration. Annotations use the same pattern but are disabled by default.

To collect annotations, add the following to your values.yaml:

insightsController:
  annotations:
    enabled: true
    patterns:
      - '.*'
    resources:
      pods: true
      namespaces: true

Label and annotation disambiguation

When CloudZero collects labels and annotations from multiple resource types, it uses prefixes to distinguish the source:

Use a Kubernetes Secret for the API key

Storing the API key in a Kubernetes Secret keeps credentials out of your command history and Helm release metadata. For stronger security, the agent also supports External Secrets Operator for integration with external secrets managers.

1. Create the secret:

kubectl create secret -n cloudzero generic cloudzero-api-key \
    --from-literal=value=<CLOUDZERO_API_KEY>

2. Reference the secret in your values.yaml:

existingSecretName: cloudzero-api-key

When existingSecretName is set, the apiKey parameter is not required.

Sizing

The default resource requests are sufficient for most clusters. For larger clusters, the agent's memory requirements scale with node count:

The formula is: 512Mi + (nodes / 100) x 768Mi.

To adjust resource requests, add the following to your values.yaml:

components:
  agent:
    resources:
      requests:
        memory: "2048Mi"
      limits:
        memory: "2560Mi"

Custom TLS certificates

By default, the agent generates a self-signed certificate for its admission webhook. To use your own certificate or integrate with cert-manager, add the following to your values.yaml:

insightsController:
  tls:
    enabled: true
    crt: ""
    key: ""
    secret:
      create: true
      name: ""
    mountPath: /etc/certs
    caBundle: ""
    useCertManager: false

Istio compatibility

The CloudZero Agent detects Istio automatically in most single-cluster deployments. No manual configuration is required.

If you encounter issues with label collection in an Istio-enabled cluster (labels not appearing in CloudZero despite the agent running), the problem is typically Istio's mutual TLS (mTLS) interfering with the agent's admission webhook. The webhook still allows pod deployments, but the mTLS conflict prevents it from capturing labels needed for cost allocation.

Use one of the following options to resolve the conflict:

Option A: Disable sidecar injection for webhook pods

Removes the Istio sidecar from webhook-server pods only. The rest of the cluster retains normal Istio functionality.

insightsController:
  server:
    podAnnotations:
      sidecar.istio.io/inject: "false"

Option B: Exclude the webhook port from Envoy

Keeps the sidecar but bypasses Envoy for webhook traffic on port 8443. See the Istio documentation on port exclusion for details.

insightsController:
  server:
    podAnnotations:
      traffic.sidecar.istio.io/excludeInboundPorts: "8443"

Option C: Disable mTLS for the agent

Disables mTLS for the webhook-server pods only, keeping it enabled for the rest of the cluster.

Apply the following PeerAuthentication resource:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: cloudzero-agent-mtls
  namespace: <CLOUDZERO_NAMESPACE>
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: webhook-server
  mtls:
    mode: DISABLE

Replace <CLOUDZERO_NAMESPACE> with the namespace where the agent is deployed (typically cloudzero), then apply:

kubectl apply -f cloudzero-agent-mtls.yaml

Custom kube-state-metrics

If your organization requires a custom kube-state-metrics instance, add the following to your values.yaml:

kubeStateMetrics:
  enabled: false
  prometheusScrape: false
  targetOverride: <SERVICE_NAME>.<NAMESPACE>.svc.cluster.local:<PORT>