GCP Billing Data Integration

The following delineates the process of mapping GCP Billing Data into CloudZero's standard Common Billing Format (CBF). This ensures consistency, accuracy, and ease of interpretation when you are examining GCP-based billing records within the CloudZero platform.

Sample GCP billing record

{
  "adjustment_info": null,
  "billing_account_id": "00AABB-CCDD12-998877",
  "cost": 1.987900000000000e-02,
  "cost_type": "regular",
  "credits": [],
  "currency": "USD",
  "currency_conversion_rate": 1.000000000000000e+00,
  "export_time": 1653012525407000,
  "invoice": {
    "month": "202205"
  },
  "labels": [
    {
      "key": "airflow-version",
      "value": "v1-10-6-composer"
    },
    {
      "key": "goog-dataproc-cluster-name",
      "value": "cust-datalake-generate-v2-20220518"
    },
    {
      "key": "goog-dataproc-cluster-uuid",
      "value": "c3f3a73c-6f27-4620-bae9-e9ba228e6a1a"
    },
    {
      "key": "goog-dataproc-location",
      "value": "us-central1"
    }
  ],
  "location": {
    "country": "US",
    "location": "us-central1",
    "region": "us-central1"
  },
  "project": {
    "ancestors": [
      {
        "display_name": "cz-project-id",
        "resource_name": "projects/123420799999"
      },
      {
        "display_name": "cloudzero.com",
        "resource_name": "organizations/123429371234"
      }
    ],
    "ancestry_numbers": "/123429371234/",
    "id": "cz-project-id",
    "labels": [],
    "name": "CZ Project",
    "number": "123420799999"
  },
  "resource": {
    "global_name": "//compute.googleapis.com/projects/123420799999/zones/us-central1-c/disk/6082010960894652432",
    "name": "cust-datalake-generate-v2-20220518-w-14"
  },
  "service": {
    "description": "Compute Engine",
    "id": "6F81-5844-456A"
  },
  "sku": {
    "description": "SSD backed PD Capacity",
    "id": "B188-61DD-52E4"
  },
  "system_labels": [],
  "tags": [],
  "usage": {
    "amount": 3.865465515688619e+14,
    "amount_in_pricing_units": 1.344084270000000e-01,
    "pricing_unit": "gibibyte month",
    "unit": "byte-seconds"
  },
  "usage_end_time": 1652943600000000,
  "usage_start_time": 1652940000000000
}

Complete details on the elements of this record can be found in the Google Cloud documentation

Mapping GCP data to CloudZero Dimensions

The following table documents the alignment between GCP billing data and CloudZero Dimensions.

DimensionGCP Record ElementNotes
LineItemTypecost_typecost_type values are mapped to equivalent line item types like: Usage, Tax, Discount
Accountproject.id
Serviceservice.description
UsageFamilysku.description
Resourceresource.global_nameElements of the global_name are extracted and converted to a CZRN. When the global_name is not provided, other elements of the line item (like account / service) are used to generate a CZRN. A local resource ID is created from a hash of all available labels.
Regionlocation.region
UsageAmountusage.amount_in_pricing_units
unitusage.pricing_unit
UsageStartDateusage_start_time
UsageEndDateusage_end_time
K8s:Clusterlabels.valueRelevant if GKE is active. Consolidates values from specific label keys including (goog-k8s-cluster-name, gke-cluster)
K8s:Namespacelabels.valueRelevant if GKE is active. Consolidates values from specific label keys including (k8s-namespace, goog-k8s-namespace)
K8s:Podlabels.valueRelevant if GKE is active. Consolidates values from specific label keys including (app, k8s-app, app.kubernetes.io/name, service, dag_id, task_id, name, statefulset.kubernetes.io/pod-name, job-name)
Tag:gcp_cz:ancestry_numbersproject.ancestry_numbers
Tag:gcp_cz:organization_nameproject.ancestors.display_nameSelects the display_name linked to a resource prefixed with "organizations".
Tag:gcp_cz:resource_nameresource.nameCurrently only available for “Compute Engine” service
Tag:gcp_cz:project_nameproject.name
Tag:gcp:*system_labels
Tag:project:*project.labels
Tag:*labelsKey/Value pairs are expended into “user tags”