GCP Billing Data Integration

Introduction:

The subsequent section delineates the process of mapping GCP Billing Data into CloudZero's standard Common Billing Format (CBF). This ensures consistency, accuracy, and ease of interpretation when examining GCP-based billing records within the CloudZero platform.

Sample GCP Billing Record

{
  "adjustment_info": null,
  "billing_account_id": "00AABB-CCDD12-998877",
  "cost": 1.987900000000000e-02,
  "cost_type": "regular",
  "credits": [],
  "currency": "USD",
  "currency_conversion_rate": 1.000000000000000e+00,
  "export_time": 1653012525407000,
  "invoice": {
    "month": "202205"
  },
  "labels": [
    {
      "key": "airflow-version",
      "value": "v1-10-6-composer"
    },
    {
      "key": "goog-dataproc-cluster-name",
      "value": "cust-datalake-generate-v2-20220518"
    },
    {
      "key": "goog-dataproc-cluster-uuid",
      "value": "c3f3a73c-6f27-4620-bae9-e9ba228e6a1a"
    },
    {
      "key": "goog-dataproc-location",
      "value": "us-central1"
    }
  ],
  "location": {
    "country": "US",
    "location": "us-central1",
    "region": "us-central1"
  },
  "project": {
    "ancestors": [
      {
        "display_name": "cz-project-id",
        "resource_name": "projects/123420799999"
      },
      {
        "display_name": "cloudzero.com",
        "resource_name": "organizations/123429371234"
      }
    ],
    "ancestry_numbers": "/123429371234/",
    "id": "cz-project-id",
    "labels": [],
    "name": "CZ Project",
    "number": "123420799999"
  },
  "resource": {
    "global_name": "//compute.googleapis.com/projects/123420799999/zones/us-central1-c/disk/6082010960894652432",
    "name": "cust-datalake-generate-v2-20220518-w-14"
  },
  "service": {
    "description": "Compute Engine",
    "id": "6F81-5844-456A"
  },
  "sku": {
    "description": "SSD backed PD Capacity",
    "id": "B188-61DD-52E4"
  },
  "system_labels": [],
  "tags": [],
  "usage": {
    "amount": 3.865465515688619e+14,
    "amount_in_pricing_units": 1.344084270000000e-01,
    "pricing_unit": "gibibyte month",
    "unit": "byte-seconds"
  },
  "usage_end_time": 1652943600000000,
  "usage_start_time": 1652940000000000
}

Complete details on the elements of this record can be found here

Mapping GCP Data to CloudZero Dimensions

To understand how GCP billing data integrates with CloudZero's platform, the table below demonstrates the alignment:

Dimension	GCP Record Element	Notes
LineItemType	cost_type	cost_type values are mapped to equivalent line item types like: Usage, Tax, Discount
Account	project.id
Service	service.description
UsageFamily	sku.description
Resource	resource.global_name	Elements of the global_name are extracted and converted to a CZRN. When the global_name is not provided, other elements of the line item (like account / service) are used to generate a CZRN. A local resource ID is created from a hash of all available labels.
Region	location.region
UsageAmount	usage.amount_in_pricing_units
unit	usage.pricing_unit
UsageStartDate	usage_start_time
UsageEndDate	usage_end_time
K8s:Cluster	labels.value	Relevant if GKE is active. Consolidates values from specific label keys including (`goog-k8s-cluster-name`, `gke-cluster`)
K8s:Namespace	labels.value	Relevant if GKE is active. Consolidates values from specific label keys including (`k8s-namespace`, `goog-k8s-namespace`)
K8s:Pod	labels.value	Relevant if GKE is active. Consolidates values from specific label keys including (`app`, `k8s-app`, `app.kubernetes.io/name`, `service`, `dag_id`, `task_id`, `name`, `statefulset.kubernetes.io/pod-name`, `job-name`)
Tag:gcp_cz:ancestry_numbers	project.ancestry_numbers
Tag:gcp_cz:organization_name	project.ancestors.display_name	Selects the display_name linked to a resource prefixed with "organizations".
Tag:gcp_cz:resource_name	resource.name	Currently only available for “Compute Engine” service
Tag:gcp_cz:project_name	project.name
Tag:gcp:*	system_labels
Tag:project:*	project.labels
Tag:*	labels	Key/Value pairs are expended into “user tags”