Configuring Cloud Build to use Static External IP Addresses

Authors: Kumail Kermalli, DevOps Engineer and Mohammad Pooladkhay, DevOps Engineer

Background

Cloud Build is Google Cloud’s serverless CI/CD platform. First launched in 2016, it allows users to execute builds in a fully-managed environment. Each build is made up of a series of steps and each runs in a specified Docker container. This allows a degree of customisation, as users can choose to use either Pre-Built Images or create their own to run their jobs on.

In 2021, an additional layer of customisation was added with the launch of “Private Pools”. This allowed the setup of dedicated worker pools to execute a build, with the ability to access resources in a Virtual Private Cloud (VPC) network. Other useful customisation options include an increase in the number of concurrent builds allowed, greater machine type choice and the ability to disable access of your private pool to the public internet.

However, a key functionality that is not available in Cloud Build (at the time of writing, February 2023) is the ability to set a static external IP address to your build, even if it is running in a private pool. This would be extremely useful for some users as it would allow them to whitelist worker pool addresses in external providers they may need to access for builds.

Unfortunately, a Serverless VPC Connector and/or Cloud NAT cannot be used here due to private worker pools being located in a Google-owned Virtual Private Cloud network called the service producer network, peering into a project VPC (see Figure 1). Whilst Google does provide a list of external IP addresses Google Cloud resources use, whitelisting all of them is impractical.

Introduction

Adding the native ability to assign an external IP address to Cloud Build is a feature request on Google’s Issue Tracker, so we may see this implemented in the near- to mid-term. However, in the interim, an alternative solution is needed. Read on to learn how to assign a static external IP address to a Cloud Build instance through the use of a proxy server hosted on Google Compute Engine.

The aim is to prevent Cloud Build instances from directly reaching out to the public internet over IP addresses we do not know the range of. Instead, a VM with a static IP address will act as their Proxy Server and all the communications between the VM and Build instances will be via private IP addresses.

Tutorial

TLDR; Assuming you already have a VPC network and required IAM permissions:

1. Enable the Cloud Build and the Service Networking APIs
2. Create a VPC network (if you don’t already have one)
3. Allocate a named IP range in your VPC network
4. Create a private connection to the service producer network
5. Create a new Worker Pool
6. Create a subnetwork for the VM
7. Allocate a Static IP address to be added to the VM
8. Create a VM instance
9. Set up an HTTP Proxy Server on the VM
10. Add Firewall Rules allowing Ingress Traffic to the Proxy only via its private IP address
11. Instruct Cloud Build instances to use the VM’s private IP address as Proxy

If you wish to use the Google Cloud Console or gcloud CLI to achieve the goal, the process is well documented on both the Networking and Cloud Build components as well as VM instances. In this tutorial, we will demonstrate how to use Terraform to implement the above steps.

Source code is available at https://github.com/teamdatatonic/blog-cloud-build-static-ip with comments indicating each step inside the main.tf file.

1. Enabling the Cloud Build and the Service Networking APIs

resource "google_project_service" "enable_cloud_build" {
  project = var.project_id
  service = "cloudbuild.googleapis.com"
  disable_dependent_services = true
}
resource "google_project_service" "enable_service_networking" {
  project = var.project_id
  service = "servicenetworking.googleapis.com"
  disable_dependent_services = true
}

2. Creating a VPC network

Almost every resource in Google Cloud requires a VPC network to be added to.

resource "google_compute_network" "vpc_network" {
 project                 = var.project_id
 name                    = var.vpc_network_name
 auto_create_subnetworks = false
 mtu                     = 1460
}

3. Allocating a named IP range In your VPC network

We need to allocate a named private IP range so that we can later use it to peer our VPC network to the VPC network where Cloud Build Private Pools reside (Service Producer Network).

Cloud Build reserves the IP range 192.168.10.0/24 for the Docker bridge network. When allocating the IP ranges for resources in your project, Google recommends selecting a range outside of 192.168.10.0/24 in cases when Cloud Build builders access these resources.

resource "google_compute_global_address" "named_private_ip" {
 provider      = google-beta
 name          = var.named_private_ip_name
 project       = var.project_id
 purpose       = "VPC_PEERING"
 address_type  = "INTERNAL"
 network       = google_compute_network.vpc_network.name
 address       = var.named_private_ip
 prefix_length = 24
}

4. Creating a private connection to the service producer network

This will create a private connection between our VPC network and Google’s own VPC network where our private worker pool will be hosted, i.e., the service producer network, using the private IP range we just allocated.

This step depends on the last step since we have referenced named_private_ip’s name.

resource "google_service_networking_connection" "service_producer_connection" {
 network                 = "projects/${var.project_id}/global/networks/${google_compute_network
.vpc_network.name}"
 service                 = "servicenetworking.googleapis.com"
 reserved_peering_ranges = [google_compute_global_address.named_private_ip.name]
}

5. Creating a new Worker Pool

When creating a worker pool, we want to prevent the Cloud Build instances from directly reaching out to the public internet and instead proxy all of their traffic through a Proxy server that we will host on a VM instance.

To achieve this, we set no_external_ip to true so that the instances won’t have direct access to the public internet. Then, we set the peered_network to our VPC network in which we created a private connection to the service producer network.

Finally, we need to make sure that the creation of the Worker Pool happens after the connection to the Service Producer Network by explicitly adding the depends_on.

resource "google_cloudbuild_worker_pool" "private_worker_pool" {
 name     = var.private_worker_pool_name
 location = var.region
 project  = var.project_id
 worker_config {
   disk_size_gb   = 100
   machine_type   = "e2-medium"
   no_external_ip = true
 }
 network_config {
   peered_network = "projects/${var.project_id}/global/networks/${google_compute_network.vpc_
network.name}"
 }
 depends_on = [google_service_networking_connection.service_producer_connection]
}

6. Creating a subnetwork for the VM

We need a subnetwork with a private range that we define so that later we can assign a private IP address from that range to the VM by which Build instances will communicate with the VM.

resource "google_compute_subnetwork" "proxy_subnet" {
 name                     = "${google_compute_network.vpc_network.name}-proxy-subnet"
 ip_cidr_range            = var.vm_subnet_range
 network                  = google_compute_network.vpc_network.name
 region                   = var.region
 private_ip_google_access = false
 project                  = var.project_id
}

7. Allocating a Static IP address to be added to the VM

This will be the public IP address for the Cloud Build instances.

resource "google_compute_address" "static_ip" {
 name         = var.static_ip_name
 project      = var.project_id
 region       = var.region
 address_type = "EXTERNAL"
 network_tier = "STANDARD"
}

8. Creating a VM instance

This VM will be our Proxy Server. Cloud Build instances talk to it via its private IP then it proxies their requests to the destination.

Keep in mind that VM’s specs should be adjusted to meet your requirements and var.vm_ip_address must be from the specified range in Step 5.

There is also a startup script added to the VM which we will discuss in the next step.

resource "google_compute_instance" "proxy_vm" {
 name         = "proxy-vm"
 project      = var.project_id
 zone         = var.zone
 machine_type = "n1-standard-1"


 boot_disk {
   initialize_params {
     image = "debian-cloud/debian-11"
   }
 }
 tags                    = ["proxy-srv"]
 metadata_startup_script = file("setup_proxy.sh")

 network_interface {
   subnetwork = google_compute_subnetwork.proxy_subnet.name
   network_ip = var.vm_ip_address
   access_config {
     nat_ip       = google_compute_address.static_ip.address
     network_tier = "STANDARD"
   }
 }
}

9. Setting up an HTTP Proxy Server on the VM

You might have noticed that there is a startup script added to the VM. This script will run right after the VM gets created. It sets up a Forward Proxy on the VM so no manual work is required.

Feel free to use any Proxy Server that suits your needs. Here we are using “github.com/elazarl/goproxy” to build our own proxy server.

The script creates a Go project, builds it, and then runs the binary using a Systemd Service so it will start automatically on VM reboots.

You can change the Listen Port number for the proxy by editing the script. However, if you change it, you also need to change the firewall rule to match the new port.

10. Adding Firewall Rules allowing Ingress Traffic to the Proxy Port

For security reasons, we only accept ingress traffic to the Proxy Server from its Internal IP address. This means that no one from the outside world can use our Proxy Server.

This policy is enforced using a firewall rule:

resource "google_compute_firewall" "proxy_ingress" {
 name    = "allow-proxy-ingress"
 network = google_compute_network.vpc_network.name


 source_ranges = ["${var.named_private_ip}/${var.named_private_ip_prefix_length
}"]
 allow {
   protocol = "tcp"
   ports    = ["9231"] # If you change this, make sure you're also changing it inside setup_
proxy.sh
 }
 target_tags = ["proxy-srv"]
}

11. Instructing Cloud Build instances to use VMs private IP address as Proxy

Now that we have everything in place, it’s time to trigger a sample build to see the results.

Here is a simple Build instruction that sets the http_proxy and https_proxy environment variables, installs curl, and then queries ipify API. After a successful run, you should be able to see that the VM’s static IP address gets printed, indicating that the outside world thinks our Build Instances have that IP address.

steps:
 - name: ubuntu
   script: |
     export http_proxy="http://10.0.3.10:9231"
     export https_proxy="http://10.0.3.10:9231"
     apt update
     apt install curl -y
     curl https://api.ipify.org/
options:
 pool:
  name: "projects/{project_id}/locations/europe-west2/workerPools/{private_worker_pool_name}"

 

Conclusion

That is all! We have managed to assign a static external IP address to a Cloud Build instance through a proxy server hosted on Google Compute Engine. We have also verified this by running a sample Cloud Build job, where we saw that an external request’s IP address matched our proxy VM’s external IP. With a static external IP for your Cloud Build instance you can whitelist the IP in external providers you may need to access for builds, adding that extra layer of security to your application!

Datatonic, Google Cloud’s 4 X Partner of the Year, is the leading cloud data + AI consultancy for the world’s most ambitious businesses, challenging traditions to deliver tangible innovation at the leading edge of Google Cloud. Our team has a wealth of experience with Cloud Build, and other Google Cloud products. Need help with your data and AI? Get in touch at datatonic.com/contact/.