As one of the main maintainers of the Edge to Mesh tutorial for the last 2-3 years, I have been a huge fan of the managed GCE ingress controller. From GKE, you could use Ingress, Service, BackendConfig manifests in order to generate all the Google Cloud Infrastructure to expose your public endpoint behind an HTTPS L7 Load Balancer and leverage advanced features like CDN, Cloud Armor, etc.

Deploying external L7 load balancing outside of the mesh along with a mesh ingress layer offers significant advantages, especially for internet traffic. Even though Anthos Service Mesh and Istio ingress gateways provide advanced routing and traffic management in the mesh, some functions are better served at the edge of the network. Taking advantage of internet-edge networking through Google Cloud’s external HTTP(S) load balancer might provide significant performance, reliability, or security-related benefits over mesh-based ingress.

I was recently involved in a project where the Nginx Ingress controller (from the Kubernetes community but same applies to the one maintained by NGINX Inc.) was used. It was for different reasons, one of them for example is that the GCE ingress controller doesn’t support multiple Ingresses on the same public IP address.

Now let’s say you want to secure your Nginx Ingress controller behind an HTTPS L7 Load Balancer and Cloud Armor. Sounds appealing, right? But actually it’s not supported like you can easily do it with the GCE ingress controller.

Fortunately, after some research I found out that we can set this up by ourself, manually. Here, here, here or here are good pointers I found.

But to be honest, after giving them a shot, I found out that they were not complete for my own needs with more security as requirements. Here is what I needed in addition to what was shared:

That’s what I will cover throughout this blog post by sharing my learnings and showing you this end-to-end setup, please bear with me! :)

Note: I’m doing this via gcloud commands, but everything can be done via Terraform too.

Define common variables:

gcloud config set project ${PROJECT_ID}


Get some information from your existing GKE cluster:

gcloud container clusters get-credentials ${CLUSTER_NAME} \
    --zone ${CLUSTER_ZONE}

CLUSTER_FIREWALL_RULE_TAG=$(gcloud compute instances describe \
    $(kubectl get nodes -o jsonpath='{.items[0]}') \
	--zone ${CLUSTER_ZONE} \
    --format "value(tags.items[0])")
CLUSTER_MASTER_IP_CIDR=$(gcloud container clusters describe ${CLUSTER_NAME} \
    --zone ${CLUSTER_ZONE} \
    --format "value(privateClusterConfig.masterIpv4CidrBlock)")
NETWORK=$(gcloud container clusters describe ${CLUSTER_NAME} \
    --zone ${CLUSTER_ZONE} \
    --format "value(network)")

Deploy the Nginx Ingress Controller not exposed as public endpoint, supporting HTTPS only, keeping the source IP address coming from the L7 load balancer and attached to the associated Network Endpoint Group:

cat <<EOF > ${CLUSTER_NAME}-nginx-ingress-controller-values.yaml
    enableHttp: false
    type: ClusterIP
    annotations: '{"exposed_ports": {"${NGINX_NEG_PORT}":{"name": "${NGINX_NEG_NAME}"}}}'
    use-forwarded-headers: true
helm upgrade \
    --install ingress-nginx ingress-nginx \
    --repo \
    --namespace ingress-nginx \
    --create-namespace \
    -f ${CLUSTER_NAME}-nginx-ingress-controller-values.yaml

This will create for you the associated Network Endpoint Group (NEG) that we will attach later to our Global Load Balancer:

gcloud compute network-endpoint-groups list

If you are using a GKE cluster with private nodes, you need to allow the Kubernetes master nodes to talk to the node pool on port 8443 for Nginx Ingress controller:

gcloud compute firewall-rules create k8s-masters-to-nodes-on-8443 \
    --network ${NETWORK} \
    --direction INGRESS \
    --source-ranges ${CLUSTER_MASTER_IP_CIDR} \
    --target-tags ${CLUSTER_FIREWALL_RULE_TAG} \
    --allow tcp:8443

Configure the backend

Now, let’s create the Load Balancer and all the required components.

Let’s define the type of HTTPS Load Balancer we want. EXTERNAL_MANAGED means that we will use the next generation Global External Load Balancer:


Note: You can use EXTERNAL if you want to still use the Global External Load Balancer (Classic) instead. You can see the differences and the limitations between both here.

Allow traffic from the Load Balancer to the node pool on port 443 for Nginx Ingress controller:

gcloud compute firewall-rules create ${CLUSTER_NAME}-allow-tcp-loadbalancer \
    --network ${NETWORK} \
    --allow tcp:${NGINX_NEG_PORT} \
    --source-ranges, \
    --target-tags ${CLUSTER_FIREWALL_RULE_TAG}

Note: these IP ranges correspond to the Google Cloud probers to connect to your backend, more information here.

Add an HTTPS Health Check configuration:

gcloud compute health-checks create https ${CLUSTER_NAME}-ingress-nginx-health-check \
    --port ${NGINX_NEG_PORT} \
    --check-interval 60 \
    --unhealthy-threshold 3 \
    --healthy-threshold 1 \
    --timeout 5 \
    --request-path /healthz

Create a backend service:

gcloud compute backend-services create ${CLUSTER_NAME}-ingress-nginx-backend-service \
    --load-balancing-scheme ${LOAD_BALANCING_SCHEME} \
    --protocol HTTPS \
    --port-name https \
    --health-checks ${CLUSTER_NAME}-ingress-nginx-health-check \
    --enable-logging \

Add the NEG to the backend service:

gcloud compute backend-services add-backend ${CLUSTER_NAME}-ingress-nginx-backend-service \
    --network-endpoint-group ${NGINX_NEG_NAME} \
    --network-endpoint-group-zone ${CLUSTER_ZONE} \
    --balancing-mode RATE \
    --capacity-scaler 1.0 \
    --max-rate-per-endpoint 100 \

Configure the frontend

That was for the backend part, let’s now do the frontend part.

Create an url map:

gcloud compute url-maps create ${CLUSTER_NAME}-ingress-nginx-loadbalancer \
    --default-service ${CLUSTER_NAME}-ingress-nginx-backend-service

Create an HTTP proxy:

gcloud compute target-http-proxies create ${CLUSTER_NAME}-ingress-nginx-http-proxy \
    --url-map ${CLUSTER_NAME}-ingress-nginx-loadbalancer

Create a public static IP address:

gcloud compute addresses create ${CLUSTER_NAME}-public-static-ip \
INGRESS_IP=$(gcloud compute addresses describe ${CLUSTER_NAME}-public-static-ip --global --format "value(address)")
echo ${INGRESS_IP}

Here you can bring your own DNS, I’m creating one to illustrate the scenario with an example:

cat <<EOF > my-dns-spec.yaml
swagger: "2.0"
  description: "Cloud Endpoints DNS"
  title: "Cloud Endpoints DNS"
  version: "1.0.0"
paths: {}
host: "${DNS}"
- name: "${DNS}"
  target: "${INGRESS_IP}"
gcloud endpoints services deploy my-dns-spec.yaml

Generate the SSL certificate for this DNS:

openssl genrsa -out my-dns-ca.key 2048
openssl req -x509 \
    -new \
    -nodes \
    -days 365 \
    -key my-dns-ca.key \
    -out my-dns-ca.crt \
    -subj "/CN=${DNS}"

Upload this SSL certificate in Google Cloud:

gcloud compute ssl-certificates create my-dns-ssl-certificate \
    --certificate my-dns-ca.crt \
    --private-key my-dns-ca.key \
gcloud compute target-https-proxies create ${CLUSTER_NAME}-ingress-nginx-http-proxy \
    --url-map ${CLUSTER_NAME}-ingress-nginx-loadbalancer \
    --ssl-certificates my-dns-ssl-certificate

Note: If you have multiple SSL certificates, that’s where you will provide them.

Create a global forwarding rule on port 443:

gcloud compute forwarding-rules create ${CLUSTER_NAME}-https-forwarding-rule \
    --load-balancing-scheme ${LOAD_BALANCING_SCHEME} \
    --network-tier PREMIUM \
    --global \
    --ports 443 \
    --target-https-proxy ${CLUSTER_NAME}-ingress-nginx-http-proxy \
    --address ${CLUSTER_NAME}-public-static-ip

Configure HTTPS redirect

Configure an HTTP to HTTPS redirect on the load balancer:

cat <<EOF > ${CLUSTER_NAME}-http-to-https-redirect.yaml
kind: compute#urlMap
name: ${CLUSTER_NAME}-http-to-https-redirect
  httpsRedirect: True
gcloud compute url-maps import ${CLUSTER_NAME}-http-to-https-redirect \
    --source ${CLUSTER_NAME}-http-to-https-redirect.yaml \
gcloud compute target-http-proxies create ${CLUSTER_NAME}-http-to-https-redirect-proxy \
    --url-map ${CLUSTER_NAME}-http-to-https-redirect \
gcloud compute forwarding-rules create ${CLUSTER_NAME}-http-to-https-redirect-rule \
    --load-balancing-scheme ${LOAD_BALANCING_SCHEME} \
    --network-tier PREMIUM \
    --address ${CLUSTER_NAME}-public-static-ip \
    --global \
    --target-http-proxy ${CLUSTER_NAME}-http-to-https-redirect-proxy \
    --ports 80

Configure Cloud Armor

Create Cloud Armor and attach it to the public endpoint:

gcloud compute security-policies create ${CLUSTER_NAME}-security-policy
gcloud compute security-policies update ${CLUSTER_NAME}-security-policy \
gcloud compute backend-services update ${CLUSTER_NAME}-ingress-nginx-backend-service \
    --global \
    --security-policy ${CLUSTER_NAME}-security-policy

Note: From here, you could use any additional features from Cloud Armor you want. In this example, we are just using the default DDoS protection

That’s it for the setup, we made it!

Deploy a sample app

Let’s now deploy a sample app.

Deploy the app:

kubectl create deployment whereami \
    --image \
    --port 8080
kubectl expose deployment whereami \
    --port 80 \
    --target-port 8080 \
    --type ClusterIP

Bind the app to the Nginx Ingress controller:

cat << EOF | kubectl apply -f -
kind: Ingress
  name: whereami
  ingressClassName: nginx
  - host: ${DNS}
      - backend:
            name: whereami
              number: 80
        path: /
        pathType: Prefix

Note: You will notice that we don’t set the TLS configuration here since it is managed on the Load Balancer directly thanks to the setup we did earlier.

Now if you hit the link displayed below you will be redirected to an HTTPS link and to eventually a working (and very secure) app:

echo -E "http:://${DNS}"


Wow! Quite a long ride to make it, right? Yup! But that’s what it takes to actually secure your Nginx Ingress controller (or any 3rd party reverse proxy you may use in GKE). Hope you liked it and that you will be able to use this for your own context.

Cheers! Stay safe out there! Happy sailing!