Budget GKE deployment
In an effort to better understand Kubernetes, the need to stand up monitoring infrastructure, and the desire to reduce the burden of maintaining a MySQL instance, I decided to check out Google’s GKE offering. As I’d be using this for hosting personal projects, I wanted to keep the cost as low as possible. This, plus latency to my ARP Networks VPSes, is why I chose GKE over other cloud providers.
Note: when I embarked on this journey, Google provided the Kubernetes control plane free of charge to all GKE users. Unfortunately this policy has since been rolled back, though for hobbyist users like myself, one zonal cluster per billing account still qualifies for a free Kubernetes control plane.
My first attempt to deploy GKE involved using f1-micro instances. Unfortunately the micro instances are too memory constrained and once Kubernetes components like kubelet, kube-proxy, and kube-dns are up and running, there was very little RAM left over for my workloads. At the time, e2 instances had not yet been rolled out, so I went with g1-small. Since then, the e2 instances have been released, and I’d recommend a minimum instance size of e2-small. You can get away with running g1-small, but you might need to disable some default services.
A GKE cluster can be created as follows:
gcloud container clusters create cluster-1 \
--release-channel regular \
--workload-pool=project-id.svc.id.goog
This will create a new GKE cluster with the name cluster-1 running the “regular” release channel. The last bit, workload-pool, is something I discovered after deploying my first GKE cluster. This enables GKE workoad identity, which makes it possible for GKE workloads to authenticate transparently to Google services. This will be important later when deploying external-dns, or if using other Google Cloud services from GKE.
The cluster just gives you a Kubernetes control plane with no nodes. This means that no workloads can be deployed. A node pool can be deployed as follows:
gcloud container node-pools create --cluster=cluster-1 \
--machine-type=e2-small --workload-metadata=GKE_METADATA \
--num-nodes=1 --zone us-west2-b pool-1 --disk-size=10GB
This will stand up a node pool named pool-1 inside the GKE cluster cluster-1 with one instance of machine type e2-small with a 10GB persistent in the us-west2-b zone. The small disk size was chosen to save money, as the Kubernetes workloads that require persistent disk will have their own persistent volume claims.
At this point the cluster is up and running and workloads can be deployed. However, generally one would want to expose those services to the outside world. That can easily be done by using a load balancer, but for hobbyist projects, that can be cost prohibitive. At the time of this writing, a load balancer on Google Cloud would cost $20/month at a minimum — excluding data costs. This is because Google’s load balancer is globally distributed — a great feature — but at quite a hefty charge. My entire deployment as documented here averages $30/month.
In order to avoid using Google’s load balancer, I deployed the nginx ingress controller. Kubernetes ingresses allow “services” deployed in the cluster to be “exposed” to the outside world. I say “exposed” because there’s a bit more magic required to make the ingress itself publicly accessible — most commonly by using a load balancer, which I was trying to avoid — but I’ll cover an alternative approach.
While Helm can be used to deploy the nginx ingress controller, I decided to stand it up manually to get a better understanding of how it works. Deploying the nginx ingress controller manually is covered in the nginx ingress controller installation guide. The controller can be deployed on GKE with one simple command:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/cloud/deploy.yaml
By default, the nginx ingress controller is set up so that it can be used with Google’s load balancer. Since I wanted to save money and not use the load balancer, I had to change a few things. To edit the nginx ingress controller deployment, use the following commands:
kubectl edit deployment -n ingress-nginx ingress-nginx-controller
I had to make the following changes to the deployment for the budget setup. First, the deployment strategy should be set to Recreate. This is because we’ll be using hostPort to expose the nginx directly on the Kubernetes node, which will cause issues with RollingUpdate. Second, the following arg needs to be removed from the container spec:
- --publish-service=ingress-nginx/ingress-nginx-controller
This is because we’ll not be using the service to expose the nginx ingress controller via a load balancer, but instead using a hostPort and DNS to reach nginx from outside the cluster. Finally, the container spec’s ports need to be modified to allow use hostPort. This should be done for the http and https ports. There’s no need to change the webhook port. The container spec’s ports property should look as follows:
ports:
- containerPort: 80
hostPort: 80
name: http
protocol: TCP
- containerPort: 443
hostPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
This will cause the container ports 80 and 443 to map directly to the node’s ports 80 and 443, making the nginx available on the node’s external IP.
With all this in place, the nginx should be reachable from the outside world on the node’s external IP, which can be found via the following comand:
gcloud compute instances list
Kubernetes ingresses deployed to the cluster will now be exposed by the nginx controller. For example, to expose Prometheus running on the cluster, deploy the following ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
external-dns.alpha.kubernetes.io/ttl: "60"
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/server-alias: prometheus.example.com
name: prometheus-ingress
namespace: default
spec:
rules:
- host: prometheus.gcp.example.com
http:
paths:
- backend:
serviceName: prometheus-service
servicePort: 9090
This ingress being tagged with the kubernetes.io/ingress.class annotation will cause it to be exposed by our nginx ingress controller. The external-dns annotation configures the TTL for the DNS record of our ingress (more on that later). The nginx.ingress.kubernetes.io/server-alias annotation configures an nginx server alias, if desired.
At this point, presuming workloads are up and running and the corresponding services and ingresses have been deployed, everything should be reachable through the node’s external IP and nginx as previously discussed. However, manual DNS configuration isn’t a great solution as the node’s IP will change whenever it is upgraded or taken down for maintenance. To solve this problem, DNS can be automated by using external-dns.
The external-dns repository includes a GKE deployment tutorial. However, this tutorial requires that the node pools be granted read/write access to Cloud DNS. While this may work fine for testing, the impact is that all workloads on these Kubernetes nodes will have read/write access to Cloud DNS. This may not be desired. I first deployed my node pool using the scope in order to get external-dns up and running, but later discovered workload identity, which is much better from a security standpoint. I’ll cover deploying external-dns with workload identity here.
external-dns will manage a zone in Cloud DNS. Above I’ve referenced a dedicated zone “gcp.example.com”, but it can also manage a top level zone so long as the names do not collide. Extra metadata is stored in Cloud DNS to determine if external-dns is the owner of the associated records. When used with the nginx ingress controller server-alias annotation shown above, a top level CNAME can be set up to point to records in the external-dns managed subdomain and the nginx ingress controller will write traffic to the correct workload.
To set up the credentials as needed for workload identity, a service account must be created:
gcloud iam service-accounts create external-dns
This service account needs to be granted the DNS administrator role:
gcloud projects add-iam-policy-binding $PROJECT_ID \
--role roles/dns.admin \
--member "serviceAccount:external-dns@$PROJECT_ID.iam.gserviceaccount.com"
Change both references of $PROJECT_ID to your project ID.
Finally, a mapping needs to be made between the Kubernetes Service Account (which will be created below) and the Google Service Account created above:
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$PROJECT_ID.svc.id.goog[default/external-dns]" \
external-dns@$PROJECT_ID.iam.gserviceaccount.com
Again, change both references of $PROJECT_ID to your project ID.
Now that the service accounts have been set up, the following manifest can be used to deploy external-dns:
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
annotations:
iam.gke.io/gcp-service-account: external-dns@$PROJECT_ID.iam.gserviceaccount.com
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: external-dns
rules:
- apiGroups: [""]
resources: ["services","endpoints","pods"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: registry.opensource.zalan.do/teapot/external-dns:latest
args:
- --source=ingress
- --domain-filter=gcp.example.com
- --provider=google
- --google-project=$PROJECT_ID
- --registry=txt
- --txt-owner-id=my-identifier
Change the iam.gke.io/gcp-service-account annotation, --domain-filter and --google-project arguments as required.
Now that external-dns has been deployed, any ingresses will automatically have corresponding DNS entries published to Cloud DNS. The TTL can be configured as shown above, and a low TTL will result in minimal downtime should the node IP change. For production workloads, one would want to have multiple nodes and ensure that workloads are always distributed across those nodes so there is zero downtime, and would likely want to use a load balancer anyway — but for low cost setups that can incur downtime, this should be fine.
I’ve been running this setup for well over a year now and it’s been working great. I first set this up just to run Prometheus in order to monitor my systems, but eventually deployed a stateful MySQL workload on the cluster as well. The MySQL service requires a patch to external-dns as it’s running as a “headless” service. There is an upstream issue and corresponding fix, which I’m hoping will be merged soon.