Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Developed by Google and released as open-source, Kubernetes is now maintained by a diverse community and shepherded by the Cloud Native Computing Foundation.

The easiest way to run a Kubernetes cluster is using Google Kubernetes Engine, a managed version of Kubernetes hosted on Google Cloud Platform. Kubernetes Engine, also known as GKE, is a managed environment for deploying containerized applications. It brings our latest innovations in developer productivity, resource efficiency, automated operations, and open source flexibility to accelerate your time to market.

This codelab shows you some of the advanced features of Google Kubernetes Engine, and will show you how to run a service which makes the most of Google Cloud Platform's features. It assumes you have basic familiarity with Docker containers and Kubernetes concepts.

Codelab-at-a-conference setup

If you see a "request account button" at the top of the main Codelabs window, click it to obtain a temporary account. Otherwise ask one of the staff for a coupon with username/password.

These temporary accounts have existing projects that are set up with billing so that there are no costs associated for you with running this codelab.

Note that all these accounts will be disabled soon after the codelab is over.

Use these credentials to log into the machine or to open a new Google Cloud Console window https://console.cloud.google.com/. Accept the new account Terms of Service and any updates to Terms of Service.

Here's what you should see once logged in:

When presented with this console landing page, please select the only project available. Alternatively, from the console home page, click on "Select a Project" :

Google Cloud Shell

While Google Kubernetes Engine can be operated remotely from your laptop, in this codelab we will be using Google Cloud Shell, a command line environment running in the Cloud.

This Debian-based virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. This means that all you will need for this codelab is a browser (yes, it works on a Chromebook).

To activate Google Cloud Shell, from the developer console simply click the button on the top right-hand side (it should only take a few moments to provision and connect to the environment):

Then accept the terms of service and click the "Start Cloud Shell" link:

Once connected to the cloud shell, you should see that you are already authenticated and that the project is already set to your PROJECT_ID :

gcloud auth list

Command output

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)
gcloud config list project

Command output

[core]
project = <PROJECT_ID>

If for some reason the project is not set, simply issue the following command :

gcloud config set project <PROJECT_ID>

Looking for your PROJECT_ID? Check out what ID you used in the setup steps or look it up in the console dashboard:

IMPORTANT: Finally, set the default zone and project configuration:

gcloud config set compute/zone us-central1-f

You can choose a variety of different zones. Learn more in the Regions & Zones documentation.

Enable APIs

In order to use the necessary APIs for this workshop we need to enable a few Google Cloud APIs. Navigate to the APIs & Services page in the Google Cloud Console. Click on the button labeled Enable APIs and Services.

Search for the Kubernetes Engine API. Click on the search result and click the Enable button.

Next, search for the Stackdriver API. Click on the search result and click the Enable button.

Next, search for the Stackdriver Monitoring API. Click on the search result and click the Enable button.

Next, search for the Stackdriver Logging API. Click on the search result and click the Enable button.

Create a Stackdriver Monitoring Account

Select Monitoring from the menu in the Google Cloud Console. This will open Stackdriver Monitoring in a new tab. Here you will be prompted to create an account. Stackdriver Monitoring allows monitoring multiple projects from a single account. For the purposes of this workshop we will only select a single project.

You will be presented with a page like this:

From here just click on the Create Account button.

  1. On the next page click Continue.
  2. On the next page click Skip AWS Setup.
  3. On the next page click Continue.
  4. On the next page select No Reports and click Continue.
  5. After a moment, a Launch Monitoring button should appear. Click on the Launch Monitoring button.
  6. Click on the Continue the free trial button.
  7. Once the Stackdriver Monitoring home page has loaded you can close the tab.

Lets create a Kubernetes cluster. You will use this cluster for all the upcoming exercises.

First, we will set our zone to be used throughout the workshop.

$ gcloud config set compute/zone europe-west1-d

Updated property [compute/zone].

When creating a cluster we will need to specify the software version for our cluster. Kubernetes Engine version numbers look like X.Y.Z.gke.N. The X.Y.Z part indicates the Kubernetes version. The gke.N part specifies a patch version for Kubernetes Engine, which include security updates or bug fixes on top of the open-source upstream Kubernetes.

With Kubernetes Engine you can specify an exact version (e.g. 1.9.0.gke.0), or you can specify a version alias. Here we will specify the 1.9 version alias, which will give us the latest patch version in the 1.9 version series.

$ gcloud container clusters create gke-workshop \
       --enable-network-policy \
       --cluster-version=1.9

Creating cluster gke-workshop...done.                                                                                                                                                                     
Created [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop].
kubeconfig entry generated for gke-workshop.
NAME            LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
gke-workshop    europe-west1-d 1.9.6-gke.0     35.193.171.226  n1-standard-1  1.9.6-gke.0   3          RUNNING

When you create a cluster, gcloud adds a context to your kubectl configuration file (~/.kube/config). It then sets it as the current context, to let you operate on this cluster immediately.

$ kubectl config current-context

gke_codelab_europe-west1-d_gke-workshop

To test it, try a kubectl command line:

$ kubectl get nodes

NAME                                            STATUS    ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready     <none>    6m        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready     <none>    6m        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready     <none>    6m        v1.9.6-gke.0

If you navigate to "Kubernetes Engine" in the Google Cloud Platform console, you will see the cluster listed:

Run a deployment

Let's run a sample deployment to verify that our cluster is working. The kubectl run command is shorthand for creating a Kubernetes deployment without the need for a YAML or JSON spec file.

$ kubectl run hello-web --image=gcr.io/google-samples/hello-app:1.0 \
       --port=8080 --replicas=3

deployment "hello-web" created

$ kubectl get pods -o wide

NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
hello-web-5d9cdb689c-5qh2h   1/1       Running   0          2s        10.60.2.6   gke-gke-workshop-default-pool-1acc373c-1txb
hello-web-5d9cdb689c-sp2bj   1/1       Running   0          2s        10.60.1.9   gke-gke-workshop-default-pool-1acc373c-cklc
hello-web-5d9cdb689c-xfcm5   1/1       Running   0          2s        10.60.1.8   gke-gke-workshop-default-pool-1acc373c-cklc

Look at the Google Cloud dashboard

Workloads that are deployed to your Kubernetes Engine cluster are displayed in the Google Cloud Console. Navigate to Kubernetes Engine, and then Workloads.

You can see the hello-web Deployment we just created. Feel free to click on it, and explore the user interface.

Congrats on deploying your application to GKE!

What we've covered

A Kubernetes Engine cluster consists of a master and nodes. Kubernetes doesn't handle provisioning of nodes, so Google Kubernetes Engine handles this for you with a concept called node pools.

A node pool is a subset of node instances within a cluster that all have the same configuration. They map to instance templates in Google Compute Engine, which provides the VMs used by the cluster. By default a Kubernetes Engine cluster has a single node pool, but you can add or remove them as you wish to change the shape of your cluster.

In the previous example, you created a Kubernetes Engine cluster. This gave us three nodes (three n1-standard-1 VMs, 100 GB of disk each) in a single node pool (called default-pool). Let's inspect the node pool:

$ gcloud container node-pools list --cluster gke-workshop
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.9.6-gke.0

If you want to add more nodes of this type, you can grow this node pool. If you want to add more nodes of a different type, you can add other node pools.

A common method of moving a cluster to larger nodes is to add a new node pool, move the work from the old nodes to the new, and delete the old node pool.

Let's add a second node pool, and migrate our workload over to it. This time we will use the larger n1-standard-2 machine type, but only create one instance.

$ gcloud container node-pools create new-pool --cluster gke-workshop \
    --machine-type n1-standard-2 --num-nodes 3
Creating node pool new-pool...done.                                                                                                                                   
Created [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/new-pool].
NAME         MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
new-pool     n1-standard-1  100           1.9.6-gke.0

$ gcloud container node-pools list --cluster gke-workshop
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.9.6-gke.0
new-pool      n1-standard-2  100           1.9.6-gke.0

$ kubectl get nodes
NAME                                            STATUS    ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready     <none>    56m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready     <none>    56m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready     <none>    57m       v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-l57x       Ready     <none>    1m        v1.9.6-gke.0

Kubernetes does not reschedule Pods as long as they are running and available, so your workload remains running on the nodes in the default pool.

Look at one of your nodes using kubectl describe. Just like you can attach labels to pods, nodes are automatically labelled with useful information which lets the scheduler make decisions and the administrator perform action on groups of nodes.

Replace "[NODE NAME]" with the name of one of your nodes from the previous step.

$ kubectl describe node [NODE NAME] | head -n 20
Name:               gke-gke-workshop-default-pool-1acc373c-1txb
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/fluentd-ds-ready=true
                    beta.kubernetes.io/instance-type=n1-standard-1
                    beta.kubernetes.io/masq-agent-ds-ready=true
                    beta.kubernetes.io/os=linux
                    cloud.google.com/gke-nodepool=default-pool
                    failure-domain.beta.kubernetes.io/region=europe-west1
                    failure-domain.beta.kubernetes.io/zone=europe-west1-d
                    kubernetes.io/hostname=gke-gke-workshop-default-pool-1acc373c-1txb
                    projectcalico.org/ds-ready=true
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
<...>

You can also select nodes by node pool using the cloud.google.com/gke-nodepool label. We'll use this powerful construct shortly.

$ kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool
NAME                                          STATUS    ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready     <none>    54m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready     <none>    54m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready     <none>    54m       v1.9.6-gke.0

Migrating pods to the new Node Pool

To migrate your pods to the new node pool, we will perform the following steps:

  1. Cordon the existing node pool: This operation marks the nodes in the existing node pool (default-pool) as unschedulable. Kubernetes stops scheduling new Pods to these nodes once you mark them as unschedulable.
  2. Drain the existing node pool: This operation evicts the workloads running on the nodes of the existing node pool (default-pool) gracefully.

You could cordon an individual node using the kubectl cordon command, but running this command on each node individually would be tedious. To speed up the process, we can embed the command in a loop. Be sure you copy the whole line - it will have scrolled off the screen to the right!

$ for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool -o=name); do kubectl cordon "$node"; done
node "gke-gke-workshop-default-pool-1acc373c-1txb" cordoned
node "gke-gke-workshop-default-pool-1acc373c-cklc" cordoned
node "gke-gke-workshop-default-pool-1acc373c-kzjv" cordoned

This loop utilizes the command kubectl get nodes to select all nodes in the default pool (using the cloud.google.com/gke-nodepool=default-pool label), and then it iterates through and runs kubectl cordon on each one.

After running the loop, you should see that the default-pool nodes have SchedulingDisabled status in the node list:

$ kubectl get nodes
NAME                                            STATUS                     ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready,SchedulingDisabled   <none>    1h        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready,SchedulingDisabled   <none>    1h        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready,SchedulingDisabled   <none>    1h        v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-l57x       Ready                      <none>    10m       v1.9.6-gke.0

Next, we want to evict the Pods already scheduled on each node. To do this, we will construct another loop, this time using the kubectl drain command:

$ for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool -o=name); do kubectl drain --force --ignore-daemonsets --delete-local-data "$node"; done

<...>
pod "hello-web-59d96f7bd6-scknr" evicted
pod "hello-web-59d96f7bd6-v5dbq" evicted
pod "hello-web-59d96f7bd6-p92b6" evicted
node "gke-gke-workshop-default-pool-cf484031-cl8m" drained
<...>

As each node is drained, the pods running on it are evicted. Eviction makes sure to follow rules to provide the least disruption to the applications as possible. Users in production may want to look at more advanced features like Pod Disruption Budgets.

Because the default node pool is unschedulable, the pods are now running on the single machine in the new node pool:

$ kubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
hello-web-5d9cdb689c-54vnv   1/1       Running   0          2m        10.60.6.5   gke-gke-workshop-new-pool-97a76573-l57x
hello-web-5d9cdb689c-pn25c   1/1       Running   0          2m        10.60.6.8   gke-gke-workshop-new-pool-97a76573-l57x
hello-web-5d9cdb689c-s6g6r   1/1       Running   0          2m        10.60.6.6   gke-gke-workshop-new-pool-97a76573-l57x

You can now delete the original node pool:

$ gcloud container node-pools delete default-pool --cluster gke-workshop
The following node pool will be deleted.
[default-pool] in cluster [gke-workshop] in [europe-west1-d]

Do you want to continue (Y/n)?  y

Deleting node pool default-pool...done.                                                                                                                                  
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/default-pool].

Creating a secondary node pool with GPUs

In addition to migrating from one node pool to another there may be situations where you would like to have a subset of your nodes to have a different configuration. For instance, perhaps some of your applications require the use of GPU hardware. However, it would be unnecessarily expensive if you were to attach GPUs to all nodes in your cluster.

In that case you can create a separate pool of nodes that have GPUs attached and schedule pods to use those GPU enabled nodes. Google Compute Engine allows you to attach up to 8 GPUs per node, assuming you have quota and the GPU device type is available in the zone you are using.

Let's first create a node pool where each node has one GPU attached:

$ gcloud beta container node-pools create nvidia-tesla-k80-pool --cluster gke-workshop --machine-type n1-standard-1 --num-nodes 1 --accelerator type=nvidia-tesla-k80,count=1

Machines with GPUs have certain limitations which may affect your workflow.
Learn more at https://cloud.google.com/kubernetes-engine/docs/concepts/gpus

Creating node pool nvidia-tesla-k80-pool...done.                                                       
Created [https://container.googleapis.com/v1beta1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/nvidia-tesla-k80-pool].
NAME                    MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
nvidia-tesla-k80-pool  n1-standard-1  100           1.9.6-gke.0

Next, we must install the NVIDIA drivers.

$ kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/nvidia-driver-installer/cos/daemonset-preloaded.yaml

daemonset "nvidia-driver-installer" created

Verify that the drivers are installed using the following command. Run the following commands to watch the status of the pod. When the pods are listed as 'Running", you will know that the drivers are finished installing. When you are done hit Ctrl-C to exit:

$ watch "kubectl get pods -n kube-system | grep nvidia-driver-installer"

nvidia-driver-installer-98ksb                                    1/1       Running   0          10m

You can verify that the new node has GPUs that are allocatable to pods with the following command:

$ kubectl get nodes -l cloud.google.com/gke-nodepool=nvidia-tesla-k80-pool -o yaml | grep allocatable -A4

    allocatable:
      cpu: 940m
      memory: 2708864Ki
      nvidia.com/gpu: "1"
      pods: "110"

Now let's create a pods that can consume GPUs:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vector-add
      image: "k8s.gcr.io/cuda-vector-add:v0.1"
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU
EOF

The pod should complete fairly quickly. You can verify that it was completed by watching the pod with the following command. Here we can see that the pod was run on one of the nodes with a GPU. When the pod enters 'Completed' status you can hit Ctrl-C to exit:

$ watch kubectl get pods --show-all -o wide

NAME              READY     STATUS      RESTARTS   AGE       IP          NODE
cuda-vector-add   0/1       Completed   0          10m       10.56.8.3   gke-gke-workshop-nvidia-tesla-k80-po-b310269d-gbs1

Verify the logs for the pod:

$ kubectl logs cuda-vector-add

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Finally we will delete the GPU node pool:

$ gcloud container node-pools delete nvidia-tesla-k80-pool --cluster=gke-workshop

The following node pool will be deleted.
[nvidia-tesla-k80-pool] in cluster [gke-workshop] in [europe-west1-d]

Do you want to continue (Y/n)? Y

Deleting node pool nvidia-tesla-k80-pool...done.                                                                                                                                                                 
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/nvidia-tesla-k80-pool].

Congratulations — you are now a master at node pools!

What we've covered

Google Kubernetes Engine comes with built-in integration with Stackdriver Logging & Monitoring. When you create a cluster these features are enabled by default.

When Stackdriver Logging support is enabled, all logs from containers in your cluster are exported to Stackdriver Logging. You can then view the logs using the Stackdriver Logging UI and perform powerful queries on your logging output. Kubernetes Engine also includes support for viewing logs emitted by the kubelet on each node in the cluster as well as audit logs from the API master.

Stackdriver Monitoring allows you to view information, such as the memory and CPU usage, of individual pods running in your cluster.

Examining Container Logs

First let's create a workload in the cluster to generate some logs. We will create an nginx Deployment and corresponding Service in order to emit some logs.

$ kubectl run nginx --image=nginx --expose --port=80 --service-overrides='{"spec":{"type":"LoadBalancer"}}'

service "nginx" created
deployment "nginx" created

Next check for the public IP of our nginx Service. When the IP address is returned you can hit Ctrl-C to exit.

$ watch kubectl get svc nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

123.123.123.123

In your terminal run the following to generate load on the nginx Pod. Enter the IP address of your service instead of the IP address below.

$ while true; do curl -s http://123.123.123.123/ > /dev/null; done

This will run in a never ending loop sending requests to our nginx Pod. You should now be able to view the logs for the Pod. View the Stackdriver Logs by selecting Logging > Logs from the menu in the Google Cloud Platform console. You should see a page like the following:

From here select the first dropdown filter and select GKE Container > gke-workshop > default. This will show all logs from the 'default' Kubernetes namespace for our 'gke-workshop' cluster. Here we should see the logs generated by our nginx instance. If we had other Pods running in the cluster we would see those logs here as well. You can further limit which containers to show logs from by using the filter that says 'All logs'.

Viewing Metrics in Stackdriver Monitoring

Next, let's view the metrics for our pod in Stackdriver Monitoring. Navigate to Monitoring in the menu for the Google Cloud Platform Console. A new tab should be opened for Stackdriver Monitoring. Within Stackdriver Monitoring navigate to Resources > Kubernetes Engine. You should see a list of Pods. One of the pods should look like nginx-XXXXXXXXX-YYYYY. Click on that pod in the list.

This page will show you CPU, Memory, and Disk usage for the Pod. This data is aggregated for all containers in the Pod. If you have multiple containers in a Pod you can drill down on individual containers by clicking on them in the 'Containers' list.

You can also view these same metrics via the Workloads page. Navigate to Kubernetes Engine and then Workloads. Drill down on a workload by clicking on it in the cloud console.

When you are finished, stop sending requests to nginx by hitting Ctrl-C in the terminal to break the loop. Then you can delete the nginx Deployment.

$ kubectl delete deployment,service nginx

deployment "nginx" deleted
service "nginx" deleted

What we've covered

Kubernetes allows autoscaling with custom metrics. Kubernetes collects and exports information about the amount of CPU your application uses but sometimes this is not the right metric for autoscaling. Custom metrics allow you to autoscale applications any type of metric that your application can export. In this section we will export a metric to Stackdriver as a custom metric and use the custom metrics feature of HorizontalPodAutoscaler to autoscale our Pods.

First we will deploy the custom metric adapter for Stackdriver.

$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/adapter-beta.yaml
<...>

In order to use a Stackdriver custom metric as a metric for autoscaling, the metric must meet the following requirements.

Let's create our application. This application exports a static value to a custom metric. We will start out with a static value of 40.

$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: custom-metric-sd
  name: custom-metric-sd
  namespace: default
spec:
  selector:
    matchLabels:
      run: custom-metric-sd
  template:
    metadata:
      labels:
        run: custom-metric-sd
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - ./direct-to-sd --metric-name=foo --metric-value=40 --pod-id=\$(POD_ID)
        image: gcr.io/google-samples/sd-dummy-exporter:latest
        name: sd-dummy-exporter
        resources:
          requests:
            cpu: 100m
        env:
          - name: POD_ID
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.uid
EOF

deployment "custom-metric-sd" created

This application will start with a single Pod replica.

$ kubectl get pods | grep custom-metric-sd

custom-metric-sd-7b9b98f96d-xw2wp   1/1       Running   0          1m

Autoscale the Deployment

Next we will create a HorizontalPodAutoscaler to autoscale our Deployment. The HorizontalPodAutoscaler has a target average value of 20 per Pod.

$ cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: custom-metric-sd
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: custom-metric-sd
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: foo
      targetAverageValue: 20

EOF

horizontalpodautoscaler "custom-metric-sd" created

Since the target average value of the autoscaler is 20 and our application is exporting a value of 40 we should quickly see that our application will begin scaling up from the initial single replica. Continue watching for a couple minutes to see the deployment scale up. HorizontalPodAutoscaler has a default upscale delay of 3 minutes. You can go on to the next step while continuing to watch the pods.

$ watch 'kubectl get pods | grep custom-metric-sd'

custom-metric-sd-7b9b98f96d-mvs8j   1/1       Running   0          1m
custom-metric-sd-7b9b98f96d-xw2wp   1/1       Running   0          6m

Viewing the Custom Metric in Stackdriver Monitoring

You can view the exported metric in Stackdriver. Navigate to the Monitoring menu item in the Cloud Console. From there navigate to Resources > Metrics Explorer. In the Metrics Explorer search for 'custom/foo'. You can then view the currently recorded values for the custom metric.

Viewing the results of Auto-Scaling

Coming back to our watch command, if it has been long enough we should see that the deployment has scaled up some more.

$ watch 'kubectl get pods | grep custom-metric-sd'

custom-metric-sd-7b9b98f96d-8c6dp   1/1       Running   0          3m 
custom-metric-sd-7b9b98f96d-mb7bh   1/1       Running   0          3m 
custom-metric-sd-7b9b98f96d-mvs8j   1/1       Running   0          6m
custom-metric-sd-7b9b98f96d-xw2wp   1/1       Running   0          11m

Let's now update our Deployment to change the statically exported value to 20.

$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: custom-metric-sd
  name: custom-metric-sd
  namespace: default
spec:
  selector:
    matchLabels:
      run: custom-metric-sd
  template:
    metadata:
      labels:
        run: custom-metric-sd
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - ./direct-to-sd --metric-name=foo --metric-value=20 --pod-id=\$(POD_ID)
        image: gcr.io/google-samples/sd-dummy-exporter:latest
        name: sd-dummy-exporter
        resources:
          requests:
            cpu: 100m
        env:
          - name: POD_ID
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.uid
EOF

deployment "custom-metric-sd" configured

The application will do a rolling update to update the pods to export the new value. If we then watch the pods for a few minutes we should see that the number of pods has leveled out and stopped scaling. Be sure to wait at least 3 minutes to verify that the Deployment has stopped auto-scaling.

$ watch 'kubectl get pods | grep custom-metric-sd'

custom-metric-sd-65b65f9b86-58f4t   1/1       Running   0          5m
custom-metric-sd-65b65f9b86-5nbhf   1/1       Running   0          5m
custom-metric-sd-65b65f9b86-j44k9   1/1       Running   0          5m
custom-metric-sd-65b65f9b86-m6bp5   1/1       Running   0          5m

If we view the custom metric in Stackdriver Monitoring again we should see the new value being reflected in the graph.

Finally let's clean up the Deployment and custom metrics adapter.

$ kubectl delete horizontalpodautoscaler custom-metric-sd

horizontalpodautoscaler "custom-metric-sd" deleted

$ kubectl delete deployment custom-metric-sd

deployment "custom-metric-sd" deleted

$ kubectl delete namespace custom-metrics

namespace "custom-metrics" deleted

What we've covered

Google Kubernetes Engine includes the ability to scale the cluster based on scheduled workloads. Kubernetes Engine's cluster autoscaler automatically resizes clusters based on the demands of the workloads you want to run.

Let us simulate some additional load to our web application by increasing the number of replicas:

$ kubectl scale deployment hello-web --replicas=20
deployment "hello-web" scaled
Allocatable:
 cpu:     940m
 memory:  2709028Ki
 pods:    110

Even with our larger machine, this is more work than we have space in our cluster to handle:

$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
hello-web-967542450-0xvdf   1/1       Running   0          1m
hello-web-967542450-3p6xf   0/1       Pending   0          1m
hello-web-967542450-5wnkc   0/1       Pending   0          1m
hello-web-967542450-7cwl6   0/1       Pending   0          1m
hello-web-967542450-9jq4b   1/1       Running   0          16m
hello-web-967542450-gbglq   0/1       Pending   0          1m
hello-web-967542450-gvmcg   1/1       Running   0          1m
hello-web-967542450-hd07p   1/1       Running   0          1m
hello-web-967542450-hgr0x   1/1       Running   0          1m
hello-web-967542450-hxkv8   1/1       Running   0          1m
hello-web-967542450-jztp8   0/1       Pending   0          1m
hello-web-967542450-k1p01   0/1       Pending   0          1m
hello-web-967542450-kplff   1/1       Running   0          1m
hello-web-967542450-nbcsg   1/1       Running   0          1m
hello-web-967542450-pks0s   1/1       Running   0          1m
hello-web-967542450-pwh42   1/1       Running   0          17m
hello-web-967542450-v44dl   1/1       Running   0          17m
hello-web-967542450-w7293   1/1       Running   0          1m
hello-web-967542450-wx1m5   0/1       Pending   0          1m
hello-web-967542450-x9595   1/1       Running   0          1m

We can see that there are many pods that are stuck with status of "Pending". This means that Kubernetes has not yet been able to schedule that pod to a node.

Copy the name of one of the pods marked Pending, and look at its events with kubectl describe. You should see a message like the following:

$ kubectl describe pod hello-web-967542450-wx1m5
<...>
Events:
  Message
  -------
  FailedScheduling        No nodes are available that match all of the following predicates:: Insufficient cpu (3).

The scheduler was unable to assign this pod to a node because there is not sufficient CPU space left in the cluster. We can add nodes to the cluster in order to make have enough resources for all of the pods in our Deployment.

Cluster Autoscaler can be enabled when creating a cluster, or you can enable it by updating an existing node pool. We will enable cluster autoscaler on our new node pool.

$ gcloud container clusters update gke-workshop --enable-autoscaling \
      --min-nodes=0 --max-nodes=5 --node-pool=new-pool

Once autoscaling is enabled, Kubernetes Engine will automatically add new nodes to your cluster if you have created new Pods that don't have enough capacity to run; conversely, if a node in your cluster is underutilized, Kubernetes Engine can scale down the cluster by deleting the node.

After the command above completes, we can see that the autoscaler has noticed that there are pods in Pending, and creates new nodes to give them somewhere to go. After a few minutes, you will see a new node has been created, and all the pods are now Running. Hit Ctrl-C when you are done:

$ watch kubectl get nodes,pods
NAME                                           STATUS    AGE       VERSION
gke-gke-workshop-new-pool-97a76573-l57x      Ready     1h        v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-t2v0      Ready     1m        v1.9.6-gke.0

NAME                        READY     STATUS    RESTARTS   AGE
hello-web-967542450-0xvdf   1/1       Running   0          17m
hello-web-967542450-3p6xf   1/1       Running   0          17m
hello-web-967542450-5wnkc   1/1       Running   0          17m
hello-web-967542450-7cwl6   1/1       Running   0          17m
hello-web-967542450-9jq4b   1/1       Running   0          32m
hello-web-967542450-gbglq   1/1       Running   0          17m
hello-web-967542450-gvmcg   1/1       Running   0          17m
hello-web-967542450-hd07p   1/1       Running   0          17m
hello-web-967542450-hgr0x   1/1       Running   0          17m
hello-web-967542450-hxkv8   1/1       Running   0          17m
hello-web-967542450-jztp8   1/1       Running   0          17m
hello-web-967542450-k1p01   1/1       Running   0          17m
hello-web-967542450-kplff   1/1       Running   0          17m
hello-web-967542450-nbcsg   1/1       Running   0          17m
hello-web-967542450-pks0s   1/1       Running   0          17m
hello-web-967542450-pwh42   1/1       Running   0          33m
hello-web-967542450-v44dl   1/1       Running   0          33m
hello-web-967542450-w7293   1/1       Running   0          17m
hello-web-967542450-wx1m5   1/1       Running   0          17m
hello-web-967542450-x9595   1/1       Running   0          17m

Cluster Autoscaler will scale down as well as up. When you enabled the autoscaler, you set a minimum of one node. If you were to resize to one Pod, or delete the Deployment and wait about 10 minutes, you would see that all but one of your nodes are considered unnecessary, and removed.

What we've covered

Preemptible VMs are Google Compute Engine VM instances that last a maximum of 24 hours and provide no availability guarantees. Preemptible VMs are priced substantially lower than standard Compute Engine VMs and offer the same machine types and options.

If your workload can handle nodes disappearing, using Preemptible VMs with the Cluster Autoscaler lets you run work at a lower cost. To specify that you want to use Preemptible VMs you simply use the --preemptible flag when you create the node pool. But if you're using Preemptible VMs to cut costs, then you don't need them sitting around idle. So let's create a node pool of Preemptible VMs that starts with zero nodes, and autoscales as needed.

Hold on though: before we create it, how do we schedule work on the Preemptible VMs? These would be a special set of nodes for a special set of work - probably low priority or batch work. For that we'll use a combination of a NodeSelector and taints/tolerations. Preemptible nodes are currently in beta so we will use the beta version of the Kubernetes Engine API and the beta commands in gcloud. The full command we'll run is:

$ gcloud config set container/use_v1_api false

Updated property [container/use_v1_api].

$ gcloud beta container node-pools create preemptible-pool \
    --cluster gke-workshop --preemptible --num-nodes 0 \
    --enable-autoscaling --min-nodes 0 --max-nodes 5 \
    --node-taints=pod=preemptible:PreferNoSchedule

Creating node pool preemptible-pool...done.
Created [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/preemptible-pool].
NAME               MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
preemptible-pool   n1-standard-1  100           1.9.6-gke.0

$ kubectl get nodes 

NAME                                                STATUS    ROLES     AGE       VERSION
gke-gke-workshop-new-pool-97a76573-l57x           Ready     <none>    1h        v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-t2v0           Ready     <none>    1h        v1.9.6-gke.0

We now have two node pools, but the new "preemptible" pool is autoscaled and is sized to zero initially so we only see the three nodes from the autoscaled node pool that we created in the previous section.

Usually as far as Kubernetes is concerned, all nodes are valid places to schedule pods. We may prefer to reserve the preemptible pool for workloads that are explicitly marked as suiting preemption — workloads which can be replaced if they die, versus those that generally expect their nodes to be long-lived.

To direct the scheduler to schedule pods onto the nodes in the preemptible pool we must first mark the new nodes with a special label called a taint. This makes the scheduler avoid using it for certain Pods.


We can then mark pods that we want to run on the preemptible nodes with a matching toleration, which says they are OK to be assigned to nodes with that taint.

Let's create a new workload that's designed to run on preemptible nodes and nowhere else.

$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: hello-web
  name: hello-preempt
spec:
  replicas: 20
  selector:
    matchLabels:
      run: hello-web
  template:
    metadata:
      labels:
        run: hello-web
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        name: hello-web
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          requests:
            cpu: "50m"
      tolerations:
      - key: pod
        operator: Equal
        value: preemptible
        effect: PreferNoSchedule
      nodeSelector:
        cloud.google.com/gke-preemptible: "true"
EOF

deployment "hello-preempt" created

$ watch kubectl get nodes,pods -o wide

NAME                                              STATUS    ROLES     AGE <...>      
gke-gke-workshop-new-pool-97a76573-l57x           Ready     <none>    1h <...>
gke-gke-workshop-new-pool-97a76573-t2v0           Ready     <none>    1h <...>
gke-gke-workshop-preemptible-pool-8f5d3454-2wmb   Ready     <none>    16m <...>
gke-gke-workshop-preemptible-pool-8f5d3454-9bcw   Ready     <none>    16m <...>
gke-gke-workshop-preemptible-pool-8f5d3454-rk9c   Ready     <none>    14m <...>

NAME                            READY     STATUS    RESTARTS   AGE       IP            NODE
hello-preempt-79cbc76b4-45msl   1/1       Running   0          6s        10.60.7.18    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-78z7s   1/1       Running   0          6s        10.60.9.18    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-8cjlr   1/1       Running   0          6s        10.60.8.21    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-fdtmx   1/1       Running   0          6s        10.60.11.12   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-fw6cg   1/1       Running   0          6s        10.60.7.21    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-gqnvp   1/1       Running   0          6s        10.60.10.20   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-hb72t   1/1       Running   0          6s        10.60.11.13   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-kps8r   1/1       Running   0          6s        10.60.7.20    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-mnfvv   1/1       Running   0          6s        10.60.10.22   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-plxsj   1/1       Running   0          6s        10.60.8.22    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-pxw2w   1/1       Running   0          6s        10.60.10.23   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-sqcst   1/1       Running   0          6s        10.60.11.14   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-tnmdt   1/1       Running   0          6s        10.60.7.19    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-v4wjw   1/1       Running   0          6s        10.60.9.20    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-vg976   1/1       Running   0          6s        10.60.11.11   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-vkjv7   1/1       Running   0          6s        10.60.9.19    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-w6jvc   1/1       Running   0          6s        10.60.8.23    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-x5hcs   1/1       Running   0          6s        10.60.10.21   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-x6v5t   1/1       Running   0          6s        10.60.8.20    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-z5sxm   1/1       Running   0          6s        10.60.9.21    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb

Due to the NodeSelector, initially there were no nodes on which we could schedule the work. The scheduler works in tandem with the Cluster Autoscaler to provision new nodes in the pool with the node labels that match the NodeSelector. We haven't demonstrated it here, but the taint would mean prefer to prevent workloads with pods that don't tolerate the taint from being scheduled on these nodes.

As we do the cleanup for this section, let's delete the preemptible node pool and see what happens to the pods that we just created. This isn't something you would want to do in production!

$ gcloud container node-pools delete preemptible-pool --cluster \
    gke-workshop

The following node pool will be deleted.
[preemptible-pool] in cluster [gke-workshop] in [europe-west1-d]
Do you want to continue (Y/n)?

Deleting node pool preemptible-pool...|

$ watch kubectl get pods 

NAME                            READY     STATUS    RESTARTS   AGE
hello-preempt-79cbc76b4-2292l   0/1       Pending   0          3m
hello-preempt-79cbc76b4-29njq   0/1       Pending   0          3m
hello-preempt-79cbc76b4-42vv6   0/1       Pending   0          3m
hello-preempt-79cbc76b4-47fgt   0/1       Pending   0          23m
hello-preempt-79cbc76b4-4d9wn   0/1       Pending   0          23m
hello-preempt-79cbc76b4-4kqsr   0/1       Pending   0          23m
hello-preempt-79cbc76b4-5hh6f   0/1       Pending   0          3m
...

As you can see, because of the NodeSelector, none of the pods are running. Now, delete the deployment.

$ kubectl delete deployment hello-preempt

deployment "hello-preempt" deleted

What we've covered

To connect to services in Google Cloud Platform, you need to provide an identity. While you might use a user's identity if operating interactively, services running on Compute Engine instances normally use an IAM service account.

Applications running on Compute Engine can access and use the service account associated with the instance, and as Kubernetes Engine nodes are Compute Engine instances, containers can access the identities provided by the node.

However, granting that identity access to a service could grant unnecessary privileges because any container that runs on the node will have access to it.

A better practice is to create a service account for your own application, and provide that to your application using Kubernetes secrets.

We will use a sample application which reads messages posted to a Google Cloud Pub/Sub topic. Cloud Pub/Sub is a simple, reliable, scalable foundation for stream analytics and event-driven computing systems. It's a global service, which makes it great for exchanging data between clusters in different regions.

Create a Pub/Sub topic

The Pub/Sub subscriber application you will deploy uses a subscription named echo-read on a Pub/Sub topic called echo. Create these resources before deploying the application:

$ gcloud pubsub topics create echo
$ gcloud pubsub subscriptions create echo-read --topic=echo

Deploy an application

Our sample application reads messages that are published to a Pub/Sub topic. This application is written in Python using Google Cloud Pub/Sub client libraries and you can find the source code on GitHub.

$ kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes-engine-samples/master/cloud-pubsub/deployment/pubsub.yaml

deployment "pubsub" created

Look at the pod:

$ kubectl get pods -l app=pubsub

NAME                     READY     STATUS    RESTARTS   AGE
pubsub-8dc9647d6-lzvvb   0/1       Pending   0          39s

You can see that the container is failing to start and went into a CrashLoopBackOff state. Inspect the logs from the Pod by running:

$ kubectl logs -l app=pubsub
...
google.gax.errors.RetryError: GaxError(Exception occurred in retry method
that was not classified as transient, caused by <_Rendezvous of RPC that
terminated with (StatusCode.PERMISSION_DENIED, Request had insufficient
authentication scopes.)>)

The stack trace and the error message indicates that the application does not have permissions to query the Cloud Pub/Sub service. This is because the "Compute Engine default service account" is not assigned any roles giving it permission to Cloud Pub/Sub.

$ PROJECT=$(gcloud config get-value project)
<...>
$ gcloud iam service-accounts create pubsub-sa --display-name "Pub/Sub demo service account"
<...>
$ gcloud projects add-iam-policy-binding $PROJECT \
    --member serviceAccount:pubsub-sa@$PROJECT.iam.gserviceaccount.com \
    --role roles/pubsub.subscriber
<...>
$ gcloud iam service-accounts keys create key.json \
    --iam-account pubsub-sa@$PROJECT.iam.gserviceaccount.com

Import secret into Kubernetes

Load in the Secret:

$ kubectl create secret generic pubsub-key --from-file=key.json=key.json

We now have a secret called pubsub-key which contains a file called key.json. You can update your Deployment to mount this Secret, and you can override the application to use this service account instead of the default identity of the node.

Configure the application with the Secret

This manifest file defines the following to make the credentials available to the application:

       volumeMounts:
        - name: google-cloud-key
          mountPath: /var/secrets/google
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/key.json

Apply this configuration:

$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes-engine-samples/master/cloud-pubsub/deployment/pubsub-with-secret.yaml

Your pod will now start:

$ kubectl get pods -l app=pubsub
NAME                     READY     STATUS    RESTARTS   AGE
pubsub-94476cd97-kzd9j   1/1       Running   0          1m

Test receiving Pub/Sub messages

Validate that your application is now able to read from Google Cloud Pub/Sub:

$ gcloud pubsub topics publish echo --message "Hello world"
NAME                     READY     STATUS    RESTARTS   AGE
pubsub-94476cd97-kzd9j   1/1       Running   0          1m
$ kubectl logs -l app=pubsub
Pulling messages from Pub/Sub subscription...
[2017-11-28 21:44:10.537970] ID=177003642109951 Data=b'Hello, world'

You have successfully configured an application on Kubernetes Engine to authenticate to Pub/Sub API using service account credentials!

Delete your subscriptions and your pod:

$ gcloud pubsub subscriptions delete echo-read
$ gcloud pubsub topics delete echo
$ kubectl delete pods -l app=pubsub

What we've covered

To publish a service to the world via HTTP, Kubernetes has an object called an Ingress. The Ingress object and its associated controller programs a load balancer, like the Google Cloud Platform HTTP Load Balancer.

Deploy an application


Let's redeploy our faithful hello-web app:

$ kubectl run hello-web --image=gcr.io/google-samples/hello-app:1.0 --port=8080 --replicas=3

deployment "hello-web" created

Reserve a static IP address

When you create an Ingress, you will be allocated a random IP address. It's good practice to instead reserve an IP address, and then allocate that to your Ingress, so that you don't need to change DNS if you ever need to delete and recreate your load balancer.

Google Cloud Platform has two types of IP addresses - regional (for Network Load Balancers, as used by Services) and global (for HTTP Load Balancers).

Reserve a static external IP address named nginx-static-ip by running:

$ gcloud compute addresses create hello-web-static-ip --global

$ gcloud compute addresses list --filter="name=hello-web-static-ip"
NAME                 REGION  ADDRESS       STATUS
hello-web-static-ip          35.227.247.6  RESERVED

This is the value you would program into DNS.

Create a Service

We now need to tell Kubernetes that these pods comprise a Service, which can be used as a target for our traffic.

In many other examples you might have used --type=LoadBalancer. That creates a Network Load Balancer, which exists in one region. By using a Service of type NodePort, we are instead exposing the service to the VM IPs, and we can then direct an Ingress object, which corresponds to a global HTTP(s) Load Balancer, to the NodePort service on those VMs.

$ kubectl expose deployment hello-web --target-port=8080 --type=NodePort

$ kubectl get service hello-web
NAME        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
hello-web   NodePort   10.63.253.73   <none>        8080:32231/TCP   2m

In this example, you can see that any traffic from the Internet to port 32231 on any node will be routed to a healthy container on port 8080. There is, however, no "EXTERNAL-IP" allocated which would have been the case if a Service of type LoadBalancer would have been used.

Create an Ingress

In order to reach the service on a single IP, we will create an Ingress object wiring together the static IP and the service:

$ cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-web-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: hello-web-static-ip
spec:
  backend:
    serviceName: hello-web
    servicePort: 8080
EOF

After a couple of minutes, the HTTP Load Balancer will be created, using the IP you reserved previously, and mapped to the ports exposed by the NodePort service. 

$ watch kubectl get ingress
NAME                HOSTS     ADDRESS        PORTS     AGE
hello-web-ingress   *         35.227.247.6   80        2m

You can now hit (for example) http://35.227.247.6/:

If you no longer need them, delete the Ingress and release the static IP:

$ kubectl delete ingress hello-web-ingress
$ gcloud compute addresses delete hello-web-static-ip --global

What we've covered

Enjoy your new-found GKE skills!