HashiCorp Vault is a popular open source tool for secrets management that codifies many of the best practices around secrets management including time-based access controls, principles of least privilege, encryption, dynamic credentials, and much more. This codelab teaches you how to use Vault's Google Cloud KMS plugin to encrypt and decrypt data through Vault using Cloud KMS.

What you'll learn

Codelab-at-a-conference setup

If you see a "request account button" at the top of the main Codelabs window, click it to obtain a temporary account. Otherwise ask one of the staff for a coupon with username/password.

These temporary accounts have existing projects that are set up with billing so that there are no costs associated for you with running this codelab.

Note that all these accounts will be disabled soon after the codelab is over.

Use these credentials to log into the machine or to open a new Google Cloud Console window https://console.cloud.google.com/. Accept the new account Terms of Service and any updates to Terms of Service.

Here's what you should see once logged in:

When presented with this console landing page, please select the only project available. Alternatively, from the console home page, click on "Select a Project" :

Start Cloud Shell

In this codelab you will use Cloud Shell, a free virtualized environment running on Google Cloud. From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. Unless otherwise instructed, run all commands from this shell.

Before deploying Vault in production, first install Vault locally. This will enable you to use the vault CLI locally and will be used to interact with the cluster later.

You could browse to the Vault website, but this section will teach you how to download, verify, and install Vault securely. Even though Vault is downloaded over a TLS connection, it may still be possible for a skilled attacker to compromise the underlying storage system or network transport. For that reason, in addition to serving the binaries over TLS, HashiCorp also signs the checksums of each release with their private key. Thus, to verify the integrity of a download, we must:

  1. Import and trust HashiCorp's GPG public key
  2. Download the Vault binary
  3. Download the Vault checksums
  4. Download the Vault checksum signature
  5. Verify the signature of the checksum against HashiCorp's GPG key
  6. Verify the checksums of the binary against the file

This way, even if an attacker were able to compromise the network transport and underlying storage component, they wouldn't be able to sign the checksums with HashiCorp's GPG key. If this operation is successful, we have an extremely high degree of confidence that the software is untainted.

Since that process can be tedious, we will leverage a Docker container to do it for us. Execute the following command to install Vault locally. We install Vault into $HOME/bin because that will persist between restarts on Cloud Shell.

$ docker run -v $HOME/bin:/software sethvargo/hashicorp-installer vault 1.2.2
$ sudo chown -R $(whoami):$(whoami) $HOME/bin/

Add the bin to our path:

$ export PATH=$HOME/bin:$PATH

Finally, optionally, explore the Vault CLI help. Most Vault commands will not work because there is no Vault server running. Do not start a Vault server yet.

$ vault -h

Start a local, development Vault server. This Vault server runs entirely in memory and does not represent a best practices installation. However, it is useful for getting started quickly and exploring Vault's functionality. We will also create an initial token in Vault with the value of "root", which will be used to authenticate to the Vault server.

$ export VAULT_ADDR=
$ vault server -dev &> vault.log &

Vault is now running in the background. You can query its status to verify:

$ vault status

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
# ...

Enable the Google Cloud KMS API. This only needs to be done once per project to make the API accessible.

$ gcloud services enable cloudkms.googleapis.com

Enable the Cloud KMS secrets engine in Vault.

$ vault secrets enable gcpkms

Create a Cloud KMS key by issuing commands to Vault. During this process, Vault makes the necessary API calls to Cloud KMS to create a key ring, crypto key, and crypto key version. The metadata is stored in Vault, but the actual encryption keys are stored and managed by Google Cloud.

$ vault write gcpkms/keys/my-key \
    key_ring=projects/${GOOGLE_CLOUD_PROJECT}/locations/global/keyRings/vault \

That's it! Vault now has an internal reference to this external Cloud KMS key. When a request is made to encrypt or decrypt data, it will be delegated to Cloud KMS.

You can verify that the key was created using the gcloud CLI:

$ gcloud kms keyrings list --location global


Encrypt plaintext data using this KMS key with Vault. You will receive back ciphertext. Neither Vault nor Google Cloud KMS store the ciphertext. You are responsible for persisting the ciphertext externally, like on the filesystem or in a database.

$ vault write gcpkms/encrypt/my-key plaintext="hello world"

Key            Value
---            -----
ciphertext     CiQA3/Sf9aWbWSI4XK...
key_version    1

When you want to retrieve the plaintext, give Vault the stored ciphertext. Vault will make the necessary API calls to KMS to decrypt the ciphertext. Be sure to replace the ciphertext with the value you received from the encrypt call.

$ vault write gcpkms/decrypt/my-key ciphertext="CiQA3/Sf9aWbWSI4XK..."

Key          Value
---          -----
plaintext    hello world

It is also possible to rotate crypto keys, upgrade ciphertext data to the latest version, and delete crypto key versions.

You learned how to run and configure HashiCorp Vault on Google Cloud to encrypt, decrypt and manage Cloud KMS keys.

Clean up

If you are done exploring, please consider deleting your project.

Learn More


This work is licensed under a Creative Commons Attribution 2.0 Generic License.