In this codelab, you'll learn how to use some advanced features of Google Cloud Security and Privacy APIs, including:

You'll take abridged data from the Enron Corpus, encrypt it and load it into Cloud Storage.

What you'll learn

What you'll use

Codelab-at-a-conference setup

You can obtain a temporary account by clicking on the request account button at the top of the main Codelabs window. These temporary accounts have existing projects that are setup with billing so that there are no costs associated with running this codelab.

Note that all these accounts will be disabled soon after the codelab is over.

Open a new Google Cloud Console window https://console.cloud.google.com/ and login with the account and password provided.

Accept the new account Terms of Service and any updates to Terms of Service.

Here's what you should see once logged in:

At the top of the window, click on "Select a Project".

Next click on the menu "No organization" and select "iocodelabs.com"

Finally click on "All" and select the "Google IO" project as indicated in the image below.

Launch Cloud Shell

While Google Cloud and KMS can be operated remotely from your laptop, in this codelab we will be using Google Cloud Shell, a command line environment running in the Cloud.

This Debian-based virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. This means that all you will need for this codelab is a browser (yes, it works on a Chromebook).

To activate Google Cloud Shell, from the developer console simply click the button on the top right-hand side (it should only take a few moments to provision and connect to the environment):

Once connected to the cloud shell, you should see that you are already authenticated and that the project is already set to your PROJECT_ID :

gcloud auth list

Command output

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)
gcloud config list project

Command output

[core]
project = <PROJECT_ID>

If for some reason the project is not set, simply issue the following command :

gcloud config set project <PROJECT_ID>

Looking for your PROJECT_ID? Check out what ID you used in the setup steps or look it up in the console dashboard:

IMPORTANT: Finally, set the default zone and project configuration:

gcloud config set compute/zone us-central1-f

You can choose a variety of different zones. Learn more in the Regions & Zones documentation.

In order to store the data in this codelab, you'll need your own Cloud Storage bucket. Pick a name for it, such as enron_corpus.

BUCKET_NAME=YOUR_NAME_enron_corpus

Create the bucket by running the following commands in your shell:

gsutil mb gs://${BUCKET_NAME}

Cloud Storage bucket names have to be unique across all buckets, so change YOUR_NAME above to your own name or some other unique identifier. For more information, see the Cloud Storage bucket naming guidelines.

After your bucket has been created, you are ready to go. Running this command should also help to verify that you've got the gsutil command line client setup correctly, authentication is working, and you have write access to the cloud project you're operating under.

The Enron Corpus is a large database of over 600,000 emails generated by 158 employees of the Enron Corporation. This data has been copied to the GCS bucket gs://enron_corpus/.

Let's download one of the source files locally so that we can see what it looks like. Run the command to copy down an email text file:

gsutil cp gs://enron_corpus/allen-p/inbox/1. .

Now tail the downloaded file to verify the email text is there:

tail 1.

This should display the contents of the plaintext mail file. There are two types of files we'll be looking at for this codelab: plaintext mail files and image files. If you're interested, use the same mechanism to check out what is in those other files.

Before you can use KMS, you need to enable KMS in your project. This can be done in the using the gcloud command-line utility. Run the following command in your shell:

gcloud service-management enable cloudkms.googleapis.com

Cloud KMS is now enabled for your project!

In order to encrypt the data, you need to create a KeyRing and a CryptoKey. KeyRings are useful for grouping keys. Keys can be grouped by environment (like test, staging, and prod) or by some other conceptual grouping. For this codelab, our KeyRing will be called test and our CryptoKey will be called codelab.

KEYRING_NAME=test CRYPTOKEY_NAME=codelab

Execute the gcloud command to create the KeyRing. You will specify the location as global, but this can also be a specific region.

gcloud beta kms keyrings create $KEYRING_NAME --location global

Next, using the new KeyRing, create a CryptoKey named codelab.

gcloud beta kms cryptokeys create $CRYPTOKEY_NAME --location global \
      --keyring $KEYRING_NAME \
      --purpose encryption

You've created a KeyRing and CryptoKey! Open the Encryption Keys Web UI and view the resources you've created. This is available under IAM & Admin in the main menu:

The Key Management Web UI allows you to view and manage your CryptoKeys and KeyRings. You will use this UI later when you manage permissions.

Next, let's actually try to encrypt some data! Take the contents of the email we previously looked at and base64 encode it.

PLAINTEXT=$(cat 1. | base64 -w0)

Using the encrypt endpoint, you can send the base64-encoded text you want to encrypt to the specified key. The response will be a JSON payload containing the encrypted text in the attribute ciphertext.

curl -v "https://cloudkms.googleapis.com/v1beta1/projects/$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
  -d "{\"plaintext\":\"$PLAINTEXT\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type: application/json"

Now that your data is encrypted, you can save it to a file and upload it to your GCS bucket. To grab the encrypted text from the JSON response and save it to a file, you will use the command-line utility jq. The response from the previous call can be piped into jq, which can parse out the ciphertext property to the file 1.encrypted.

curl -v "https://cloudkms.googleapis.com/v1beta1/projects/$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
  -d "{\"plaintext\":\"$PLAINTEXT\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type:application/json" \
| jq .ciphertext -r > 1.encrypted

To verify the encrypted data can be decrypted, call the decrypt endpoint to verify the decrypted text matches the original email. The encrypted data has information on which CryptoKey version was used to encrypt it, so the specific version is never supplied to the decrypt endpoint.

curl -v "https://cloudkms.googleapis.com/v1beta1/projects/$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:decrypt" \
  -d "{\"ciphertext\":\"$(cat 1.encrypted)\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type:application/json" \
| jq .plaintext -r | base64 -d

Now that you have verified the text has been encrypted successfully, upload the encrypted file to your GCS bucket.

gsutil cp 1.encrypted gs://${BUCKET_NAME}

In KMS, there are two major permissions to focus on. One permissions allows a user or service account to manage KMS resources, the other allows a user or service account to use keys to encrypt and decrypt data.

The permission to manage keys is cloudkms.admin, and allows anyone with the permission to create KeyRings and create, modify, disable, and destroy CryptoKeys. The permission to encrypt and decrypt is cloudkms.cryptoKeyEncrypterDecrypter, and is used to call the encrypt and decrypt API endpoints.

For this exercise, you will use the current authorized user to assign IAM permissions. To get the current authorized user, run the command below.

USER_EMAIL=$(gcloud auth list --limit=1 2>/dev/null | grep '@' | awk '{print $2}') 

Next, you will assign that user the ability to manage KMS resources. Run the following gcloud command to assign the IAM permission to manage the KeyRing you just created:

gcloud beta kms keyrings add-iam-policy-binding $KEYRING_NAME \
    --location global \
    --member user:$USER_EMAIL \
    --role roles/cloudkms.admin

Since CryptoKeys belong to KeyRings, and KeyRings belong to Projects, a user with a specific role or permission at a higher level in that hierarchy inherits the same permissions on the child resources. That is, a user who has the role of owner on a Project is also an owner on all the KeyRings and CryptoKeys in that project. Similarly, if a user is granted the cloudkms.admin role on a KeyRing, they have the associated permissions on the CryptoKeys in that KeyRing.

Without the cloudkms.cryptoKeyEncrypterDecrypter permission, the authorized user will not be able to use the keys to encrypt or decrypt data. Run the following gcloud command to assign the IAM permission to encrypt and decrypt data for any CryptoKey under the KeyRing you created:

gcloud beta kms keyrings add-iam-policy-binding $KEYRING_NAME \
    --location global \
    --member user:$USER_EMAIL \
    --role roles/cloudkms.cryptoKeyEncrypterDecrypter

Now you can view the assigned permissions in the Key Management Web UI. Click the three vertical dots by the KeyRing and select Edit Permissions.

This will open up a menu where you can see the accounts and permissions for the KeyRing you just added.

Now that you have an understanding of how to encrypt a single file, and have permission to do so, you can run a script to backup all files in a directory. For this example, we will copy all emails for allen-p, encrypt them, and upload them to a Cloud Storage bucket. First, copy all emails for allen-p into your current working directory:

gsutil -m cp -r gs://enron_corpus/allen-p .

Now copy and paste the following into the Cloud Shell terminal to backup and encrypt all the files in the allen-p directory to your Cloud Storage bucket:

MYDIR=allen-p
FILES=$(find $MYDIR -type f -not -name "*.encrypted")
for file in $FILES; do
  PLAINTEXT=$(cat $file | base64 -w0)
  curl -v "https://cloudkms.googleapis.com/v1beta1/projects/$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
    -d "{\"plaintext\":\"$PLAINTEXT\"}" \
    -H "Authorization:Bearer $(gcloud auth application-default print-access-token)" \
    -H "Content-Type:application/json" \
  | jq .ciphertext -r > $file.encrypted
done
gsutil -m cp *.encrypted gs://${BUCKET_NAME}

This script loops over all the files in a given directory, encrypts them using the KMS API, and uploads them to Google Cloud Storage.

After the script completes, you can view the encrypted files in the Google Cloud Storage UI. You should see something like this:

Google Cloud Audit Logging consists of two log streams, Admin Activity and Data Access, which are generated by Google Cloud Platform services to help you answer the question of "who did what, where, and when?" within your Google Cloud Platform projects.

To view the activity for any resource in KMS, Click the three vertical dots by the KeyRing and select View Activity.

This will take you to the Cloud Activity UI, where you should see the creation and all modifications made to the KeyRing.

You've now encrypted and uploaded data using KMS and Cloud Storage!

What we've covered

Next Steps

Learn More

Give us your feedback