In this code lab, you create a new Cloud Run service, collage service, that will be triggered by Cloud Scheduler at a regular interval of time. The service fetches the latest pictures uploaded and creates a collage of those pictures: it finds the list of recent pictures in Cloud Firestore, and then downloads the actual picture files from Cloud Storage.

What you'll learn

Codelab-at-a-conference setup

If you see a "request account button" at the top of the main Codelabs window, click it to obtain a temporary account. Otherwise ask one of the staff for a coupon with username/password.

These temporary accounts have existing projects that are set up with billing so that there are no costs associated for you with running this codelab.

Note that all these accounts will be disabled soon after the codelab is over.

Use these credentials to log into the machine or to open a new Google Cloud Console window https://console.cloud.google.com/. Accept the new account Terms of Service and any updates to Terms of Service.

Here's what you should see once logged in:

When presented with this console landing page, please select the only project available. Alternatively, from the console home page, click on "Select a Project" :

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

You will need a Cloud Scheduler to trigger the Cloud Run service on a regular interval. Make sure it is enabled:

gcloud services enable cloudscheduler.googleapis.com

You should see the operation to finish successfully:

Operation "operations/acf.5c5ef4f6-f734-455d-b2f0-ee70b5a17322" finished successfully.

Clone the code, if you haven't already in the previous code lab:

git clone https://github.com/GoogleCloudPlatform/serverless-photosharing-workshop

You can then go to the directory containing the service:

cd serverless-photosharing-workshop/services/collage/nodejs

You will have the following file layout for the service:

services
 |
 ├── collage
      |
      ├── nodejs
           |
           ├── Dockerfile
           ├── index.js
           ├── package.json

Inside the folder,, you have 3 files:

Dependencies

The package.json file defines the needed library dependencies:

{
  "name": "collage_service",
  "version": "0.0.1",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@google-cloud/storage": "^4.0.0",
    "@google-cloud/firestore": "^3.4.1",
    "express": "^4.16.4",
    "bluebird": "^3.5.0",
    "imagemagick": "^0.1.3"
  }
}

We depend on the Cloud Storage library to read and save image files within Cloud Storage. We declare a dependency on Cloud Firestore, to fetch picture metadata that we stored previously. Express is a JavaScript / Node web framework. Bluebird is used for handling promises, and imagemagick is a library for manipulating images.

Dockerfile

Dockerfile defines the container image for the application:

FROM node:10-slim

# installing Imagemagick
RUN set -ex; \
  apt-get -y update; \
  apt-get -y install imagemagick; \
  rm -rf /var/lib/apt/lists/*

WORKDIR /picadaily/services/collage
COPY package*.json ./
RUN npm install --production
COPY . .
CMD [ "npm", "start" ]

We're using a light Node 10 base image. We're installing the imagemagick library. Then we're installing the NPM modules needed by our code, and we run our node code with npm start.

index.js

Let's have a closer look at our index.js code:

const express = require('express');
const im = require('imagemagick');
const Promise = require("bluebird");
const path = require('path');
const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
const Firestore = require('@google-cloud/firestore');

We require the various dependencies needed for our program to run: Express is the Node web framework we will be using, ImageMagick the library for doing image manipulation, Bluebird is a library for handling JavaScript promises, Path is used for dealing with files and directories paths, and then Storage and Firestore are for working respectively with Google Cloud Storage (our buckets of images), and the Cloud Firestore datastore.

const app = express();

app.post('/', async (req, res) => {
    try {
        console.log('Collage request');

        /* ... */

    } catch (err) {
        console.log(`Error: creating the collage: ${err}`);
        console.error(err);
        res.status(500).send(err);
    }
});

Above, we have the structure of our Node handler: our app responds to HTTP POST requests. And we're doing a bit of error handling in case something goes wrong. Let's now have a look at what is inside this structure.

const thumbnailFiles = [];
const pictureStore = new Firestore().collection('pictures');
const snapshot = await pictureStore
    .where('thumbnail', '==', true)
    .orderBy('created', 'desc')
    .limit(4).get();

if (snapshot.empty) {
    console.log('Empty collection, no collage to make');
    res.status(204).send("No collage created.");
} else {

    /* ... */

}

Our collage service will need at least four pictures (whose thumbnails have been generated), so be sure to upload 4 pictures first.

We retrieve the 4 latest pictures uploaded by our users, from the metadata stored in Cloud Firerstore. We check if the resulting collection is empty or not, and then proceed further in the else branch of our code.

Let's collect the list of file names:

snapshot.forEach(doc => {
    thumbnailFiles.push(doc.id);
});
console.log(`Picture file names: ${JSON.stringify(thumbnailFiles)}`);

We are going to download each of those files from the thumbnail bucket, whose name is coming from an environment variable that we set at deployment time:

const thumbBucket = storage.bucket(process.env.BUCKET_THUMBNAILS);

await Promise.all(thumbnailFiles.map(async fileName => {
    const filePath = path.resolve('/tmp', fileName);
    await thumbBucket.file(fileName).download({
        destination: filePath
    });
}));
console.log('Downloaded all thumbnails');

Once the latest thumbnails are uploaded, we're going to use the ImageMagick library to create a 4x4 grid of those thumbnail pictures. We use the Bluebird library and its Promise implementation to transform the callback-driven code into async / await friendly code, then we await on the promise that is making the image collage:

const collagePath = path.resolve('/tmp', 'collage.png');

const thumbnailPaths = thumbnailFiles.map(f => path.resolve('/tmp', f));
const convert = Promise.promisify(im.convert);
await convert([
    '(', ...thumbnailPaths.slice(0, 2), '+append', ')',
    '(', ...thumbnailPaths.slice(2), '+append', ')',
    '-size', '400x400', 'xc:none', '-background', 'none',  '-append',
    collagePath]);
console.log("Created local collage picture");

As the collage picture has been saved to disk locally in the temporary folder, we now need to upload it to Cloud Storage, and then return a successful response (status code 2xx):

await thumbBucket.upload(collagePath);
console.log("Uploaded collage to Cloud Storage bucket ${process.env.BUCKET_THUMBNAILS}");

res.status(204).send("Collage created.");

Now time to make our Node script listen to incoming requests:

const PORT = process.env.PORT || 8080;

app.listen(PORT, () => {
    console.log(`Started collage service on port ${PORT}`);
});

At the end of our source file, we have the instructions to have Express actually start our web application on the 8080 default port.

Test the code locally to make sure it works before deploying to cloud.

Inside collage/nodejs folder, install npm dependencies and start the server:

npm install; npm start

If everything went well, it should start the server on port 8080:

Started collage service on port 8080

Use CTRL-C to exit.

Inside the collage/nodejs folder where the Dockerfile, is issue the following command to build the container image with Cloud Build:

gcloud builds submit --tag gcr.io/${GOOGLE_CLOUD_PROJECT}/collage_service

Before deploying to Cloud Run, set the Cloud Run region to one of the supported regions and platform to managed:

gcloud config set run/region europe-west1
gcloud config set run/platform managed

You can check that the configuration is set:

gcloud config list

...
[run]
platform = managed
region = europe-west1

Run the following command to deploy the container image on Cloud Run:

export BUCKET_THUMBNAILS=thumbnails-${GOOGLE_CLOUD_PROJECT}
export SERVICE_NAME=collage-service
gcloud run deploy ${SERVICE_NAME} \
    --image gcr.io/${GOOGLE_CLOUD_PROJECT}/collage_service \
    --no-allow-unauthenticated \
    --update-env-vars BUCKET_THUMBNAILS=${BUCKET_THUMBNAILS},PROJECT_ID=${GOOGLE_CLOUD_PROJECT}

Note the --no-allow-unauthenticated flag. This makes the Cloud Run service an internal service that will only be triggered by specific service accounts.

Now that the Cloud Run service is ready and deployed, it's time to create the regular schedule, to invoke the service every minute.

First, set some variables we'll need in the next steps. TOPIC_NAME is for the Pub/Sub topic as the communication pipeline and the rest are variables we need along the way:

export SERVICE_ACCOUNT=collage-scheduler-sa
export SERVICE_URL="$(gcloud run services list --platform managed --filter=${SERVICE_NAME} --format='value(URL)')"

Create a service account:

gcloud iam service-accounts create ${SERVICE_ACCOUNT} \
   --display-name "Collage Scheduler Service Account"

Give service account permission to invoke the Cloud Run service:

gcloud run services add-iam-policy-binding ${SERVICE_NAME} \
   --member=serviceAccount:${SERVICE_ACCOUNT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com \
   --role=roles/run.invoker \
   --platform managed \
   --region=europe-west1

Create a Cloud Scheduler job to execute every 1 minute:

gcloud scheduler jobs create http ${SERVICE_NAME}-job --schedule "* * * * *" \
   --http-method=POST \
   --uri=${SERVICE_URL} \
   --oidc-service-account-email=${SERVICE_ACCOUNT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com \
   --oidc-token-audience=${SERVICE_URL}

You can go to Cloud Scheduler section in Cloud Console to see that it is setup and pointing to Cloud Run service url:

To test the setup is working, check in the thumbnails bucket for the collage image (called collage.png). You can also check the logs of the service:

If you don't intend to continue with the other labs in the series, you can clean up resources to save costs and to be an overall good cloud citizen. You can clean up resources individually as follows.

Delete the service:

gcloud run services delete ${SERVICE_NAME} -q

Delete the Cloud Scheduler job:

gcloud scheduler jobs delete ${SERVICE_NAME}-job -q

Alternatively, you can delete the whole project:

gcloud projects delete ${GOOGLE_CLOUD_PROJECT} 

Congratulations! You created a scheduled service: thanks to Cloud Scheduler, which pushes a message every minute on a Pub/Sub topic, your Cloud Run collage service is invoked and is able to append pictures together to create the resulting picture.

What we've covered

Next Steps