In this code lab, you create a new Cloud Run service, collage service, that will be triggered by Cloud Scheduler at a regular interval of time. The service fetches the latest pictures uploaded and create a collage of those pictures.

What you'll learn

Codelab-at-a-conference setup

If you see a "request account button" at the top of the main Codelabs window, click it to obtain a temporary account. Otherwise ask one of the staff for a coupon with username/password.

These temporary accounts have existing projects that are set up with billing so that there are no costs associated for you with running this codelab.

Note that all these accounts will be disabled soon after the codelab is over.

Use these credentials to log into the machine or to open a new Google Cloud Console window https://console.cloud.google.com/. Accept the new account Terms of Service and any updates to Terms of Service.

Here's what you should see once logged in:

When presented with this console landing page, please select the only project available. Alternatively, from the console home page, click on "Select a Project" :

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

You will need a Cloud Scheduler to trigger the Cloud Run service on a regular interval. Make sure it is enabled:

gcloud services enable cloudscheduler.googleapis.com

You should see the operation to finish successfully:

Operation "operations/acf.5c5ef4f6-f734-455d-b2f0-ee70b5a17322" finished successfully.

Clone the code, if you haven't already:

git clone https://github.com/GoogleCloudPlatform/serverless-photosharing-workshop

You will have the following file layout for the service:

services
 |
 ├── collage
      |
      ├── Dockerfile
      ├── index.js
      ├── package.json

Inside the folder,, you have 3 files:

Dependencies

The package.json file defines the needed library dependencies:

{
  "name": "collage_service",
  "version": "0.0.1",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@google-cloud/storage": "^4.0.0",
    "@google-cloud/firestore": "^3.4.1",
    "express": "^4.16.4",
    "bluebird": "^3.5.0",
    "imagemagick": "^0.1.3"
  }
}

We depend on the Cloud Storage library to read and save image files within Cloud Storage. We declare a dependency on Cloud Firestore, to fetch picture metadata that we stored previously. Express is a JavaScript / Node web framework. Bluebird is used for handling promises, and imagemagick is a library for manipulating images.

Dockerfile

Dockerfile defines the container image for the application:

FROM node:10-slim

# installing Imagemagick
RUN set -ex; \
  apt-get -y update; \
  apt-get -y install imagemagick; \
  rm -rf /var/lib/apt/lists/*

WORKDIR /picadaily/services/collage
COPY package*.json ./
RUN npm install --production
COPY . .
CMD [ "npm", "start" ]

We're using a light Node 10 base image. We're installing the imagemagick library. Then we're installing the NPM modules needed by our code, and we run our node code with npm start.

index.js

Let's have a closer look at our index.js code:

const express = require('express');
const im = require('imagemagick');
const Promise = require("bluebird");
const path = require('path');
const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
const Firestore = require('@google-cloud/firestore');

We require the various dependencies needed for our program to run: Express is the Node web framework we will be using, ImageMagick the library for doing image manipulation, Bluebird is a library for handling JavaScript promises, Path is used for dealing with files and directories paths, and then Storage and Firestore are for working respectively with Google Cloud Storage (our buckets of images), and the Cloud Firestore datastore.

const app = express();

app.post('/', async (req, res) => {
    try {
        console.log('Collage request');

        /* ... */

    } catch (err) {
        console.log(`Error: creating the collage: ${err}`);
        console.error(err);
        res.status(500).send(err);
    }
});

Above, we have the structure of our Node handler: our app responds to HTTP POST requests. And we're doing a bit of error handling in case something goes wrong. Let's now have a look at what is inside this structure.

const thumbnailFiles = [];
const pictureStore = new Firestore().collection('pictures');
const snapshot = await pictureStore.orderBy('created', 'desc').limit(4).get();

if (snapshot.empty) {
    console.log('Empty collection, no collage to make');
    res.status(204).send("No collage created.");
} else {

    /* ... */

}

We retrieve the 4 latest pictures uploaded by our users, from the metadata stored in Cloud Firerstore. We check if the resulting collection is empty or not, and then proceed further in the else branch of our code.

Let's collect the list of file names:

snapshot.forEach(doc => {
    thumbnailFiles.push(doc.id);
});
console.log(`Picture file names: ${JSON.stringify(thumbnailFiles)}`);

We are going to download each of those files from the thumbnail bucket

const thumbBucket = storage.bucket(process.env.BUCKET_THUMBNAILS);

await Promise.all(thumbnailFiles.map(async fileName => {
    const filePath = path.resolve('/tmp', fileName);
    await thumbBucket.file(fileName).download({
        destination: filePath
    });
}));
console.log('Downloaded all thumbnails');

Once the latest thumbnails are uploaded, we're going to use the ImageMagick library to create a 4x4 grid of those thumbnail pictures. We use the Bluebird library and its Promise implementation to transform the callback-driven code into async / await friendly code, then we await on the promise that is making the image collage:

const collagePath = path.resolve('/tmp', 'collage.png');

const thumbnailPaths = thumbnailFiles.map(f => path.resolve('/tmp', f));
const convert = Promise.promisify(im.convert);
await convert([
    '(', ...thumbnailPaths.slice(0, 2), '+append', ')',
    '(', ...thumbnailPaths.slice(2), '+append', ')',
    '-size', '400x400', 'xc:none', '-background', 'none',  '-append',
    collagePath]);
console.log("Created local collage picture");

As the collage picture has been saved to disk locally in the temporary folder, we now need to upload it to Cloud Storage, and then return a successful response (status code 2xx):

await thumbBucket.upload(collagePath);
console.log("Uploaded collage to Cloud Storage bucket ${process.env.BUCKET_THUMBNAILS}");

res.status(204).send("Collage created.");

Now time to make our Node script listen to incoming requests:

const PORT = process.env.PORT || 8080;

app.listen(PORT, () => {
    console.log(`Started collage service on port ${PORT}`);
});

At the end of our source file, we have the instructions to have Express actually start our web application on the 8080 default port.

Test the code locally to make sure it works before deploying to cloud.

Inside thumbnails folder, install npm dependencies and start the server:

npm install; npm start

If everything went well, it should start the server on port 8080:

Started collage service on port 8080

Inside the folder where Dockerfile is, issue the following command to build the container image:

gcloud builds submit --tag gcr.io/${GOOGLE_CLOUD_PROJECT}/collage_service

Before deploying the service, set a variable for thumbnails bucket:

BUCKET_THUMBNAILS=thumbnails-${GOOGLE_CLOUD_PROJECT}

Run the following command to deploy the container image on Cloud Run:

gcloud run deploy collage-service \
    --image gcr.io/${GOOGLE_CLOUD_PROJECT}/collage_service \
    --platform=managed \
    --region=europe-west1 \
    --no-allow-unauthenticated \
    --update-env-vars BUCKET_THUMBNAILS=${BUCKET_THUMBNAILS}

Note the --no-allow-unauthenticated flag. This makes the Cloud Run service an internal service that will only be triggered by specific service accounts.

Now that the Cloud Run service is ready and deployed, it's time to create the regular schedule, to invoke the service every minute.

First, set some variables we'll need in the next steps. TOPIC_NAME is for the Pub/Sub topic as the communication pipeline and the rest are variables we need along the way:

SERVICE_ACCOUNT=collage-scheduler-sa
SERVICE_NAME=collage-service
SERVICE_URL="$(gcloud run services list --platform managed --filter=${SERVICE_NAME} --format='value(URL)')"

Create a service account:

gcloud iam service-accounts create ${SERVICE_ACCOUNT} \
   --display-name "Collage Scheduler Service Account"

Give service account permission to invoke the Cloud Run service:

gcloud run services add-iam-policy-binding ${SERVICE_NAME} \
   --member=serviceAccount:${SERVICE_ACCOUNT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com \
   --role=roles/run.invoker \
   --platform managed

Create a Cloud Scheduler job to execute every 1 minute:

gcloud beta scheduler jobs create http collage-schedule-job --schedule "* * * * *" \
   --http-method=POST \
   --uri=${SERVICE_URL} \
   --oidc-service-account-email=${SERVICE_ACCOUNT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com \
   --oidc-token-audience=${SERVICE_URL}

To test the setup is working, check in the thumbnails bucket for the collage image. You can also check the logs of the service:

Congratulations! You created a scheduled service: thanks to Cloud Scheduler, which pushes a message every minute on a Pub/Sub topic, your Cloud Run collage service is invoked and is able to append pictures together to create the resulting picture.

What we've covered

Next Steps