Build a sophisticated Conversational Action for Google Assistant

Actions on Google is a developer platform that lets you create software to extend the functionality of Google Assistant, Google's virtual personal assistant, across more than 1 billion devices, including smart speakers, phones, cars, TVs, headphones, and more. Users engage Assistant in conversation to get things done, like buying groceries or booking a ride. (For a complete list of what's possible, see the Actions directory.) As a developer, you can use Actions on Google to easily create and manage delightful and effective conversational experiences between users and your third-party service.

What you'll build

In the codelab, you'll build a sophisticated Conversational Action that does the following:

Supports deep links to directly launch the user into certain points of dialog.
Uses utilities provided by the Actions on Google platform to fetch the user's name and address him or her personally.
Responds with follow-up questions to further the conversation.
Presents users with a rich visual response complete with sound effects.

Figure 1: The conversational design of the Action we’ll build in this codelab.

What you'll learn

How to locally develop your fulfillment using the Node.js client library
How to deploy Actions using the Firebase command-line
How to create deep links
How to use helper intents
How to add Dialogflow custom entities
How to create sound effects with SSML
How to add Dialogflow follow-up intents with contexts
How to create rich responses (basic cards)
How to store session data

What you'll need

The following tools must be in your environment:

An IDE/text editor of your choice, such as WebStorm, Atom or Sublime
A terminal to run shell commands with NodeJS, npm, and git installed.
A web browser, such as Google Chrome.

In addition, familiarity with JavaScript (ES6) is strongly recommended, although not required, to understand the webhook code that you'll use.

Optional: Get the sample code

You can optionally get the full project code from the GitHub repository.

You're going to start with the Dialogflow intents from the Level 1 codelab, but locally develop and deploy the webhook on your machine using Cloud Functions for Firebase.

Why develop your Actions locally?

In contrast to using the Dialogflow inline editor, you can use a local machine, which gives you more control over your programming and deployment environment. That provides several advantages, including:

A locally developed webhook supports a much larger suite of Actions on Google features.
You are not restricted to only one webhook implementation file (index.js).
Developing in a local environment also lets you follow software engineering best practices, such as using version control, for your Actions project.

Download your base files

To get the base files, run the following command to clone the GitHub repository for the Level 1 codelab.

git clone https://github.com/actions-on-google/codelabs-nodejs

The repository contains the following important files:

level1-complete/functions/index.js, a Javascript file, contains your webhook's fulfillment code. It's the main file that you'll edit to add additional Actions and functionality.
level1-complete/functions/package.json outlines dependencies and other metadata for the Node.js project, but you can ignore it. You should only need to edit that file if you want to use different versions of the Actions on Google client library or other Node.js modules.

For the sake of clarity, rename the /level1-complete directory name to /level2. You can do so by using the mv command in your terminal.

$ cd codelabs-nodejs
$ mv ./level1-complete ./level2

Check your Google permission settings

In order to test the Action that you'll build for this codelab, you need to enable the necessary permissions.

Go to the Activity Controls page.
Sign in with your Google Account, if you have not already done so.
Ensure that the following permissions are enabled:

Web & App Activity (You should also enable the option to Include Chrome history)
Device Information
Voice & Audio Activity

Set up your project and agent

Next, you'll need to set up the Actions project and the Dialogflow agent.

Do the following:

Open the Actions Console.
Click on New project. Instead of creating a new project, click on the Project name field and then select the existing project that's been created for you (it should start with "IOCodelabs 19" followed by additional digits). Click Import Project.
Rather than pick a category, scroll down to the More options section and click the Conversational card.
Click Build your Action to expand the options and select Add Action(s).
Click Add your first Action.
Select at least one language for your Action, followed by Update. (Select English.)
On the Custom intent dialog, click Build to launch the Dialogflow console.
Click Create.
Click the gear icon in the left navigation bar.
Click Export and Import.
Click Restore From Zip.

Figure 1: Dialogflow Console (Restoring the base Dialogflow agent for this codelab.)

Upload the codelab-level-one.zip file.
Type "RESTORE" and click the Restore button.
Click Done.

Deploy your fulfillment

Now that your Actions project and Dialogflow agent are ready, do the following to deploy your local index.js file using the command-line interface.

In a terminal, navigate to the /level2/functions directory of your base files clone.
Using the Actions project ID that you set, run the following command:

firebase use <PROJECT_ID>

Figure 2: Actions Console (Opening the Project settings page.)

Run the following command in the terminal to install dependencies:

npm install

Run the following command in the terminal to deploy your webhook to Firebase.

firebase deploy --project <PROJECT_ID>

After a few minutes, you should see a message that says, "Deploy complete," indicating that you deployed your webhook to Firebase.

Retrieve the deployment URL

You need to provide Dialogflow with the URL to the Cloud Function. To retrieve the URL, follow these steps:

Open the Firebase Console.
Select your Actions project from the list of options.
Navigate to Develop > Functions in the left navigation bar. If you're prompted to "Choose data sharing settings," then you click Do this later.
Under the Dashboard tab, you should see an entry for "dialogflowFirebaseFulfillment" with a URL under Trigger. Copy the URL.

Set the URL in Dialogflow

Now you need to update your Dialogflow agent to use your webhook for fulfillment. To do so, follow these steps:

Open the Dialogflow console. (You can close the Firebase console if you'd like.)
Navigate to Fulfillment in the left navigation bar.
Enable Webhook.
Paste the URL that you copied from the Firebase dashboard if it doesn't already appear.
Click Save.

Figure 4: Dialogflow Console (Setting the webhook URL.)

Verify that your project is correctly set up

At this point, users can start a conversation by explicitly invoking your Action. Once users are mid-conversation, they can trigger the ‘favorite color' custom intent by providing a color. Dialogflow will parse the user's input to extract the information your fulfillment needs—namely, the color—and send it to your fulfillment. Your fulfillment then autogenerates a lucky number to send back to the user.

To test out your Action in the Actions simulator, do the following:

In the Dialogflow console left navigation bar, click on Integrations > Google Assistant.
Make sure Auto-preview changes is enabled, then click Test to update your Actions project.
The Actions simulator loads your Actions project. To test your Action, type "Talk to my test app" in the Input field and hit Enter.
You should see a response that says, "Welcome! What is your favorite color?"
Type "blue."
You should see a response that says, "Your lucky number is 4."

Your Actions project always has an invocation name, like "Google IO 18." When users say "Talk to Google IO 18," they trigger the Dialogflow welcome intent. Every Dialogflow agent has one welcome intent, which acts as an entry point for users to start conversations.

Most of the time, users would rather jump to the specific task they want to accomplish than start at the beginning of the conversation every time. You can provide explicit deep links and implicit invocations as shortcuts into the conversation to help users get things done more efficiently.

Adding deep links and implicit invocations to your Actions is a simple, single-step process using the Assistant integration page in the Dialogflow console.

Add intent for deep linking and implicit invocation

In your Actions project, you should have defined a custom Dialogflow intent called "favorite color" in an agent. The agent parses your training phrases—such as "I love yellow" and "Purple is my favorite"—extracts the color parameter from each phrase, and makes it available to your fulfillment.

For this codelab, you're going to add the ‘favorite color' intent as an implicit invocation, meaning that users can invoke that intent and skip the welcome intent. Doing so also enables users to explicitly invoke the "favorite color" intent as a deep link (for example, "Hey Google, talk to my test app about blue"). The training phrases and parameters you defined for the "favorite color" intent enable Dialogflow to extract the color parameter when users invoke that deep link.

To add your intent for deep linking and implicit invocation, do the following:

In the Dialogflow console left navigation bar, click on Integrations.
In the Google Assistant card, click Integration Settings.
Under Discovery > Implicit Invocations, click on Add intent followed by favorite color.

Google Assistant will now listen for users to provide a color in their invocation and extract the color parameter for your fulfillment.

Figure 1: Dialogflow Console (Creating the implicit invocation in the Assistant integration dialog).

Test your deep link

To test out your deep link in the Actions simulator:

Click Test to update your Actions project.
Type "Talk to my test app about blue" in the Input field and hit Enter.

Figure 2: Actions Console simulator (Testing your deep link.)

Define a custom fallback intent

It's good practice to create a custom fallback intent to handle invocation phrases that don't provide the parameters you are looking for. For example, instead of saying a color, the user might say something unexpected like, "Talk to my test app about bananas." The term "bananas" would not fit into any of your Dialogflow intents, so you need to build a catch-all intent.

Given that Assistant now listens for any phrases that match the "favorite color" intent, you should provide a custom fallback intent specific for catching anything else.

To set up your custom fallback intent, do the following:

In the Dialogflow console, click on Intents in the left navigation bar, then click Create Intent.
Name your intent "Unrecognized Deep Link" or something equivalent. The intent name won't be referenced in your webhook, so you can call it whatever you like.

Under Contexts, click Add input context and type "google_assistant_welcome." By specifying that the intent uses the "google_assistant_welcome" input context, it can only be invoked at the start of the conversation. After you enter your input context, "google_assistant_welcome" appears as an output context as well. Click the x to remove that output context.

Figure 3: Dialogflow Console (Creating the ‘Unrecognized Deep Link’ intent).

Under Training phrases, add "banana" (or any other noun) as a user expression.
Use the @sys.any entity to tell Dialogflow to generalize the expression to any grammar (not just "banana"). Double click on "banana" and filter for or select @sys.any.
A warning message will pop up to avoid using the @sys.any entity, which you can safely ignore for now. Click OK. (Generally, it's not advisable to use the @sys.any entity, which can overpower any other intent's speech biasing, but this is a special case where you ensure that it'll only be triggered at invocation time when other intents have not been matched.)

Figure 3: Dialogflow Console (Adding training phrases to the ‘Unrecognized Deep Link’ intent).

Under Responses, add "Sorry, I am not sure about $any. What's your favorite color?" as a Text response.

Figure 4: Dialogflow Console (Adding a text response to the ‘Unrecognized Deep Link’ intent).

Click Save.

Test your custom fallback intent

To test your custom fallback intent in the Actions simulator, type "Talk to my test app about banana" in the Input field and hit Enter.

Figure 5: Actions Console simulator (Testing the fallback intent).

You can make your Actions more engaging and interactive by using personalized information from the user. To request access to user information, your webhook can use helper intents to obtain values with which to personalize your responses.

Get user information using permission helper intent

You can use the actions_intent_PERMISSION helper intent to obtain the user's display name with permission. To use the permission helper intent, do the following:

In the Dialogflow console, navigate to Intents.
Select the Default Welcome Intent.
Under Fulfillment, turn on Enable webhook call for this intent. Note that the response from the webhook will override any response you typed into Text responses above.
Click Save.
Navigate back to Intents.
Click Create Intent.
Name your intent actions_intent_PERMISSION.

Under Events, click Add event and type actions_intent_PERMISSION.
Under Fulfillment, turn on Enable webhook call for this intent.
Click Save.
In the terminal, navigate to the /level2/functions folder and open the index.js file in any text editor on your local machine.
Replace this code:

const {dialogflow} = require('actions-on-google');

with this code:

index.js

// Import the Dialogflow module and response creation dependencies from the 
// Actions on Google client library.
const {
  dialogflow,
  Permission,
  Suggestions,
} = require('actions-on-google');

Notice that you're importing the Suggestions dependency, which lets your webhook include suggestion chips in your responses.

Append the following code to the end of the file, but before exports.dialogflowFirebaseFulfillment = functions.https.onRequest(app);

index.js

// Handle the Dialogflow intent named 'Default Welcome Intent'.
app.intent('Default Welcome Intent', (conv) => {
  conv.ask(new Permission({
    context: 'Hi there, to get to know you better',
    permissions: 'NAME'
  }));
});

Save your file.

Customize responses with user information

Next, you need to update your webhook to handle the response. Use the user's information in your response if they granted permission and gracefully move the conversation forward regardless of whether permission was granted.

To respond to the user, do the following:

Add the following code to index.js:

index.js

// Handle the Dialogflow intent named 'actions_intent_PERMISSION'. If user
// agreed to PERMISSION prompt, then boolean value 'permissionGranted' is true.
app.intent('actions_intent_PERMISSION', (conv, params, permissionGranted) => {
  if (!permissionGranted) {
    conv.ask(`Ok, no worries. What's your favorite color?`);
    conv.ask(new Suggestions('Blue', 'Red', 'Green'));
  } else {
    conv.data.userName = conv.user.name.display;
    conv.ask(`Thanks, ${conv.data.userName}. What's your favorite color?`);
    conv.ask(new Suggestions('Blue', 'Red', 'Green'));
  }
});

You register a callback function to handle the actions_intent_PERMISSION intent you created earlier. In the callback, you first check whether the user granted permission to know their display name. The client library passes the argument to the callback function as the third parameter, here called permissionGranted.

The conv.user.name.display value represents the user's display name sent to your webhook as part of the HTTP request body. If the user grants permission, you store the value of conv.user.name.display in a property called userName of the conv.data object.

To provide additional hints to the user on how to continue the conversation, you call the Suggestions() function to create suggestion chips that recommend some example colors. If the user is on a device with a screen, he or she can provide input by tapping on a chip rather than by saying or typing a response.

The conv.data object is a data structure provided by the client library for in-dialog storage. You can set and manipulate the properties on that object throughout the duration of the conversation for the user.

Replace this code:

// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'.
app.intent('favorite color', (conv, {color}) => {
    const luckyNumber = color.length;
    // Respond with the user's lucky number and end the conversation.
    conv.close('Your lucky number is ' + luckyNumber);
});

with this:

index.js

// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'
app.intent('favorite color', (conv, {color}) => {
  const luckyNumber = color.length;
  if (conv.data.userName) {
    conv.close(`${conv.data.userName}, your lucky number is ${luckyNumber}.`);
  } else {
    conv.close(`Your lucky number is ${luckyNumber}.`);
  }
});

Here you modify the callback function for the ‘favorite color' intent to use the userName property to address the user by name. If the conv.data object doesn't have a property called userName (that is, the user previously denied permission to know their name, so the property was never set), then your webhook still responds, but without the user's name.

Save your file .
In the terminal, run the following command to deploy your webhook to Firebase:

firebase deploy --project <PROJECT_ID>

Test your code

To test your Action in the Actions simulator, do the following:

Type "Talk to my test app" in the Input field and hit enter.
Type "yes." You should see a prompt followed by suggestion chips displayed under Suggested input.
Type "blue."

Figure 1: Actions Console simulator (Testing custom responses.)

You can embed SSML in your response strings to alter the sound of your spoken responses or even embed sound effects and other audio clips.

The following shows an example of SSML markup:

<speak>
  Mandy, your lucky number is 5.
  <audio src="https://actions.google.com/sounds/v1/cartoon/clang_and_wobble.ogg"></audio>
</speak>

Use SSML to add sound effects

You'll use a sound clip from the Actions on Google sound library.

To add a sound effect to the "favorite color" response, do the following:

Open index.js in an editor.
Replace this code:

// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'
app.intent('favorite color', (conv, { color }) => {
  const luckyNumber = color.length;
  if (conv.data.userName) {
    conv.close(`${conv.data.userName}, your lucky number is ` + `${luckyNumber}.`);
  } else {
    conv.close(`Your lucky number is ` + `${luckyNumber}.`);
  }
});

with this code:

index.js

// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'
app.intent('favorite color', (conv, {color}) => {
 const luckyNumber = color.length;
 const audioSound = 'https://actions.google.com/sounds/v1/cartoon/clang_and_wobble.ogg';
 if (conv.data.userName) {
   // If we collected user name previously, address them by name and use SSML
   // to embed an audio snippet in the response.
   conv.close(`<speak>${conv.data.userName}, your lucky number is ` +
     `${luckyNumber}.<audio src="${audioSound}"></audio></speak>`);
 } else {
   conv.close(`<speak>Your lucky number is ${luckyNumber}.` +
     `<audio src="${audioSound}"></audio></speak>`);
 }
});

Here, you declare an audioSound variable containing the string URL for a statically hosted audio file on the web. You use the <speak> SSML tags around the strings for the user response, indicating to Assistant that your response should be parsed as SSML.

The <audio> tag embedded in the string indicates that you want Assistant to play some audio played at that point in the response. The src attribute of that tag indicates where the audio is hosted.

Save your file.
In the terminal, run the following command to deploy your webhook to Firebase.

firebase deploy --project <PROJECT_ID>

Test your code

To test your Action in the Actions simulator, do the following:

Type "Talk to my test app" in the Input field and hit enter.
Type "yes."
Type "blue."

If everything works correctly, then the user should hear this sound effect in the response.

To keep the conversation going, you can add follow-up intents that will trigger based on the user's response after a particular intent. To add follow-up intents to "favorite color," do the following:

In the Dialogflow console left navigation bar, click Intents.
Hover your cursor over favorite color, then click Add follow-up intent. Do this twice, once selecting yes and again selecting no.

Under favorite color - no, do the following:

Under Responses, add "Goodbye, see you next time!" as a Text response.
Turn on Set this intent as end of conversation.
Click Save.
Under favorite color - yes, do the following:

Under Responses, add "Which color, indigo taco, pink unicorn, or blue grey coffee?" as a Text response.
Click Save.

Figure 1: Add follow-up intents to the favorite color intent.

As you expand your conversational app, you can use custom entities to deepen and personalize the conversation.

Add a custom entity

So far, you've only been using built-in entities (@sys.color, @sys.any) to match user input. You're going to create a custom entity (also called a developer entity) in Dialogflow so that when a user provides one of a few fake colors, you can follow up with a custom response from your webhook.

To create a custom entity:

In the Dialogflow console left navigation bar, click on Entities.
Click Create entity and call it "fakeColor."
Add the following entries and synonyms, then click Save:

indigo taco / the first one
pink unicorn / the second one
blue grey coffee / the third one

Figure 1: Adding a fakeColor custom entity.

In the left navigation bar, click Intents.
Click Create intent and call it "favorite fake color."

Under Training phrases, type:

"indigo taco"
"tell me about pink unicorn"
"I want to know about blue grey coffee"

You should see the "fakeColor" parameter show up under Actions and parameters now that Dialogflow recognizes your custom entity.

Under Fulfillment, turn on Enable webhook call for this intent.
Click Save.

When a user selects one of the fake colors that you defined, your webhook will respond with basic cards that show each color.

To configure your webhook, do the following:

On your device, navigate to the /level2/functions folder and open index.js in an editor.
Replace this code:

// Import the Dialogflow module and response creation dependencies from the 
// Actions on Google client library.
const {
  dialogflow,
  Permission,
  Suggestions,
} = require('actions-on-google');

with this code:

index.js

// Import the Dialogflow module and response creation dependencies from the 
// Actions on Google client library.
const {
  dialogflow,
  Permission,
  Suggestions,
  BasicCard,
} = require('actions-on-google');

Replace this code:

// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'
app.intent('favorite color', (conv, {color}) => {
 const luckyNumber = color.length;
 const audioSound = 'https://actions.google.com/sounds/v1/cartoon/clang_and_wobble.ogg';
 if (conv.data.userName) {
   // If we collected user name previously, address them by name and use SSML
   // to embed an audio snippet in the response.
   conv.close(`<speak>${conv.data.userName}, your lucky number is ` +
     `${luckyNumber}<audio src="${audioSound}"></audio>.`);
 } else {
   conv.close(`<speak>Your lucky number is ${luckyNumber}` +
     `<audio src="${audioSound}"></audio>.`);
 }
});

with this code:

index.js

// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'.
app.intent('favorite color', (conv, {color}) => {
  const luckyNumber = color.length;
  const audioSound = 'https://actions.google.com/sounds/v1/cartoon/clang_and_wobble.ogg';
  if (conv.data.userName) {
    // If we collected user name previously, address them by name and use SSML
    // to embed an audio snippet in the response.
    conv.ask(`<speak>${conv.data.userName}, your lucky number is ` +
      `${luckyNumber}.<audio src="${audioSound}"></audio> ` +
      `Would you like to hear some fake colors?</speak>`);
    conv.ask(new Suggestions('Yes', 'No'));
  } else {
    conv.ask(`<speak>Your lucky number is ${luckyNumber}.` +
      `<audio src="${audioSound}"></audio> ` +
      `Would you like to hear some fake colors?</speak>`);
    conv.ask(new Suggestions('Yes', 'No'));
  }
});

Add the following code at the end of the file, but before this line: exports.dialogflowFirebaseFulfillment = functions.https.onRequest(app);

index.js

// Define a mapping of fake color strings to basic card objects.
const colorMap = {
  'indigo taco': {
    title: 'Indigo Taco',
    text: 'Indigo Taco is a subtle bluish tone.',
    image: {
      url: 'https://storage.googleapis.com/material-design/publish/material_v_12/assets/0BxFyKV4eeNjDN1JRbF9ZMHZsa1k/style-color-uiapplication-palette1.png',
      accessibilityText: 'Indigo Taco Color',
    },
    display: 'WHITE',
  },
  'pink unicorn': {
    title: 'Pink Unicorn',
    text: 'Pink Unicorn is an imaginative reddish hue.',
    image: {
      url: 'https://storage.googleapis.com/material-design/publish/material_v_12/assets/0BxFyKV4eeNjDbFVfTXpoaEE5Vzg/style-color-uiapplication-palette2.png',
      accessibilityText: 'Pink Unicorn Color',
    },
    display: 'WHITE',
  },
  'blue grey coffee': {
    title: 'Blue Grey Coffee',
    text: 'Calling out to rainy days, Blue Grey Coffee brings to mind your favorite coffee shop.',
    image: {
      url: 'https://storage.googleapis.com/material-design/publish/material_v_12/assets/0BxFyKV4eeNjDZUdpeURtaTUwLUk/style-color-colorsystem-gray-secondary-161116.png',
      accessibilityText: 'Blue Grey Coffee Color',
    },
    display: 'WHITE',
  },
};

// Handle the Dialogflow intent named 'favorite fake color'.
// The intent collects a parameter named 'fakeColor'.
app.intent('favorite fake color', (conv, {fakeColor}) => {
  // Present user with the corresponding basic card and end the conversation.
  conv.close(`Here's the color`, new BasicCard(colorMap[fakeColor]));
});

This new code performs two main tasks:

First, it sets up a mapping (colorMap) of color strings (e.g. "indigo taco," "pink unicorn," "blue grey coffee") to the content needed for BasicCard objects. BasicCard is a client library class for constructing visual responses corresponding to the basic card type.

In the constructor calls, then you pass configuration options relevant to each specific color, including:

A title with the color name
A text description providing more detail about the card's contents
A URL for an image which represents that color and its alt-text
A display configuration, which determines a background color (in this case, plain white).

Finally, you set a callback function for the ‘favorite fake color' intent, which uses the fakeColor option that the user selected to create a card corresponding to that fake color and present it to the user.

In the terminal, run the following command to deploy your webhook to Firebase:

firebase deploy --project <PROJECT_ID>

Test your code

To test out your Action in the Actions simulator, do the following:

Type "Talk to my test app" in the Input field and hit Enter.
Type "yes."
Type "blue."
Type "sure."
Type "tell me about the first one."

When you select a fake color, you should receive a response that includes a basic card, similar to the following screenshot:

Congratulations!

You now possess the intermediate skills necessary to build conversational user interfaces with Actions on Google.

What you covered

How to set up for local fulfillment development using the Node.js client library and Firebase command-line deployment.
How to create deep links for your Actions using the Assistant integration in Dialogflow.
How to use Actions on Google helper intents to personalize your responses.
How to add Dialogflow custom entities, SSML-based audio effects, and follow-up intents.

What's next

Explore the following resources for learning about Actions on Google:

Follow @ActionsOnGoogle on Twitter to stay tuned to the latest announcements and tweet with #AoGDevs to share what you build!

Feedback survey

Before you go, please fill out this form