Internet of Things (IoT) developers can build devices that can be remotely controlled from the Google Assistant through direct voice commands using the Smart Home integration. This occurs through a cloud-to-cloud integration between the Google Assistant and your server.
Smart Home apps rely on Home Graph, a database that stores and provides contextual data about the home and its devices. For example, Home Graph can store the concept of a living room that contains multiple types of devices (a thermostat, lamp, fan, and vacuum) from different manufacturers. This information is passed to the Google Assistant in order to execute user requests based on the appropriate context.
What you will build
In this codelab, you're going to create your own cloud integration and connect the Google Assistant to a smart washing machine.
What you'll learn
- How to deploy a Smart Home server
- How to connect a washer to the Google Assistant
- How to add toggles and modes
- How to use the Request Sync and Report State APIs
What you'll need
- Web browser
- Node.js version 8
Enabling Activity controls
Enable the following Activity controls in the Google Account you plan to use with the Assistant:
- Web & App Activity
- Device Information
- Voice & Audio Activity
Using the Actions on Google console
- Go to the Actions on Google Developer Console.
- Click on Add Project, enter a name for the project, and click CREATE PROJECT.
Select the Smart Home App
On the Overview screen in the Actions console, click Home control and then Smart home.
Install the Firebase Command Line Interface
The Firebase Command Line Interface (CLI) will allow you to serve your web apps locally and deploy your web app to Firebase hosting.
To install the CLI run the following npm command:
npm -g install firebase-tools
To verify that the CLI has been installed correctly open a console and run:
firebase --version
Make sure the Firebase version is above 3.3.0
Authorize the Firebase CLI by running:
firebase login
Get the sample
Click the following link to download the sample for this codelab on your development machine:
...or you can clone the GitHub repository from the command line:
$ git clone https://github.com/googlecodelabs/smarthome-washer.git
Unpack the downloaded zip file.
Connect to Firebase
Make sure you are in the washer-start directory then set up the Firebase CLI to use your Firebase Project:
cd washer-start firebase use --add
Then select your Project ID and follow the instructions.
Deploy to Firebase
Navigate to the functions folder inside of washer-start and run npm install.
npm --prefix functions/ install
Now that you have installed the dependencies and configured your project, you are ready to run the app for the first time.
firebase deploy
This is the console output you should see:
... ✔ Deploy complete! Project Console: https://console.firebase.google.com/project/<project-id>/overview Hosting URL: https://<project-id>.firebaseapp.com
The web app should now be served from your Hosting URL - which is of the form https://<project-id>.firebaseapp.com.
This command will also deploy several cloud functions for Firebase. You will now use these URLs in the Actions on Google console.
Configure your project in the Actions on Google console
Select Actions on the left-hand side and click ADD YOUR FIRST ACTION. Enter a URL for the backend server that will provide fulfillment for the Smart Home intents and then click SAVE.
https://us-central1-<project-id>.cloudfunctions.net/smarthome
Then click DONE.
Select the Advanced Settings > Account Linking option in the sidebar. Select No for account creation by voice, and make sure the linking type is OAuth and Authorization code.
Enter the following client information:
Client ID |
|
Client secret |
|
Authorization URL |
|
Token URL |
|
For Testing Instructions, put "No authentication needed".
Then select the SAVE button at the bottom to save this information and return to the Overview page.
Open the Test > Simulator page from the sidebar and click the START TESTING button to finish your project setup.
Link to the Google Assistant
To finish setup, you will need to link to your Google account to your Smart Home cloud, which are the functions that you have deployed.
- On your phone, open the Google Assistant settings. Note that you should be logged in as the same account as in the console.
- Navigate to Google Assistant > Settings > Home Control (under Services).
- Select the plus icon in the bottom right corner
- You should see your test app with the [test] prefix
- Select that item. The Google Assistant will then authenticate with your server and send a
SYNCrequest, asking your server to provide a list of devices for the user. Your washer will show up.
Now that your Smart Home account is connected to your Google Assistant, you can start adding devices and sending data. There are three intents that your server will need to handle in the smarthome function.
- A SYNC intent occurs when the Google Assistant wants to know what devices a user has. You should respond with a JSON payload of all the user's devices. This is sent to your server when a user links their account.
- A QUERY intent occurs when the Google Assistant wants to know the current state or status of a device. You should respond with a JSON payload with the state of each requested device.
- An EXECUTE intent occurs when the Google Assistant wants to control a device on a user's behalf. You should respond with a JSON payload with the execution status of each requested device.
In order to dynamically get a list of devices, you can use Cloud Functions for Firebase. It will handle these three intents with responses that you can define. In order to host the state of your washer, you can use the Firebase Realtime Database.
Update SYNC response
Open functions/index.js. This contains the code to respond to requests from the Google Assistant. First, we will need to handle a SYNC intent by responding with our washer. You will see right now that it will always give the same basic response.
Implement the full JSON in the array to represent your washer.
app.onSync(body => {
return {
requestId: body.requestId,
payload: {
agentUserId: '123',
devices: [{
id: 'washer',
type: 'action.devices.types.WASHER',
traits: [
'action.devices.traits.OnOff',
'action.devices.traits.StartStop',
'action.devices.traits.RunCycle'
],
name: {
defaultNames: ['My Washer'],
name: 'Washer',
nicknames: ['Washer']
},
deviceInfo: {
manufacturer: 'Acme Co',
model: 'acme-washer',
hwVersion: '1.0',
swVersion: '1.0.1'
},
attributes: {
pausable: true
}
}]
}
};
});Now deploy the updated function.
firebase deploy
Re-link your integration to the Google Assistant
In order to test your new SYNC response, you will need to unlink your integration and link to it again. Later, you will add the Request Sync feature so that you will be able to cause a SYNC request without having to unlink your account.
- Open the Google Assistant settings.
- Navigate to Google Assistant > Settings > Home Control (under Services).
- Select the plus icon in the bottom right corner
- You will see your test app with the [test] prefix under the Linked services header
- Select that item and select Unlink account.
- After you unlink your account, the app will appear again under the Add new header.
- Select that item. The Google Assistant will then authenticate with your server and send a
SYNCrequest, asking your server to provide a list of devices for the user. You should see your washer appear. While there is no visible change, the device will now have the additional traits.
Initializing the database
Now you will have to handle the two other intents, to let the user know the current state of their washer, or control it. Let's add the EXECUTE intent.
Go to the Firebase console and select your project. Then, open the Database page and CREATE DATABASE with using the Realtime Database. When asked, configure the database for Test Mode. The database is organized to store the states for each device. The hierarchy is shown below.
To set this up, you can open the hosted website for your project <project-id>.firebaseapp.com and click on the UPDATE button. It will immediately set the values to their defaults.
Handling EXECUTE intent
In functions/index.js edit the EXECUTE handler as below:
app.onExecute((body) => {
const {requestId} = body;
const payload = {
commands: [{
ids: [],
status: 'SUCCESS',
states: {
online: true,
},
}],
};
for (const input of body.inputs) {
for (const command of input.payload.commands) {
for (const device of command.devices) {
const deviceId = device.id;
payload.commands[0].ids.push(deviceId);
for (const execution of command.execution) {
const execCommand = execution.command;
const {params} = execution;
switch (execCommand) {
case 'action.devices.commands.OnOff':
firebaseRef.child(deviceId).child('OnOff').update({
on: params.on,
});
payload.commands[0].states.on = params.on;
break;
case 'action.devices.commands.StartStop':
firebaseRef.child(deviceId).child('StartStop').update({
isRunning: params.start,
});
payload.commands[0].states.isRunning = params.start;
break;
case 'action.devices.commands.PauseUnpause':
firebaseRef.child(deviceId).child('StartStop').update({
isPaused: params.pause,
});
payload.commands[0].states.isPaused = params.pause;
break;
}
}
}
}
}
return {
requestId: requestId,
payload: payload,
};
});
In the implementation above, you iterate through each command, update the value in Firebase, then respond with a payload stating the current state of your device.
Now deploy the updated function.
firebase deploy
Now you can see the value change when you give a voice command. You can use your phone to give these commands.
"Turn on my washer"
"Pause my washer"
"Stop my washer"
In order to support interrogative questions, like "Is my washer on?" you will need to implement a QUERY intent. You do this in the next step.
Handling QUERY intent
A QUERY intent will include a set of devices. For each device, you should respond with its current state.
In functions/index.js edit the QUERY handler as below:
app.onQuery(async (body) => {
const {requestId} = body;
const payload = {
devices: {},
};
const queryPromises = [];
for (const input of body.inputs) {
for (const device of input.payload.devices) {
const deviceId = device.id;
queryPromises.push(queryDevice(deviceId)
.then((data) => {
// Add response to device payload
payload.devices[deviceId] = data;
}
));
}
}
// Wait for all promises to resolve
await Promise.all(queryPromises)
return {
requestId: requestId,
payload: payload,
};
});Now deploy the updated function.
firebase deploy
You can see the current state of your washer by asking questions.
"Is my washer on?"
"Is my washer running?"
Now that you have implemented all three intents, you can add additional features to your washer.
Modes and toggles allow you to control a specific component of your device with a name defined by the developer. In this codelab, your washer has a mode to define the size of the laundry load: small or large.
To start, uncomment the section of public/index.html to show the modes:
<div id='demo-washer-modes-main'>
<label>Washer Mode</label>
<br>
<label id='demo-washer-modes-small' class='mdl-radio mdl-js-radio mdl-js-ripple-effect' for='demo-washer-modes-small-in'>
<input checked class='mdl-radio__button' id='demo-washer-modes-small-in' name='load' type='radio'
value='on'>
<span class='mdl-radio__label'>Small</span>
</label>
<label id='demo-washer-modes-large' class='mdl-radio mdl-js-radio mdl-js-ripple-effect' for='demo-washer-modes-large-in'>
<input class='mdl-radio__button' id='demo-washer-modes-large-in' name='load' type='radio' value='off'>
<span class='mdl-radio__label'>Large</span>
</label>
<br>
<br>
</div>
When you press the UPDATE button, the mode will be stored in Firebase.
In your SYNC response, you will need to add information about this new mode. This will appear in the attributes object as shown in the response below.
app.onSync(body => {
return {
requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
payload: {
agentUserId: '123',
devices: [{
id: 'washer',
type: 'action.devices.types.WASHER',
traits: [
'action.devices.traits.OnOff',
'action.devices.traits.StartStop',
'action.devices.traits.RunCycle',
'action.devices.traits.Modes',
],
name: {
defaultNames: ['My Washer'],
name: 'Washer',
nicknames: ['Washer']
},
deviceInfo: {
manufacturer: 'Acme Co',
model: 'acme-washer',
hwVersion: '1.0',
swVersion: '1.0.1'
},
attributes: {
pausable: true,
availableModes: [{
name: 'load',
name_values: [{
name_synonym: ['load'],
lang: 'en'
}],
settings: [{
setting_name: 'small',
setting_values: [{
setting_synonym: ['small'],
lang: 'en'
}]
}, {
setting_name: 'large',
setting_values: [{
setting_synonym: ['large'],
lang: 'en'
}]
}],
ordered: true
}]
}
}]
}
};
});
In your EXECUTE intent, you will need to add the action.devices.commands.SetModes command, as shown below.
app.onExecute((body) => {
const {requestId} = body;
const payload = {
commands: [{
ids: [],
status: 'SUCCESS',
states: {
online: true,
},
}],
};
for (const input of body.inputs) {
for (const command of input.payload.commands) {
for (const device of command.devices) {
const deviceId = device.id;
payload.commands[0].ids.push(deviceId);
for (const execution of command.execution) {
const execCommand = execution.command;
const {params} = execution;
switch (execCommand) {
case 'action.devices.commands.OnOff':
firebaseRef.child(deviceId).child('OnOff').update({
on: params.on,
});
payload.commands[0].states.on = params.on;
break;
case 'action.devices.commands.StartStop':
firebaseRef.child(deviceId).child('StartStop').update({
isRunning: params.start,
});
payload.commands[0].states.isRunning = params.start;
break;
case 'action.devices.commands.PauseUnpause':
firebaseRef.child(deviceId).child('StartStop').update({
isPaused: params.pause,
});
payload.commands[0].states.isPaused = params.pause;
break;
case 'action.devices.commands.SetModes':
firebaseRef.child(deviceId).child('Modes').update({
load: params.updateModeSettings.load,
});
break;
}
}
}
}
}
return {
requestId: requestId,
payload: payload,
};
});
Now deploy the updated function.
firebase deploy
You can give a command to set the mode of the washer.
"Set the washer to a large load"
Finally, you will need to update your QUERY response to respond to questions about the washer's current state. Add the updated changes to the queryFirebase and queryDevice functions to obtain the mode.
const queryFirebase = async (deviceId) => {
const snapshot = await firebaseRef.child(deviceId).once('value');
const snapshotVal = snapshot.val();
return {
on: snapshotVal.OnOff.on,
isPaused: snapshotVal.StartStop.isPaused,
isRunning: snapshotVal.StartStop.isRunning,
load: snapshotVal.Modes.load,
};
}
const queryDevice = async (deviceId) => {
const data = await queryFirebase(deviceId);
return {
on: data.on,
isPaused: data.isPaused,
isRunning: data.isRunning,
currentRunCycle: [{
currentCycle: 'rinse',
nextCycle: 'spin',
lang: 'en',
}],
currentTotalRemainingTime: 1212,
currentCycleRemainingTime: 301,
currentModeSettings: {
load: data.load,
}
};
};
You will be able to ask questions about your washer such as:
"Is my washer small load?"
Toggles represent named aspects of a device that have a true/false state, such as whether the washer is in Turbo mode.
To start, uncomment the section of public/index.html to show the modes. When you press the UPDATE button, the mode will be stored in Firebase.
<div id='demo-washer-toggles-main'>
<label id='demo-washer-toggles' class='mdl-switch mdl-js-switch mdl-js-ripple-effect' for='demo-washer-toggles-in'>
<input type='checkbox' id='demo-washer-toggles-in' class='mdl-switch__input'>
<span class='mdl-switch__label'>Is in Turbo</span>
</label>
</div>In your SYNC response, you will need to add information about this new mode. This will appear in the attributes object as shown in the response below.
app.onSync(body => {
return {
requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
payload: {
agentUserId: '123',
devices: [{
id: 'washer',
type: 'action.devices.types.WASHER',
traits: [
'action.devices.traits.OnOff',
'action.devices.traits.StartStop',
'action.devices.traits.RunCycle',
'action.devices.traits.Modes',
'action.devices.traits.Toggles',
],
name: {
defaultNames: ['My Washer'],
name: 'Washer',
nicknames: ['Washer']
},
deviceInfo: {
manufacturer: 'Acme Co',
model: 'acme-washer',
hwVersion: '1.0',
swVersion: '1.0.1'
},
attributes: {
pausable: true,
availableModes: [{
name: 'load',
name_values: [{
name_synonym: ['load'],
lang: 'en'
}],
settings: [{
setting_name: 'small',
setting_values: [{
setting_synonym: ['small'],
lang: 'en'
}]
}, {
setting_name: 'large',
setting_values: [{
setting_synonym: ['large'],
lang: 'en'
}]
}],
ordered: true
}],
availableToggles: [{
name: 'Turbo',
name_values: [{
name_synonym: ['turbo'],
lang: 'en'
}]
}]
}
}]
}
};
});In your EXECUTE intent, you will need to add the action.devices.commands.SetToggles command, as shown below.
app.onExecute((body) => {
const {requestId} = body;
const payload = {
commands: [{
ids: [],
status: 'SUCCESS',
states: {
online: true,
},
}],
};
for (const input of body.inputs) {
for (const command of input.payload.commands) {
for (const device of command.devices) {
const deviceId = device.id;
payload.commands[0].ids.push(deviceId);
for (const execution of command.execution) {
const execCommand = execution.command;
const {params} = execution;
switch (execCommand) {
case 'action.devices.commands.OnOff':
firebaseRef.child(deviceId).child('OnOff').update({
on: params.on,
});
payload.commands[0].states.on = params.on;
break;
case 'action.devices.commands.StartStop':
firebaseRef.child(deviceId).child('StartStop').update({
isRunning: params.start,
});
payload.commands[0].states.isRunning = params.start;
break;
case 'action.devices.commands.PauseUnpause':
firebaseRef.child(deviceId).child('StartStop').update({
isPaused: params.pause,
});
payload.commands[0].states.isPaused = params.pause;
break;
case 'action.devices.commands.SetModes':
firebaseRef.child(deviceId).child('Modes').update({
load: params.updateModeSettings.load,
});
break;
case 'action.devices.commands.SetToggles':
firebaseRef.child(deviceId).child('Toggles').update({
Turbo: params.updateToggleSettings.Turbo,
});
break;
}
}
}
}
}
return {
requestId: requestId,
payload: payload,
};
});
Now deploy the updated function.
firebase deploy
You can give a command to set the mode of the washer.
"Turn on turbo for the washer"
Finally, you will need to update your QUERY response to respond to questions about the washer's turbo mode. Add the updated changes to the queryFirebase and queryDevice functions to obtain the toggle state.
const queryFirebase = async (deviceId) => {
const snapshot = await firebaseRef.child(deviceId).once('value');
const snapshotVal = snapshot.val();
return {
on: snapshotVal.OnOff.on,
isPaused: snapshotVal.StartStop.isPaused,
isRunning: snapshotVal.StartStop.isRunning,
load: snapshotVal.Modes.load,
turbo: snapshotVal.Toggles.Turbo,
};
}
const queryDevice = async (deviceId) => {
const data = queryFirebase(deviceId);
return {
on: data.on,
isPaused: data.isPaused,
isRunning: data.isRunning,
currentRunCycle: [{
currentCycle: 'rinse',
nextCycle: 'spin',
lang: 'en',
}],
currentTotalRemainingTime: 1212,
currentCycleRemainingTime: 301,
currentModeSettings: {
load: data.load,
},
currentToggleSettings: {
Turbo: data.turbo,
},
};
};Now deploy the updated function.
firebase deploy
You will be able to ask questions about your washer such as
"Is my washer in turbo mode?"
Now you have completely implemented your washer. You can control and get the current state of this washer. However, each time you added a new feature you had to unlink and relink to your integration.
A SYNC request has only reached your server during this linking step. By adding the Request Sync API, you will be able to trigger a SYNC request and allow a user's list of devices to be updated without unlinking and relinking their account.
The Request Sync API takes a user id as a parameter. This should be in the user id in your server that represents your user. In this codelab, the value is hardcoded to "123".
- In the Cloud Platform Console, go to the Projects page. Select the project that matches your Smart Home project id.
- Enable the HomeGraph API.
- Generate an API key. From the left navbar, select Credentials under APIs & Services. Click the Create Credentials button and select API key.
- Add the API key to the Smart Home constructor in functions/index.js
const app = smarthome({
debug: true,
key: '<api-key>'
});In the frontend web UI, there is a refresh icon in the header. You can add a click listener to that in order to make your request sync call.
In main.js:
this.requestSync = document.getElementById('request-sync');
this.requestSync.addEventListener('click', () => {
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
console.log('Request SYNC success!');
}
};
xhttp.open('POST', 'https://us-central1-<project-id>.cloudfunctions.net/requestsync', true);
xhttp.send();
});
Make sure to replace https://us-central1-<project-id>.cloudfunctions.net/requestsync with the equivalent function for your project.
Now when you click that icon, you will see a SYNC request in your server logs. Each time you update the device, add new devices, or remove this device, you will see it appear in your list of devices when you send a new request sync call.
The response will also appear in your logs, which you can view through Firebase.
With the Report State API, a Smart Home integration can proactively send the device's state to the home graph. This allows user queries to be completed faster and enables them to use rich UIs on their phones or smart displays in order to know the status of all their devices.
When you click the UPDATE button to change the washer's state, you can add an additional step which will report this to the home graph.
Create JWT
Before you can write our function, you will need to make sure our data is sent securely. This is done through JWT (JSON web tokens). In this codelab you will be using the googleapis npm dependency which will help facilitate the creation of this token.
Go to the Google Cloud Console for your project. Select Credentials in the APIs & Services section. Click the Create Credentials button and select Service account key. For Role, select Project -> Editor.
After creating your service account, you will download a JSON file. Save this file under the functions folder in your project with the name key.json.
Add API call
This can be done using a Firebase database trigger. When a certain write event happens, it can automatically report the state.
In functions/index.js:
const app = smarthome({
debug: true,
key: '<api-key>',
jwt: require('./key.json'),
});exports.reportstate = functions.database.ref('{deviceId}').onWrite(async (change, context) => {
console.info('Firebase write event triggered this cloud function');
const snapshot = change.after.val();
const postData = {
requestId: 'ff36a3cc', /* Any unique ID */
agentUserId: '123', /* Hardcoded user ID */
payload: {
devices: {
states: {
/* Report the current state of our washer */
[context.params.deviceId]: {
on: snapshot.OnOff.on,
isPaused: snapshot.StartStop.isPaused,
isRunning: snapshot.StartStop.isRunning,
},
},
},
},
};
const data = await app.reportState(postData);
console.log('Report state came back');
console.info(data);
});