Google Assistant Smart Home Part 2: API Implementation

Google Assistant Smart Home Part 2: API Implementation

This is part 2 of a two-part series on building voice-controlled devices with Google Assistant. If you haven’t read part 1, please do so here! Specific things I’ll be covering here include:

  1. Incoming calls: SYNC, QUERY, EXECUTE, and DISCONNECT
  2. Outgoing calls: Report State and Request Sync
  3. Error types and concepts

Quick Primer:

From part 1, we learned that your IoT device needs to function independently of Google Assistant with its own app and cloud service. The Assistant communicates with your cloud service and augments your service with voice, allowing users to control your devices using a more natural interface.

This separation of developer responsibility is shown below:

In part 2, we need to understand the sequence of API calls, why they exist, and how they are structured. In order to fully understand what’s happening in each intent, we need to understand attributesstates, and commands.

Attributes, states, and commands all relate to a device and are dependent on the set of traits you implement. You can have many traits per device, but only one type (like the example below). Attributes, states, and commands are used at different stages of the lifecycle of a device on Google Assistant. For example, the SYNC intent is used at the very beginning of the device lifecycle.

Let’s see how a light bulb that can turn on/off and change color would look using these concepts:

Take a look at the documentation for the LIGHT device type, OnOff attribute, and ColorSetting attribute. From the example above, attributes like colorTemperatureRange are used to configure the ColorSetting trait and define what colors are supported by the light bulb. The state color is used for QUERY and EXECUTE intents to set and retrieve the current color of the device.

I’ll be using this same example device when explaining each intent below. As I go through these API calls, you can try out and play with all the JSON using Google’s JSON validator for Actions on Google: https://developers.google.com/actions/smarthome/tools/validator/.

Smart Home Lifecycle

The lifecycle of a smart home device on Google Assistant consists of four intents: SYNC, QUERY, EXECUTE, and DISCONNECT.

Let’s take a look at an example where a user purchases a light bulb from Acme Co. The user would first setup the light bulb on Acme’s cloud service and can control the device through Acme’s app — Google is not involved at this stage.

Next, the user downloads and configures the device in the Google Home App. When configuring the device in the Home App, Acme will receive a SYNC intent.

After setting up the device in the Home App, the user can start using the Assistant to control their Acme light. Acme’s cloud service receives EXECUTE and QUERY intents each time the user converses with the Assistant to control their light bulb.

Lastly, the user decides to replace their Acme light bulb with a new light bulb from SuperDuperLights; the user removes their Acme light from their Home App and Acme receives a DISCONNECT intent.

Let’s dive into each intent!

1. SYNC

Google sends a SYNC intent to get the list of devices that the user can control through your service. When the user first creates and configures their home in the Home App, this is the very first call to your service and is made during the account linking process, so that the user can assign devices to specific rooms. You see the list of devices in the final stage of account linking, as shown in the diagram below.

Let’s examine example SYNC payloads and take note of some important fields. First, an example SYNC request from Google:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "inputs": [{
    "intent": "action.devices.SYNC"
  }]
}

Next, an example response for the request above. The user has one smart light bulb from Acme Co:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "agentUserId": "1836.15267389",
    "devices": [{
      "id": "123",
      "type": "action.devices.types.LIGHT",
      "traits": [
        "action.devices.traits.OnOff"         "action.devices.traits.ColorSetting"],
      "name": {
        "defaultNames": ["Acme Co. bulb A19 On/Off only"],
        "name": "lamp1",
        "nicknames": ["reading lamp"]
      },
      "attributes": {
          "colorModel": "rgb",
          "colorTemperatureRange": {
            "temperatureMinK": 2000,
            "temperatureMaxK": 9000
          },
          "commandOnlyColorSetting": false
        },
      "willReportState": true,
      "deviceInfo": {
        "manufacturer": "Acme Co",
        "model": "hg11",
        "hwVersion": "1.2",
        "swVersion": "5.4"
      }
    }]
  }
}

Key fields from these payloads:

  • requestId: this is a unique identifier generated by Google for this request and response transaction. The requestId of the response should be identical to the requestId of the request. This is useful when debugging because you can easily compare a response to its request in server logs by matching the requestId. If the requestId is not identical it will make debugging more difficult.
  • agentUserId: this is a unique identifier generated and stored by Acme Co’s smart home service for each user of Acme’s service. This agentUserId is important when updating the synced devices with Google in the future because it’s used to identify which user to sync for a Request Sync API call.
  • deviceId: this is an ID generated by Acme Co’s cloud service and given to Google. Google will use this ID when requesting the state or executing commands for that specific device.
  • willReportState: this tells Google if your service will proactively push state updates of the devices by using the Report State API. Proactively pushing the state of devices helps reduce latency, which improves user experience. You want to always set this to true and implement the Report State API.

I only went over a few elements in the JSON structure. To see a description of every element, check out the reference for SYNC.

Not all attributes for a given trait are required, so be sure to refer to the documentation to determine what you need to provide.

From Part 1 we learned that a device type can have any set of traits. Meaning, a vacuum cleaner with a device type of action.devices.types.VACUUM could change colors with a trait of action.devices.traits.ColorSetting or even have an integrated camera and allow people to live-stream their vacuum with a trait of action.devices.traits.CameraStream.

2. QUERY

The Query intent is sent to your service in order to get the most up-to-date state of a particular device. This can be called when a user asks a question, rather than giving a command, “Hey Google, is my refrigerator running?”

Notice that in diagram 3, I don’t include the actual IoT device when querying state — Google Assistant refers to your cloud service as the source of truth for any devices that are controlled through your cloud service.

Let’s look at the request payload from Google for this intent:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "inputs": [{
    "intent": 'action.devices.QUERY',
    "payload": {
      "devices": [{
        "id": "123"
      }]
    }
  }]
}

Important elements of this request:

  1. requestId: we see this again, same as we saw it in the SYNC request. We return the exact same ID back in our response to this request.
  2. intent: notice QUERY — this is how we match our business logic with what the user is wanting to do.
  3. device array: this will have a set of device objects, each with their ID and any custom data associated with it if custom data was given to Google from the SYNC call.

I only went over a few elements in the JSON structure. To see a description of every element, check out the reference for QUERY.

How do we structure the response? Like this:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "devices": {
      "123": {
        "on": true,
        "online": true
      }
    }
  }

Important elements of this response:

  • The requestId in the response matches the requestId from the request.
  • You’ll notice the devices element is not an array like the request. Instead it is an object with each key being the id of one of the devices queried.
  • The object associated with each device id key is the set of states for that device, or an error status if your service could not retrieve the state of the device.

3. EXECUTE

The Execute intent is the call that actually tells your service to change the state of your IoT device or otherwise run a function on your device in some way.

Let’s dive into the request payload:

{
    "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
    "inputs": [{
      "intent": "action.devices.EXECUTE",
      "payload": {
        "commands": [{
          "devices": [{
            "id": "123",
            "customData": {
              "fooValue": 74,
              "barValue": true,
              "bazValue": "foo"
            }
          },
          "execution": [{
            "command": "action.devices.commands.OnOff",
            "params": {
              "on": true
            }
          }]
        }]
      }
    }]
}

Key concepts from the EXECUTE request:

  • Execute intents can specify multiple devices in the same request. Notice the commands array consists of an object with a list of devices and a list of executions.
  • Device commands are associated with a corresponding device trait. There is always a list of available commands given the list of traits for a device. For example, a fan with a trait action.devices.traits.FanSpeed has a command for changing the speed: action.devices.commands.SetFanSpeed.
  • The params always map to command, and when possible, will usually map the exact param name with the corresponding state name. For example, an IoT device with trait action.devices.traits.Timer has a state timerPaused with a corresponding command action.devices.commands.TimerPause.

Check out the reference for EXECUTE for a detailed list of all fields in the request and response.

You need to implement every command for every trait you choose — there are no optional commands. The only exception to this rule is if there is a configured attribute which enables or disables a particular command. For example, the action.devices.traits.FanSpeed trait has a reversible attribute which enables or disables the action.devices.commands.Reverse command.

4. DISCONNECT

The purpose of the disconnect request is to notify your cloud service that the user no-longer wants to use the Assistant to control their device. For example, the user may be throwing an old IoT light bulb away or giving the device to another user.

{
    "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
    "inputs": [{
      "intent": "action.devices.DISCONNECT",
    }]
}

There is no response needed for this call — nice!

Once your service receives this request, you should stop all future outgoing calls, such as Report State.

Sending calls to Google

There are two main outgoing calls that you need to implement in your cloud service. These two calls notify Google when something changes.

Request Sync

If the user adds or removes a device from your service, you must enable the new device for users on the Assistant using the Request Sync API. For example, if I already owned and linked two smart lights to Google Assistant and then purchased a third light bulb, your service needs to alert Google of the additional light bulb added to my account via a Request Sync call.

Specifically, you should always alert the Google Assistant when the following events happen:

  • If the user adds a new device.
  • If the user removes an existing device.
  • If the user renames an existing device.
  • If the user changes a device location such as rooms (only if your application supports this).
  • If you implement a new device type, trait, or add a new device feature.

An important piece on this request is the userAgentId from the original SYNC call we made here.

Report State

Report State is an API for pushing the real-time state of devices to Google Assistant. This is especially important since Google Assistant only augments your IoT service with conversational capabilities, it does not act as the source of truth for the state of the user’s device.

In the light bulb example, users could ask Google Assistant to turn on the light or they could flip a physical switch to turn it on. In the case of flipping a switch, the Assistant has no way of knowing the new state of the device without some form of pushed-state notification to Google.

Home Graph stores the state that is sent with Report State. Some readers may have noticed that we return state in the response to EXECUTE and QUERY intents. Google only uses QUERY and EXECUTE states for speech responses; not for storing in the Home Graph. As a result, you should call Report State even if the new state of the device has already been returned in the response to an EXECUTE or QUERY intent. The ReportState API should also be called right after a SYNC intent.

To inspect your device state as seen by the Assistant, and for live debugging, you can use the Test Suite for Smart Home.

Error Responses

Error responses are critical for properly communicating the status of a command’s execution to the end-user.

For example, if the user asks to turn on the light, but the light is not responding to your cloud service, then your cloud service would want to properly relay the failed execution intent back to the user with an errorCode of deviceOffline. Google Assistant uses an errorCode to tell the user success for failure, and if a command failed, why it failed.

There are two major kinds of errors: global and device-level. Global errors pertain to problems related to your cloud service or if there is a hub controlling multiple devices. Device-level errors are always meant to only be related to a particular device.

Global errors:

{
  "requestId": "12345",
  "payload": {
    "errorCode": "inSoftwareUpdate",
    "status" : "ERROR"
  }
}

Device-level errors:

{
  "requestId": "12345",
  "payload": {
    "devices": {
      "device-id-1": {
        "errorCode": "deviceOffline",
        "status" : "ERROR"
      },
      "device-id-2": {
        "errorCode": "deviceOffline",
        "status" : "ERROR"
      }
    }
  }
}

When adding errorCodes to your implementation, be sure to carefully review all the available codes. These codes can get very specific, for example unknownFoodPreset is an available errorCode. You can view all errorCodes here: https://developers.google.com/actions/smarthome/reference/errors-exceptions

Tools, Links, & TL;DR

You must handle the following intents in your cloud service:

  • SYNC — gives Google a list of all the user’s devices and their various configurations
  • QUERY — returns the state of devices when the user asks, “Hey Google, is my refrigerator running?”
  • EXECUTE — changes the state or runs a function on the device, “Hey Google turn on my lights.”
  • DISCONNECT — removes all associations between Google and the 3rd party cloud service for the user, used when the user no longer wants to control their device with the Assistant.

You must implement the following APIs for pushing information:

  • Request Sync — the user purchased a new device and we need to let the Assistant know about it.
  • Report State — any change-of-state for any device, we need to let the Assistant know about it.

You can validate your JSON for each intent using Google’s online validator: https://developers.google.com/actions/smarthome/tools/validator/

You can test and debug your implementation using Google’s online test suite:

https://developers.google.com/actions/smarthome/tools/smart-home-test-suite

Lastly, be sure to properly implement errorCodes for your device so that the Assistant can properly communicate to the user not just that something didn’t work, but why it didn’t work. Take a look at all of our available errorCodes here: https://developers.google.com/actions/smarthome/reference/errors-exceptions

You can also check out some additional material here: