Cortana

Cortana is now at the center of Windows Phone user’s day-to-day lives and as we see more integration with Microsoft’s other platforms, its going to be essential for developers to get their apps ready for Cortana and a great way of doing this is by getting your Windows Phone apps updated to universal apps and getting Cortana integrated!

Developing with Cortana will need voice commands that your application can understand and respond to when asked by the user outside of the app.

To get yourself started, you’ll need to create a Voice Command Definition (VCD) XML file, which represents a list of commands and actions that Cortana can recognize to interact with your app, in your solution. You’ll need to change the Build Action of the file to Content and set it’s Copy to output directory to Copy if newer. This is required so that your application can pick up the file within code when you install the commands XML file.

Adding Voice Commands

Now we have the starting point to integrating with Cortana in your Windows Phone app. In its current state, your XML file is empty (unless you added the VCD template to your solution). To get creating voice commands, we need to pad out the XML as follows:


<?xml version="1.0" encoding="utf-8"?> 
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.1">
    <CommandSet xml:lang="en-us" Name="James App">
        <CommandPrefix>James</CommandPrefix>
        <Example>Open Page 1</Example>
    </CommandSet>
</VoiceCommands>

The XML above allows us to start adding Voice Commands to Cortana with an English CommandSet. If you were to make your app localized, you’d add more CommandSet’s under the English one and change the language to your supported language.

Within our CommandSet, we set a CommandPrefix. This is the word or phrase that will prefix the command when speaking to Cortana so that she can recognise your application. For example in the case above, I would say “James Open Page 1” which would activate my application and send the command ‘Open Page 1’ for my app to recognize and handle. Simply saying “Open Page 1” would cause Cortana to do a Bing search.

We also give an example of a command that we can use in Cortana. This is the line of text that shows up underneath your application when you tap the “see more” button within Cortana to see what apps you can interact with.

Now that we have the basic setup, we need to add the commands themselves. This is done as follows:


<?xml version="1.0" encoding="utf-8"?> 
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.1">
    <CommandSet xml:lang="en-us" Name="James App">
        <CommandPrefix>James</CommandPrefix>
        <Example>Open Page 1</Example>

        <Command Name="openPages">
            <Example>Open Page 1</Example>
            <ListenFor>[Open] {page}</ListenFor>
            <Feedback>Opening {page}</Feedback>
            <Navigate/>
        </Command>

        <PhraseList Label="page">
            <Item>Page 1</Item>
            <Item>Page 2</Item>
        </PhraseList>
    </CommandSet>
</VoiceCommands>

In the above extract from my VCD file, you’ll notice a few things that have been added from the previous example. Adding a command for Cortana to recognize for your app is as simple as adding a Command at the top of your CommandSet.

The Command, like your CommandSet, needs an example. This example must be an exact representation of what the command will achieve, unlike the CommandSet’s which can be from any of the commands within it.

Your command also requires three more tags which are ListenFor, Feedback and Navigate. ListenFor, as it says, is what Cortana will be listening for. Feedback is what Cortana will speak back to the user when they have requested the command. Navigate, if set, will navigate the user to a page within your application. You can handle this differently in your application so that when your command is recognized, you perform the navigate within the app itself.

Within the ListenFor tag in the command, you’ll notice two parts to it. The first part, [Open], is a word that can be said. Using the square brackets around words within your ListenFor tag means that they aren’t required, but can be said in conjunction with any other words within the command. Without the square brackets, the word is required. This is essential if you’re looking for natural language when the user is requesting commands with Cortana.

The other part, {page}, is an object which is represented within a PhraseList. As you can see in the sample above, we have a PhraseList with the Label “page”. Within that PhraseList is a list of items. These items are what will replace that bracketed object within the command. Reading the XML, your Command is [Open] {page}, however Cortana will recognize this as “Open Page 1” or “Open Page 2”.

Now we have a VCD file which our app can recognize as commands for Cortana. However, we now need to get your application set-up to install the command file and make sure your app responds to the commands it receives.

Setting up your app

The first step here is to update your Package.appxmanifest file to include the Microphone capability. Easy.

Now we need to install the VCD file into the VoiceCommandManager. This is done as follows:


var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///JamesCommands.xml"));
await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(storageFile);

Obviously, replace the JamesCommands.xml line within the Uri to the one of your application’s VCD file. You’ll want to call this within your App.xaml.cs file so that it become installed and ready to use as soon as your application has been activated.

We’ve now got our voice commands installed and ready to use. If you’ve installed your application on your phone and allows the VCD file to install, you can exit your app and start using your commands within Cortana. However, your application will not know what to do with them. This is where we need to grab the voice command from the OnActivated method within the App.xaml.cs file which is done as follows:


protected override void OnActivated(IActivatedEventArgs args)
{
    base.OnActivated(args);
   
    if(args.Kind == ActivationKind.VoiceCommand)
    {
        var voiceCommandArgs = args as VoiceCommandActivatedEventArgs;

        if(voiceCommandArgs != null)
        {
            SpeechRecognitionResult result = voiceCommandArgs.Result;

            // ToDo: Do something with the result.
        }
    }
}

Using the above code in your OnActivated method, you’ll be able to get the command that was passed through to your application from Cortana. The result itself is split into various properties which you can use to get to the information you want from the spoken command.

There are two parts that you’ll need, first is the command name and the second is the text which was spoken. The code below will replace the // ToDo section within the code sample above:


string commandName = result.RulePath.FirstOrDefault();
string text = result.Text;

From those two values, you’ll be able to handle what to do within your application which can be done as follows:


switch(commandName)
{
    case "openPages":
        if(text.ToLower().Contains("page 1")
        {
            // Do Something
        }
        break;
}

Now you’ve got your application integrating with Cortana. When you’re handling the spoken text, you’ll need to factor in the optional parts that you’ve added into your ListenFor command as these may not come through with your text as part of the result due to them not being spoken by the user.

It’s very likely that Windows 10 applications built with Cortana will work in very much the same way as it currently does for phone, so getting your phone app ready now will allow you to easily move the code when you develop your universal Windows 10 apps.

If you have any questions, let me know in the comments section below and I’ll get back to you!

Author James Croft

James is a .NET developer for Black Marble Ltd. with over 5 years app development experience in Microsoft platforms. He enjoys blogging, helping others learn to code, making YouTube videos, spending time with his girlfriend but most of all, penguins.

More posts by James Croft

Leave a comment