Following the previous blog (Using Unity To Control Arduino) I have decided to step up the challenge and introducing a character in Unity powered by Speech to Text. Ryan Anderson created a series of youtube videos that inspired me to try this out by using the 2020 version of Unity and the Watson SDK.
This is the result of this tutorial:
If you are lazy like me and you want directly the code, make sure to check out my GitHub repo containing the compelte Unity Project used for this blog.
Requirements:
- Unity
- Arduino Uno, 3 LEDs, 3 resistors (220 ohm), wires, breadboard
- IBM Cloud Account & Watson Speech to Text LITE instance (free)
What you will learn:
- How to send inputs from a Unity character powered by Watson Speech to Text to Arduino
Language:
- C# in Unity
- C/C++ functions for Arduino IDE
First: the basics & Arduino Script
First, I would suggest to follow my previous blog on how to use Unity with Arduino because in this tutorial we will build on top of it. The same Arduino script will be used ("Traffic-Light") so make sure to upload the script to your Arduino.
In this Blog we will tackle the remaining tasks:
Create a Character in Unity
Include Speech to Text in Unity
Take the input from Speech to Text and use it to activate Arduino and change Unity Object.
Important: In Unity, change the setting of the player by going into Edit --> Project settings --> Player --> API Compatibility Level --> .NET 4.x (not subset)
1. Create a Character in Unity
I cannot take credit for this great character Jammo build by Mix and Jam / Curiomatic. Download and import this asset using the Unity Asset store.
Once you have imported it, look for the following scene: Asset --> Jammo-Character --> Scene --> Jammo Scene (double click) - If you click on the PLAY button you should be able to play with the character straightaway.
2. Include Speech to Text in Unity
First, go to your IBM Cloud Account https://cloud.ibm.com/ and get a free instance of Watson Speech to Text (go to Create Resources -> type in "Speech to text" -> Choose a Lite plan and click on Create).
Once you have the instance, click on it and you should see your API Key and URL. Copy them.
Then, you need the IBM Watson SDK for Unity. I couldn't find it anymore on the Unity Asset Store but you have the Git repo available here.
Follow the guidelines by checking out this paragraph in the ReadME "Getting the Watson SDK and adding it to Unity". Make sure to get the latest stable releases. Move the downloaded files into the Assets directory of your Unity project.
Now, from the Example folder, drag and drop the Scene called "ExampleStreaming.unity" into your Hierarchy tab.
This way, we can leverage what the Watson Team has already created for us using the Speech to Text services. The only thing left is to do, is to add your API Key and URL in the "ExampleStreaming" Object and add the credentials as parameters in the Example Streaming Script section (see on the left):
Now, Click on PLAY button and use your microphone to see if the Instance is working. You should see a grey text in the middle of the screen transcribing the word that you are saying.
Easy so far right?
3. Take the input from Speech to Text and use it to activate Arduino & change objects colors in Unity
Now the speech to text instace is able to read your words as input. I want it to be sensitive to 3 words: "Green", "Yellow" and "Red"and based on those changing the color of:
an object in Unity (i.e. a Sphere)
light up a specific LED in Arduino
3.1 Object in Unity
Create a new Object (sphere) and create 3 materials: Green material, yellow material and red material.
3.2. LEDs in Arduino
Follow the Schema shown in the How to use Unity to control Arduino and the script used as already mentioned is the "Traffic-Light.ino".
3.3. Assemble
Now, access the Example Streaming Script. On GitHub you will be able to find the whole script (ExampleStreaming.cs but let's try to comment what are the additional lines needed.
For the Arduino connection part you have to add the following libraries:
// libraries to include for Arduino
using System.IO.Ports;
using System.Threading;
Serial Port & Connection
On a Mac, use the command "ls /dev/tty.*" to check what is the Serial Port of your Arduino.
// For Arduino Serial Port creation
public static SerialPort sp = new SerialPort("/dev/tty.usbmodemxxxxxx", 9600);
// For Arduino open connection ad hoc function
public void OpenConnection()
{
if (sp!=null)
{
if(sp.IsOpen)
{
sp.Close();
print("Closing port, because it's already open");
}
else
{
sp.Open();
sp.ReadTimeout = 100;
print("port open");
}
}
else
{
if(sp.IsOpen)
{
print("port is already open");
}
else{
print("port == null");
}
}
}
in the Start function of the Script add also the string to open the connection to Arduino
void Start()
{
LogSystem.InstallDefaultReactors();
Runnable.Run(CreateService());
OpenConnection(); // Add the function in Start
}Object in Unity
Add 4 public fields so that Unity knows
what object to change the colors to
the three materials to invoke when the color word is detected.
public MeshRenderer ObjectMeshRenderer;
public Material red_material;
public Material green_material;
public Material yellow_material;Change the Color and Light up the LED
In the private void OnRecognize function add the code that will activate the two functions: 1. Change the color of the Sphere and 2. Light up the LED on Arduino.
foreach (var alt in res.alternatives)
{
string text = string.Format("{0} ({1}, {2:0.00})\n", alt.transcript, res.final ? "Final" : "Interim", alt.confidence);
Log.Debug("ExampleStreaming.OnRecognize()", text);
ResultsField.text = text;
// Add this part-----------
if(alt.transcript.Contains("red")) {
ObjectMeshRenderer.material = red_material; //Change UnityObj
sp.Write("r");// Send key letter to Arduino LED
}
if(alt.transcript.Contains("green")) {
ObjectMeshRenderer.material = green_material;
sp.Write("g");
}
if(alt.transcript.Contains("yellow")) {
ObjectMeshRenderer.material = yellow_material;
sp.Write("y");
}
}
[...]Save the script and the last thing that you need to do is the following : drag and drop the objects in the public fields.
Now, click on the PLAY button and by saying out loud the colors, the Sphere in Unity should change color and the correct Arduino LED should light up.
The whole project:
The complete code is Available on GitHub. All the steps shown so far are already included in the project! The only thing that you need to do to is the following:
Clone the repository
Open the Unity Project "UnityArduinoSTT" in Unity (use Unity Hub)
Add the Watson Speech to Text Instance credentials (API Key and URL) as shown in Step 2 of this tutorial
Upload the "Traffic-Light.ino" Script to your Arduino
Update the Serial Port with your Arduino Port in the "Example Streaming" Script as shown in step 3.3 . Just replace "/dev/tty.usbmodemxxxxxx" with your serial port of your Arduino.
PLAY!
Conclusion:
This is a basic example on how to integrate Unity, Watson Speech to Text Service and Arduino. Now, no one can stop you from adding more Watson Services to create a more interactive virtual agent :)