February 4, 2016 at 08:10Custom gestures, Kinect + Unity. Part 2
- Tutorial
We continue our tutorial on using custom gestures in the Kinect + Unity bundle. In the first part, we examined the process of learning gestures, as a result of which we got a trained model in the form of a .gdb file. Today we will use this model in Unity.
We will create a project in Unity, add the downloaded Kinect packages: Assets -> Import package -> Custom package ..., select Kinect.VisualGestureBuilder.2.0.1410.19000.unitypackage and Kinect.2.0.1410.19000.unitypackage (versions may vary). There may be a problem with the fact that some files in these packages are the same, Unity adds both files with the names File.cs and File 1.cs, in this case we simply delete all files with index 1 (the list will be in error messages).
The structure of an empty project with added packages:
Run the ready-made example to make sure everything works. There are two examples in the downloaded package for Unity: GreenScreen and KinectView. From the KinectView example, we add the two folders “Materials” and “Scripts” and the file “MainScene.unity”. Open the scene (MainScene.unity) and run it. You should get something like the following:
If we have achieved such a result, then everything works for us, we can proceed to the main part.
First, we will ensure that a message is displayed in the log when the gesture is triggered.
From the Kinect SDK we need:
In addition, we will need two classes to process received frames from Kinect: VisualGestureBuilderFrameReader and BodyFrameReader .
It is understood that at this stage we have an empty Unity project with imported Kinect packages, as in the example above.
Create an empty object (GameObject -> Create Empty), call it KinectManager. Add a new script called KinectManagerScript to our object (in this tutorial only scripts written in C # are considered). Open the script, add using'i:
Inside the class, declare the objects of the above classes:
Now we need to initialize the values and add event handlers. Create the InitKinect () method and call it in the Start () method. To get started, upload gestures from our model:
Note: we encountered a problem when Kinect does not turn on immediately, and therefore the Open () method did not work. This was treated by periodically checking the IsAvailable flag in Update and opening it if Kinect is available.
Starting the application, and if everything is fine, we will see the following in the log:
This means that we successfully opened and loaded the gestures (in our case, one) from our trained models.
Obviously, gestures need to be detected in humans, so we need to get a list of all the people Kinect sees. We initialize the _bodyFrameSource and _bodyFrameReader objects and add the “frame came” event handler (we do all this in InitKinect):
We initialize _gestureFrameSource in the same way, pause it and add an event handler:
In the _bodyFrameReader_FrameArrived handler, we want to get information about people and, if there is someone in the frame, select the first one (for simplicity), whose gestures we will track.
Launch the application. When Kinect sees at least one person, we will see "_currentBody is not null", if there is nobody in the frame - "_currentBody is null". Log example:
We were looking for an active person in order to recognize his gestures. If we find a person, save his id in _gestureFrameSource and remove from pause. Our condition will change to the following:
The last thing we need from Kinect is a gesture handler. We check the validity of our person id, we get the current frame in the same way as we did it in _bodyFrameReader_FrameArrived:
We get the current discrete gesture recognition:
if there are any gestures, look, our gesture or not:
The threshold value of Confidence depends on the quality of your training, that is, it is the value below which the noise, and above which the gesture (remember the "Christmas tree" from the first part), it is selected empirically :)
Running the application, we see that multiple gestures occur detection. This is due to the fact that the results for each frame are independent, and since the gesture is not instantaneous (several frames), we get a positive result for each frame. This can be fixed with a logical flag:
The last thing we need to do with KinectManagerScript is to generate an event when we saw the gesture. Declare:
and when they found the gesture:
This concludes with KinectManagerScript. Let's go back to our scene and create a sphere. We will name “MainSphere”, add a new script “MainSphereScript”. In the Start () method, create an event handler, check with the log that everything works:
In principle, this is the goal of our tutorial to make our gesture a certain event, which we can subscribe to and perform the actions that we need. For the purity of the experiment, we add the motion of our sphere in a gesture, as a result we get approximately the following:
Scripts:
MainSphereScript.cs
KinectManagerScript.cs
Kinect + Unity Setup
We will create a project in Unity, add the downloaded Kinect packages: Assets -> Import package -> Custom package ..., select Kinect.VisualGestureBuilder.2.0.1410.19000.unitypackage and Kinect.2.0.1410.19000.unitypackage (versions may vary). There may be a problem with the fact that some files in these packages are the same, Unity adds both files with the names File.cs and File 1.cs, in this case we simply delete all files with index 1 (the list will be in error messages).
The structure of an empty project with added packages:
Run the ready-made example to make sure everything works. There are two examples in the downloaded package for Unity: GreenScreen and KinectView. From the KinectView example, we add the two folders “Materials” and “Scripts” and the file “MainScene.unity”. Open the scene (MainScene.unity) and run it. You should get something like the following:
If we have achieved such a result, then everything works for us, we can proceed to the main part.
First, we will ensure that a message is displayed in the log when the gesture is triggered.
From the Kinect SDK we need:
- KinectSensor - a class representing the Kinect sensor;
- VisualGestureBuilderDatabase - a class that represents a gesture base;
- VisualGestureBuilderFrameSource - handling gestures on frames received from Kinect;
- Body - a class representing a person;
- Gesture - a class representing a gesture;
- BodyFrameSource - information about people found on frames received from Kinect.
In addition, we will need two classes to process received frames from Kinect: VisualGestureBuilderFrameReader and BodyFrameReader .
Loading gestures in Unity
It is understood that at this stage we have an empty Unity project with imported Kinect packages, as in the example above.
Create an empty object (GameObject -> Create Empty), call it KinectManager. Add a new script called KinectManagerScript to our object (in this tutorial only scripts written in C # are considered). Open the script, add using'i:
using Microsoft.Kinect.VisualGestureBuilder;
using Microsoft.Kinect;
Inside the class, declare the objects of the above classes:
VisualGestureBuilderDatabase _dbGestures;
Windows.Kinect.KinectSensor _kinect;
VisualGestureBuilderFrameSource _gestureFrameSource;
Windows.Kinect.BodyFrameSource _bodyFrameSource;
VisualGestureBuilderFrameReader _gestureFrameReader;
Windows.Kinect.BodyFrameReader _bodyFrameReader;
Gesture _swipeUpDown; // наш жест
Windows.Kinect.Body[] _bodies; // все пользователи, найденные Kinect'ом
Windows.Kinect.Body _currentBody = null; //Текущий пользователь, жесты которого мы отслеживаем
public string _getsureBasePath = "upDown.gbd"; //Путь до нашей обученной модели
Now we need to initialize the values and add event handlers. Create the InitKinect () method and call it in the Start () method. To get started, upload gestures from our model:
void InitKinect()
{
_dbGestures = VisualGestureBuilderDatabase.Create(_getsureBasePath);
_bodies = new Windows.Kinect.Body[6];
_kinect = Windows.Kinect.KinectSensor.GetDefault();
_kinect.Open();
_gestureFrameSource = VisualGestureBuilderFrameSource.Create(_kinect, 0);
foreach (Gesture gest in _dbGestures.AvailableGestures)
{
if (gest.Name == "UpDownSwipe_Right")
{
_gestureFrameSource.AddGesture(gest);
_swipeUpDown = gest;
Debug.Log("Added:" + gest.Name);
}
}
}
Note: we encountered a problem when Kinect does not turn on immediately, and therefore the Open () method did not work. This was treated by periodically checking the IsAvailable flag in Update and opening it if Kinect is available.
Starting the application, and if everything is fine, we will see the following in the log:
This means that we successfully opened and loaded the gestures (in our case, one) from our trained models.
Gesture detection
Obviously, gestures need to be detected in humans, so we need to get a list of all the people Kinect sees. We initialize the _bodyFrameSource and _bodyFrameReader objects and add the “frame came” event handler (we do all this in InitKinect):
_bodyFrameSource = _kinect.BodyFrameSource;
_bodyFrameReader = _bodyFrameSource.OpenReader();
_bodyFrameReader.FrameArrived += _bodyFrameReader_FrameArrived;
We initialize _gestureFrameSource in the same way, pause it and add an event handler:
_gestureFrameReader = _gestureFrameSource.OpenReader();
_gestureFrameReader.IsPaused = true;
_gestureFrameReader.FrameArrived += _gestureFrameReader_FrameArrived;
In the _bodyFrameReader_FrameArrived handler, we want to get information about people and, if there is someone in the frame, select the first one (for simplicity), whose gestures we will track.
Method source code
void _bodyFrameReader_FrameArrived(object sender, Windows.Kinect.BodyFrameArrivedEventArgs args)
{
var frame = args.FrameReference;
using (var multiSourceFrame = frame.AcquireFrame())
{
multiSourceFrame.GetAndRefreshBodyData(_bodies); //обновляем данные о найденных людях
_currentBody = null;
foreach (var body in _bodies)
{
if (body != null && body.IsTracked)
{
_currentBody = body; // для простоты берем первого найденного человека
break;
}
}
if (_currentBody != null)
{
Debug.Log("_currentBody is not null");
}
else
{
Debug.Log("_currentBody is null");
}
}
}
Launch the application. When Kinect sees at least one person, we will see "_currentBody is not null", if there is nobody in the frame - "_currentBody is null". Log example:
We were looking for an active person in order to recognize his gestures. If we find a person, save his id in _gestureFrameSource and remove from pause. Our condition will change to the following:
if (_currentBody != null)
{
Debug.Log("_currentBody is not null");
_gestureFrameSource.TrackingId = _currentBody.TrackingId;
_gestureFrameReader.IsPaused = false;
}
else
{
Debug.Log("_currentBody is null");
_gestureFrameSource.TrackingId = 0;
_gestureFrameReader.IsPaused = true;
}
The last thing we need from Kinect is a gesture handler. We check the validity of our person id, we get the current frame in the same way as we did it in _bodyFrameReader_FrameArrived:
if (_gestureFrameSource.IsTrackingIdValid)
{
Debug.Log("Tracking id is valid, value = " + _gestureFrameSource.TrackingId);
using (var frame = args.FrameReference.AcquireFrame())
{
if (frame != null)
{
/*…*/
}
}
}
We get the current discrete gesture recognition:
var results = frame.DiscreteGestureResults;
if there are any gestures, look, our gesture or not:
if (results != null && results.Count > 0)
{
DiscreteGestureResult swipeUpDownResult;
results.TryGetValue(_swipeUpDown, out swipeUpDownResult);
Debug.Log("Result not null");
if (swipeUpDownResult.Confidence > 0.1)
{
Debug.Log("Up Down Gesture");
}
}
The threshold value of Confidence depends on the quality of your training, that is, it is the value below which the noise, and above which the gesture (remember the "Christmas tree" from the first part), it is selected empirically :)
Running the application, we see that multiple gestures occur detection. This is due to the fact that the results for each frame are independent, and since the gesture is not instantaneous (several frames), we get a positive result for each frame. This can be fixed with a logical flag:
bool gestureDetected = false;
…
if (swipeUpDownResult.Confidence > 0.1)
{
if (!gestureDetected)
{
gestureDetected = true;
Debug.Log("Up Down Gesture");
}
}
else
{
gestureDetected = false;
}
Using gestures
The last thing we need to do with KinectManagerScript is to generate an event when we saw the gesture. Declare:
public delegate void SimpleEvent();
public static event SimpleEvent OnSwipeUpDown;
and when they found the gesture:
if (!gestureDetected)
{
gestureDetected = true;
Debug.Log("Up Down Gesture");
if (OnSwipeUpDown!= null)
OnSwipeUpDown();
}
This concludes with KinectManagerScript. Let's go back to our scene and create a sphere. We will name “MainSphere”, add a new script “MainSphereScript”. In the Start () method, create an event handler, check with the log that everything works:
void Start ()
{
KinectManagerScript.OnSwipeUpDown += new KinectManagerScript.SimpleEvent(KinectManagerScript_OnSwipeUpDown);
}
void KinectManagerScript_OnSwipeUpDown()
{
Debug.Log("upDown From listener");
}
In principle, this is the goal of our tutorial to make our gesture a certain event, which we can subscribe to and perform the actions that we need. For the purity of the experiment, we add the motion of our sphere in a gesture, as a result we get approximately the following:
Scripts:
MainSphereScript.cs
KinectManagerScript.cs