Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)

The Kinect For Windows SDK v1.0 is awesome, but it doesn't recover well when it loses sight of you whilst tracking your skeleton. With a timer control and a little code to restart the KinectSensor, you can recover is 5 seconds or less.

This is the same code I use in GesturePak, a gesture recording and recognition SDK for Kinect for Windows.

Create a new WPF application and add a reference to the Kinect SDK assembly:

C:\Program Files\Microsoft SDKs\Kinect\v1.0\Assemblies\Microsoft.Kinect.dll

You don't need any controls for this demo. It simply turns the form red when tracking, and white when not tracking.

The basic idea is that when we lose tracking for the first time, you kick off a timer to fire it's Tick event in 1/2 a second. In the timer Tick handler you stop the Kinect Sensor, Start it up again, and then set the timer's Interval to a more reasonable amount of time. I use 5 seconds, but you should test it in your app. If it's too short, you don't give the Kinect time to track you, and if it's too long and it doesn''t track you, you could be sitting and waiting too long. 5 seconds seems reasonable to me.

When the state changes from not tracking to tracking, you simply disable the timer. 

Here's the code in both C# and VB.NET. In each case you can replace the default code behind MainWindow.xaml and run it. The C# app has to have a reference to System.Windows.Forms

C#

using System;

using System.Collections;

using System.Collections.Generic;

using System.Data;

using System.Diagnostics;

using System.Windows.Forms;

using System.Windows.Media;

using Microsoft.Kinect;

namespace TrackingTestCSharp

{

    /// <summary>

    /// This WPF demo shows you how gracefull recovery when the Kinect loses

    /// skeletal tracking. The Kinect device does not recover well on it's own.

    /// While restarting the Kinect may seem drastic, it's the only way I can 

    /// figure out how to do it. If you find a better way, email me at 

    /// carl@franklins.net. Thanks!

    /// </summary>

    /// <remarks></remarks>

    partial class MainWindow

    {

        private KinectSensor sensor;

        

        //-- boolean to keep track of the state

        private bool isTracking;

        //-- This is the interval after which the kinect is restarted the second 

        //   and subsequent times.

        private TimeSpan autoTrackingRecoveryInterval = TimeSpan.FromSeconds(5);

        //-- Timer used to restart

        private System.Windows.Threading.DispatcherTimer trackingRecoveryTimer = 

                new System.Windows.Threading.DispatcherTimer();

        public MainWindow()

        {

            Loaded += MainWindow_Loaded;

        }

        private void MainWindow_Loaded(object sender, System.Windows.RoutedEventArgs e)

        {

            //-- Make sure we have a sensor connected

            if (KinectSensor.KinectSensors.Count == 0) {

                MessageBox.Show("No Kinect Found");

                System.Windows.Application.Current.Shutdown();

            }

            //-- Start the first sensor, enabling the skeleton

            sensor = KinectSensor.KinectSensors[0];

            sensor.SkeletonStream.Enable(new TransformSmoothParameters());

            sensor.Start();

            //-- Hook the events

            sensor.SkeletonFrameReady += sensor_SkeletonFrameReady;

            trackingRecoveryTimer.Tick += trackingRecoveryTimer_Tick;

        }

        private void sensor_SkeletonFrameReady(object sender, Microsoft.Kinect.SkeletonFrameReadyEventArgs e)

        {

            //-- Get the frame

            dynamic frame = e.OpenSkeletonFrame();

            //-- Bail if it's nothing

            if (frame == null) return;

            using (frame) {

                //-- Bail if no data is returned.

                if (frame.SkeletonArrayLength == 0) return;

                //-- Get the data from the frame into an array

                Skeleton[] data = new Skeleton[frame.SkeletonArrayLength];

                frame.CopySkeletonDataTo(data);

                //-- Is there really no data?

                if (data.Length == 0) return

                //-- Is this skeleton being tracked?

                if (data[0].TrackingState == SkeletonTrackingState.Tracked) {

                    if (!isTracking) {

                        //-- If we weren't tracking, set the background to red

                        // to indicate that we are tracking

                        this.Background = Brushes.Red;

                        //-- we're tracking

                        isTracking = true;

                        //-- disable the timer

                        trackingRecoveryTimer.IsEnabled = false;

                        }

                 } else {

                     if (isTracking) {

                        //-- State changed from tracking to not tracking

                        isTracking = false;

                        //-- Set the background white to indicated we're NOT tracking

                        this.Background = Brushes.White;

                        //-- Since this is the first time we've lost tracking,

                        //   restart after 1/2 a second

                        trackingRecoveryTimer.Interval = TimeSpan.FromSeconds(0.5);

                        trackingRecoveryTimer.IsEnabled = true;
                    }

                }

            }

        }

        private void trackingRecoveryTimer_Tick(object sender, System.EventArgs e)

        {

            //-- Disable the timer

            trackingRecoveryTimer.IsEnabled = false;

            //-- If we're already tracking, bail. 

            if (isTracking) return;

             //-- Stop the sensor, ignoring errors

            try {

                sensor.Stop();

            } catch  {

            }

            //-- Start the sensor, ignoring errors

            try {

                sensor.Start();

            } catch  {

            }

            //-- Come back here in the defined interval

            trackingRecoveryTimer.Interval = autoTrackingRecoveryInterval;

            trackingRecoveryTimer.IsEnabled = true;

        }

    }

}

VB.NET

Imports Microsoft.Kinect
 
''' <summary>
''' This WPF demo shows you how gracefull recovery when the Kinect loses
''' skeletal tracking. The Kinect device does not recover well on it's own.
''' While restarting the Kinect may seem drastic, it's the only way I can 
''' figure out how to do it. If you find a better way, email me at 
''' carl@franklins.net. Thanks!
''' </summary>
''' <remarks></remarks>
Class MainWindow
 
    Private WithEvents sensor As KinectSensor
 
    Private isTracking As Boolean   '-- boolean to keep track of the state
 
    '-- This is the interval after which the kinect is restarted the second 
    '   and subsequent times.
    Private autoTrackingRecoveryInterval = TimeSpan.FromSeconds(5)
 
    '-- Timer used to restart
    Private WithEvents trackingRecoveryTimer As New Windows.Threading.DispatcherTimer
 
    Private Sub MainWindow_Loaded(sender As Object, e As System.Windows.RoutedEventArgs) Handles Me.Loaded
        '-- Make sure we have a sensor connected
        If KinectSensor.KinectSensors.Count = 0 Then
            MessageBox.Show("No Kinect Found")
            Application.Current.Shutdown()
        End If
 
        '-- Start the first sensor, enabling the skeleton
        sensor = KinectSensor.KinectSensors(0)
        sensor.SkeletonStream.Enable(New TransformSmoothParameters)
        sensor.Start()
    End Sub
 
    Private Sub sensor_SkeletonFrameReady(sender As Object, e As Microsoft.Kinect.SkeletonFrameReadyEventArgs) Handles sensor.SkeletonFrameReady
        '-- Get the frame
        Dim frame = e.OpenSkeletonFrame
        '-- Bail if it's nothing
        If frame Is Nothing Then Return
 
        Using frame
            '-- Bail if no data is returned.
            If frame.SkeletonArrayLength = 0 Then Return
 
            '-- Get the data from the frame into an array
            Dim data(frame.SkeletonArrayLength - 1) As Skeleton
            frame.CopySkeletonDataTo(data)
 
            '-- Is there really no data?
            If data.Length = 0 Then Return
 
            '-- Is this skeleton being tracked?
            If data(0).TrackingState = SkeletonTrackingState.Tracked Then
                If Not isTracking Then
                    '-- If we weren't tracking, set the background to red
                    '   to indicate that we are tracking
                    Me.Background = Brushes.Red
                    '-- we're tracking
                    isTracking = True
                    '-- disable the timer
                    trackingRecoveryTimer.IsEnabled = False
                End If
            Else
                If isTracking Then
                    '-- State changed from tracking to not tracking
                    isTracking = False
                    '-- Set the background white to indicated we're NOT tracking
                    Me.Background = Brushes.White
                    '-- Since this is the first time we've lost tracking,
                    '   restart after 1/2 a second
                    trackingRecoveryTimer.Interval = TimeSpan.FromSeconds(0.5)
                    trackingRecoveryTimer.IsEnabled = True
                End If
            End If
 
        End Using
    End Sub
 
    Private Sub trackingRecoveryTimer_Tick(sender As Object, e As System.EventArgs) Handles trackingRecoveryTimer.Tick
        '-- Disable the timer
        trackingRecoveryTimer.IsEnabled = False
        '-- If we're already tracking, bail. 
        If isTracking Then Return
        '-- Stop the sensor, ignoring errors
        Try
            sensor.Stop()
        Catch ex As Exception
        End Try
        '-- Start the sensor, ignoring errors
        Try
            sensor.Start()
        Catch ex As Exception
        End Try
        '-- Come back here in the defined interval
        trackingRecoveryTimer.Interval = autoTrackingRecoveryInterval
        trackingRecoveryTimer.IsEnabled = True
    End Sub
End Class

Comments

[Jean-Philippe Encausse]

Hi,
Thanks for the article, I have the same issue explained on StackOverflow:
http://stackoverflow.com/questions/18288111/kinect-gesture-recognition-reset-recovering-and-accuracy

Is there a way to detect the skeleton is completly Messed Up ?

I can't reset each time Skeleton.Tracked is false because sometimes it switch on/off many times. I don't undestand why it provides wrong values even if I'm sit in front of computer with Joint absolutly not visible