Monday
Apr302012

Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)

The Kinect For Windows SDK v1.0 is awesome, but it doesn't recover well when it loses sight of you whilst tracking your skeleton. With a timer control and a little code to restart the KinectSensor, you can recover is 5 seconds or less.

This is the same code I use in GesturePak, a gesture recording and recognition SDK for Kinect for Windows.

Create a new WPF application and add a reference to the Kinect SDK assembly:

C:\Program Files\Microsoft SDKs\Kinect\v1.0\Assemblies\Microsoft.Kinect.dll

You don't need any controls for this demo. It simply turns the form red when tracking, and white when not tracking.

The basic idea is that when we lose tracking for the first time, you kick off a timer to fire it's Tick event in 1/2 a second. In the timer Tick handler you stop the Kinect Sensor, Start it up again, and then set the timer's Interval to a more reasonable amount of time. I use 5 seconds, but you should test it in your app. If it's too short, you don't give the Kinect time to track you, and if it's too long and it doesn''t track you, you could be sitting and waiting too long. 5 seconds seems reasonable to me.

When the state changes from not tracking to tracking, you simply disable the timer. 

Here's the code in both C# and VB.NET. In each case you can replace the default code behind MainWindow.xaml and run it. The C# app has to have a reference to System.Windows.Forms

C#

using System;
using System.Collections;
using System.Collections.Generic;
using System.Data;
using System.Diagnostics;
using System.Windows.Forms;
using System.Windows.Media;
using Microsoft.Kinect;
namespace TrackingTestCSharp
{
    /// <summary>
    /// This WPF demo shows you how gracefull recovery when the Kinect loses
    /// skeletal tracking. The Kinect device does not recover well on it's own.
    /// While restarting the Kinect may seem drastic, it's the only way I can 
    /// figure out how to do it. If you find a better way, email me at 
    /// carl@franklins.net. Thanks!
    /// </summary>
    /// <remarks></remarks>
    partial class MainWindow
    {
        private KinectSensor sensor;
        
        //-- boolean to keep track of the state
        private bool isTracking;
        //-- This is the interval after which the kinect is restarted the second 
        //   and subsequent times.
        private TimeSpan autoTrackingRecoveryInterval = TimeSpan.FromSeconds(5);
        //-- Timer used to restart
        private System.Windows.Threading.DispatcherTimer trackingRecoveryTimer = 
                new System.Windows.Threading.DispatcherTimer();
        public MainWindow()
        {
            Loaded += MainWindow_Loaded;
        }
        private void MainWindow_Loaded(object sender, System.Windows.RoutedEventArgs e)
        {
            //-- Make sure we have a sensor connected
            if (KinectSensor.KinectSensors.Count == 0) {
                MessageBox.Show("No Kinect Found");
                System.Windows.Application.Current.Shutdown();
            }
            //-- Start the first sensor, enabling the skeleton
            sensor = KinectSensor.KinectSensors[0];
            sensor.SkeletonStream.Enable(new TransformSmoothParameters());
            sensor.Start();
            //-- Hook the events
            sensor.SkeletonFrameReady += sensor_SkeletonFrameReady;
            trackingRecoveryTimer.Tick += trackingRecoveryTimer_Tick;
        }
        private void sensor_SkeletonFrameReady(object sender, Microsoft.Kinect.SkeletonFrameReadyEventArgs e)
        {
            //-- Get the frame
            dynamic frame = e.OpenSkeletonFrame();
            //-- Bail if it's nothing
            if (frame == null) return;
            using (frame) {
                //-- Bail if no data is returned.
                if (frame.SkeletonArrayLength == 0) return;
                //-- Get the data from the frame into an array
                Skeleton[] data = new Skeleton[frame.SkeletonArrayLength];
                frame.CopySkeletonDataTo(data);
                //-- Is there really no data?
                if (data.Length == 0) return
                //-- Is this skeleton being tracked?
                if (data[0].TrackingState == SkeletonTrackingState.Tracked) {
                    if (!isTracking) {
                        //-- If we weren't tracking, set the background to red
                        // to indicate that we are tracking
                        this.Background = Brushes.Red;
                        //-- we're tracking
                        isTracking = true;
                        //-- disable the timer
                        trackingRecoveryTimer.IsEnabled = false;
                        }
                 } else {
                     if (isTracking) {
                        //-- State changed from tracking to not tracking
                        isTracking = false;
                        //-- Set the background white to indicated we're NOT tracking
                        this.Background = Brushes.White;
                        //-- Since this is the first time we've lost tracking,
                        //   restart after 1/2 a second
                        trackingRecoveryTimer.Interval = TimeSpan.FromSeconds(0.5);
                        trackingRecoveryTimer.IsEnabled = true;
                    }
                }
            }
        }
        private void trackingRecoveryTimer_Tick(object sender, System.EventArgs e)
        {
            //-- Disable the timer
            trackingRecoveryTimer.IsEnabled = false;
            //-- If we're already tracking, bail. 
            if (isTracking) return;
             //-- Stop the sensor, ignoring errors
            try {
                sensor.Stop();
            } catch  {
            }
            //-- Start the sensor, ignoring errors
            try {
                sensor.Start();
            } catch  {
            }
            //-- Come back here in the defined interval
            trackingRecoveryTimer.Interval = autoTrackingRecoveryInterval;
            trackingRecoveryTimer.IsEnabled = true;
        }
    }
}

 

VB.NET

 

Imports Microsoft.Kinect
 
''' <summary>
''' This WPF demo shows you how gracefull recovery when the Kinect loses
''' skeletal tracking. The Kinect device does not recover well on it's own.
''' While restarting the Kinect may seem drastic, it's the only way I can 
''' figure out how to do it. If you find a better way, email me at 
''' carl@franklins.net. Thanks!
''' </summary>
''' <remarks></remarks>
Class MainWindow
 
    Private WithEvents sensor As KinectSensor
 
    Private isTracking As Boolean   '-- boolean to keep track of the state
 
    '-- This is the interval after which the kinect is restarted the second 
    '   and subsequent times.
    Private autoTrackingRecoveryInterval = TimeSpan.FromSeconds(5)
 
    '-- Timer used to restart
    Private WithEvents trackingRecoveryTimer As New Windows.Threading.DispatcherTimer
 
    Private Sub MainWindow_Loaded(sender As Object, e As System.Windows.RoutedEventArgs) Handles Me.Loaded
        '-- Make sure we have a sensor connected
        If KinectSensor.KinectSensors.Count = 0 Then
            MessageBox.Show("No Kinect Found")
            Application.Current.Shutdown()
        End If
 
        '-- Start the first sensor, enabling the skeleton
        sensor = KinectSensor.KinectSensors(0)
        sensor.SkeletonStream.Enable(New TransformSmoothParameters)
        sensor.Start()
    End Sub
 
    Private Sub sensor_SkeletonFrameReady(sender As Object, e As Microsoft.Kinect.SkeletonFrameReadyEventArgs) Handles sensor.SkeletonFrameReady
        '-- Get the frame
        Dim frame = e.OpenSkeletonFrame
        '-- Bail if it's nothing
        If frame Is Nothing Then Return
 
        Using frame
            '-- Bail if no data is returned.
            If frame.SkeletonArrayLength = 0 Then Return
 
            '-- Get the data from the frame into an array
            Dim data(frame.SkeletonArrayLength - 1) As Skeleton
            frame.CopySkeletonDataTo(data)
 
            '-- Is there really no data?
            If data.Length = 0 Then Return
 
            '-- Is this skeleton being tracked?
            If data(0).TrackingState = SkeletonTrackingState.Tracked Then
                If Not isTracking Then
                    '-- If we weren't tracking, set the background to red
                    '   to indicate that we are tracking
                    Me.Background = Brushes.Red
                    '-- we're tracking
                    isTracking = True
                    '-- disable the timer
                    trackingRecoveryTimer.IsEnabled = False
                End If
            Else
                If isTracking Then
                    '-- State changed from tracking to not tracking
                    isTracking = False
                    '-- Set the background white to indicated we're NOT tracking
                    Me.Background = Brushes.White
                    '-- Since this is the first time we've lost tracking,
                    '   restart after 1/2 a second
                    trackingRecoveryTimer.Interval = TimeSpan.FromSeconds(0.5)
                    trackingRecoveryTimer.IsEnabled = True
                End If
            End If
 
        End Using
    End Sub
 
    Private Sub trackingRecoveryTimer_Tick(sender As Object, e As System.EventArgs) Handles trackingRecoveryTimer.Tick
        '-- Disable the timer
        trackingRecoveryTimer.IsEnabled = False
        '-- If we're already tracking, bail. 
        If isTracking Then Return
        '-- Stop the sensor, ignoring errors
        Try
            sensor.Stop()
        Catch ex As Exception
        End Try
        '-- Start the sensor, ignoring errors
        Try
            sensor.Start()
        Catch ex As Exception
        End Try
        '-- Come back here in the defined interval
        trackingRecoveryTimer.Interval = autoTrackingRecoveryInterval
        trackingRecoveryTimer.IsEnabled = True
    End Sub
End Class

 

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (22)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    If you adore football, you in all probability have a favourite team from the National Football League or two and have a list of players who like to have seen.
  • Response
    Response: check over here
    Lovely Web-site, Keep up the excellent job. Thanks a lot.
  • Response
    Response: Asics Whizzer Lo
    and stay alert also any possible sudden changes simply because environment th Asics Top Seven at may pose Women's Asics Gel Noosa Tri 7 Asics Top Seven a multi Women's Asics Gel Noosa Tri 7 functional threat.The successive 2 quarters make an appointment with a reduction to educate yourself reg Men's ...
  • Response
    Response: hut kaufen
    Endesa League: 7 What Are the Limits of Life days 9 notes Classification: News, Paperw Dr. McKay Toluca Lake Dentist Office Reviews How Dental Implants are Structured and Supported ork, Competitive events, Liga Endesa, Endesa League, Prospect Express Endesa Little league Round nine: synopsis. We have now attained the ...
  • Response
    comunicação visual em curitiba letras caixa toldos luminosos, banners venha conferir
  • Response
    Response: www.wrapcandy.com
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Response: Len McDowall
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Response: Niagara therapy
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Response: Anthony Alles
    Carl Franklin - Intellectual Hedonism - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Response: Whittier dental
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Response: Anthony Alles
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Response: Anthony Alles
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)
  • Response
    Response: www.niagara.com.au
    Carl Franklin - Blog - Recovering Gracefully from Loss of Skeletal Tracking (Kinect For Windows)

Reader Comments (1)

Hi,
Thanks for the article, I have the same issue explained on StackOverflow:
http://stackoverflow.com/questions/18288111/kinect-gesture-recognition-reset-recovering-and-accuracy

Is there a way to detect the skeleton is completly Messed Up ?

I can't reset each time Skeleton.Tracked is false because sometimes it switch on/off many times. I don't undestand why it provides wrong values even if I'm sit in front of computer with Joint absolutly not visible

August 17, 2013 | Unregistered CommenterJean-Philippe Encausse

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>
« Automatically Adjust Input Gain During Speech Recognition | Main | Getting Started with Kinect For Windows in Visual Studio .NET »