These are the slides from my LIDNUG presentation on Developing with the Microsoft Kinect using the Kinect for Windows SDK. You can find the presentation recording at http://www.youtube.com/watch?v=0arzMSlqnHk

1. KINECT – THE HOW,
WHEN, AND WHERE OF
DEVELOPING WITH IT
PHILIP WHEAT
MANAGER, CHAOTIC MOON LABS
PHIL@CHAOTICMOON.COM

2. WHO I AM
Philip Wheat – Currently managing the Labs Division of
Chaotic Moon (http://www.chaoticmoon.com/Labs)
Former Evangelist with Microsoft – Architecture and
Startups.
Active in the Maker Movement – it’s about Bits AND Atoms,
not Bits or Atoms.
My neglected blog can be found at http://PhilWheat.net
I can be found on Twitter as @pwheat

3. WHY THIS TALK?
There has been a lot of talk about next generation interfaces
– but most interface talk today still seems to be around
HTML5/Native.
User interaction is very focused today on touch interfaces –
but this limits various scenarios and usage models.
We’ll look at the components of interacting with the user
through Kinect today and help you start looking at new
scenarios.

4. THE PROJECTS

5. DEVELOPING WITH KINECT
THE HOW

6. HARDWARE
VERSIONS
Remember, there are two versions of Kinect Hardware –
Kinect 360 and Kinect for Windows.
• Kinect for 360
• Can be used for SDK development
• Some Kinect for Windows Functionality not supported
• Not supported for Microsoft SDK production
• Kinect for Windows
• Can be used for development and deploymen
• Full Kinect for Windows Functionality (Near Mode,
Additional resolutions)
• Supported for production

7. HARDWARE
CAPABILITIES
• Cameras
• RGB Camera – 320x240, 640x480, 1024x768 (KFW)
• Depth Camera – 320x240, 640x480
• Depth Camera – .8M – 4M (standard Mode)
– .5M – 3M (near Mode)
• Audio
• Microphone Array
• 4 Microphones
• Audio directional detection
• Audio detection steering
• Tilt
• Tilt from -27 to 27 degrees from horizontal (accelerometer)

8. SOFTWARE
CAPABILITIES
• Cameras
• RGB Frames – event driven and polled
• Depth Frames – event driven and polled
• Skeleton Tracking – event driven and polled
• Seated Skeleton Tracking (Announced for KFW 1.5)
• Audio
• Voice Recognition in English
• Voice Recognition – French, Spanish, Italian, Japanese
(Announced for KFW 1.5)

9. OTHER HARDWARE
ITEMS
• Kinect needs 12V for operations – this requires a non-
standard connector. For use with a PC, Kinect requires a
supplemental power supply which is included for the
Kinect for Windows hardware but is not included with
Kinect 360’s that are bundled with Xbox slim models.
• Kinect tilt motor is update limited to prevent overheating.
• Lenses are available to reduce focus range – but produce
distortion and are not supported.
• Kinect has a fan to prevent overheating – be careful of
enclosed areas.
• Kinect contains a USB Hub internally – connecting it
through another USB Hub is not supported and will only
work with certain Hubs (through practical testing.)

10. CONNECTING TO
YOUR KINECT
Some Key code –
• KinectSensor.KinectSensors.Count - # of Kinects.
• kinectSensor.Status – Enumerator of
• Connected (it’s on and ready)
• Device Not Genuine (not a Kinect)
• Device Not Supported (Xbox 360 in Production mode)
• Disconnected (has been removed after being inited)
• Error (duh)
• Initializing (can be in this state for seconds)
• InsufficientBandwidth (USB Bus controller contention)
• Not Powered (5V power is good, 12V power problems)
• NotReady (Some component is still initing)

11. DEVELOPING WITH KINECT
WHEN

12. VIDEO FRAMES

13. SIMPLE VIDEO
Kinect can be used to simply get RGB Frames
Enable the stream
kinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
Start the Kinect -
kinectSensor.Start();
Set up your event handler (if doing events – if not you’ll likely drop frames) -
kinectSensor.ColorFrameReady += new
EventHandler<Microsoft.Kinect.ColorImageFrameReadyEventArgs>(kinect_VideoFrameReady);
Then the handler –
void kinect_VideoFrameReady(object sender,Microsoft.Kinect.ColorImageFrameReadyEventArgs
e)
{ using (ColorImageFrame image = e.OpenColorImageFrame())
{ if (image != null)
{ if (colorPixelData == null)
{ colorPixelData = new byte[image.PixelDataLength]; }
else
{ image.CopyPixelDataTo(colorPixelData);
BitmapSource source = BitmapSource.Create(image.Width,
image.Height, 96, 96, PixelFormats.Bgr32, null, colorPixelData, image.Width *
image.BytesPerPixel);
videoImage.Source = source;
...

14. DEPTH FRAMES

15. DEPTH FRAMES
Additionally you can use Depth frames to give you more information about your environment.
kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30);
kinectSensor.Start();
kinectSensor.DepthFrameReady += new
EventHandler<Microsoft.Kinect.DepthImageFrameReadyEventArgs>(kinect_DepthImageFrameReady
);
void kinect_DepthImageFrameReady(object sender,
Microsoft.Kinect.DepthImageFrameReadyEventArgs e)
{
using (DepthImageFrame imageFrame = e.OpenDepthImageFrame())
{
if (depthPixelData == null)
{
depthPixelData = new short[imageFrame.PixelDataLength];
}
if (imageFrame != null)
{
imageFrame.CopyPixelDataTo(this.depthPixelData);
…
But!
int depth = depthData[x + width * y] >> DepthImageFrame.PlayerIndexBitmaskWidth;

16. SKELETON TRACKING

17. SKELETON TRACKING
And one of the most interesting items is skeleton tracking –
(This should start looking familiar)
kinectSensor.SkeletonStream.Enable();
kinectSensor.Start();
kinectSensor.SkeletonFrameReady += new EventHandler<Microsoft.Kinect.SkeletonFrameReadyEventArgs>(kinect_SkeletonFrameReady);
void kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{ using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
{ if (skeletonFrame != null)
{ if ((this.skeletonData == null) || (this.skeletonData.Length != skeletonFrame.SkeletonArrayLength))
{ this.skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength]; }
skeletonFrame.CopySkeletonDataTo(this.skeletonData);
Skeleton thisSkeleton = null;
foreach (Skeleton skeleton in this.skeletonData)
{ if ((SkeletonTrackingState.Tracked == skeleton.TrackingState)
&& (thisSkeleton == null))
{ thisSkeleton = skeleton; }
}
if (thisSkeleton != null)
{
thisSkeleton.Position
}

18. SKELETON TRACKING
(CONT)
Key items for Skeleton Tracking –
SkeletonPoint – X, Y, Z
JointType enumeration – AnkleLeft, AnkleRight, ElbowLeft, ElbowRight,
FootLeft, FootRight, HandLeft, HandRight, Head, HipCenter,
HipLeft,HipRight,KneeLeft,KneeRight, ShoulderCenter, ShoulderLeft,
ShoulderRight,Spine, WristLeft, WristRight
JointTrackingState enumeration – Inferred, NotTracked, Tracked
These together tell you not just where joints are, but how confident the
system is. Even so – remember that these are always estimates – you’ll
need to be prepared to handle jittery data.

19. SPEECH
RECOGNITION
Building a grammar is very accessible and relatively pain
free.
(A bit more code than the others.)
Key items – it takes up to 4 seconds for the recognizer to be
ready.
Each recognition has a confidence level of 0-1.
Results are text and match text you send to a Grammar
Builder (“yes”, “no”, “launch”)
Multiple word commands are helpful for disambiguation, but
chained commands work much better. The recognition
engine can have nested grammars to help you with this.

20. DEVELOPING WITH KINECT
WHERE

21. LOCATION
• Human Interface
• Kinect skeleton tracking is optimized for full body imaging
and waist to head height camera position.
• Kinect 1.5 software update (est May 2012) is scheduled to
support sitting skeleton tracking.
• Speech Recognition
• Depth or Skeleton tracking can enable recognition
vectoring to increase confidence.
• Confidence level can be misleading if grammar items are
close together (false matching.)
• If possible, use multiple word commands for validation.
Build a command grammar and use it for error/confidence
checks.

22. LOCATION (CONT)
• Depth Frames
• Sunlight/IR can affect results.
• Items in depth frame have a shadow – be prepared for
interactions in those areas.
• Depth Frames are not required to be the same resolution
as associated Video frames.
• Environment
• Kinect is surprisingly robust for environment
• IR washout is the biggest factor
• Camera angle biggest factor for Skeleton Tracking.

23. FURTHER
INFORMATION
Kinect for Windows information:
http://www.KinectForWindows.com
Team blog:
http://blogs.msdn.com/b/kinectforwindows
Channel 9
http://channel9.msdn.com/Tags/kinect
And of course, our projects pages
http://ChaoticMoon.com/Labs !

24. Q&A
OR CONTACT ME AT:
@PWHEAT
PHILWHEAT.NET