SlideShare a Scribd company logo
1 of 43
Download to read offline
Kinect 2 Hands On
Luigi Oliveto
Researcher, Developer, IT Consultant
Email: luigi.oliveto@gmail.com
Twitter: @LuigiOliveto
LinkedIn: https://it.linkedin.com/in/luigioliveto
Agenda
• The Sensor
• System Requirements
• Architecture
• Data Sources
• Kinect Studio
• Gesture Recognition
2
The Sensor 3
Kinect 2 Sensor
Depth resolution: 512×424 pixels
RGB resolution: 1920×1080 pixels (16:9)
Frame rate: 30 FPS
Mic frequecy: 48 kHz
Range: from 0.5 to 4.5 m
4
USB hub
Power supply
3D DEPTH SENSOR
RGB CAMERA
MULTI-ARRAY MIC
Sensor
Kinect 1 VS Kinect 2 5
Feature Kinect for Windows 1 Kinect for Windows 2
Color Camera 640 x 480 @ 30 fps 1920 x 1080 @ 30 fps
Depth Camera 320 x 240 512 x 424
Max Depth Distance ~4.0 M ~4.5 M
Min Depth Distance 80cm (40 cm in near mode) 50 cm
Horizontal Field of View 57 degrees 70 degrees
Vertical Field of View 43 degrees 60 degrees
Tilt Motor yes no
Skeleton Joints Defined 20 joints 25 joints
Full Skeletons Tracked 2 6
USB Standard 2.0 3.0
Supported OS Win 7, Win 8 Win 8-8.1 (WSA)
Price (sensor + adapter) ~ €160 ~ €200
System Requirements 6
System Requirements
• Operating System
• Windows 8/8.1 (x64)
• Windows 8/8.1 Embedded Standard (x64)
• Hardware
• 64 bit processor (x64) i7 3.1Ghz (or higher)
• 4 GB memory (or more)
• Built-in USB 3.0 host controller
• DirectX11 capable graphics adapter:
ATI Radeon (HD 5400 series, HD 6570, HD
7800),
NVidia Quadro (600, K1000M), NVidia
GeForce (GT 640, GTX 660),
Intel HD 4400
• Kinect v2 sensor (with power supply and USB
hub)
• Software
• .NET Framework 4.5
• Visual Studio 2012 or higher
• Microsoft Speech Platform Software
Development Kit (Version 11)
• Kinect for Windows SDK
http://www.microsoft.com/en-
us/download/details.aspx?id=44561
• Applications
• Windows Presentation Foundation (WPF)
• Windows Store App
• Programming languages
• C++, C#, VB.NET, …
7
https://dev.windows.com/en-us/kinect
Architecture 8
Architecture (1)
• Multiple Kinect-enabled applications can run simultaneously
9
Architecture (2)
• The sensor is a resource  many applications can access it simultaneously
• The sensor gives a set of sources (functionalities)
• From every source it is possible to start readers
• Every reader gives events to acquire references to the device’s frames.
• From every frame it is possible to get data about the specific source (e.g. color
image, body data, etc…)
10
Sensor Sources Reader
Frame
Ref
Frame
Sensor
• Sensor usage
• Get an instance of KinectSensor
• Open the sensor
• Use the sensor
• Close the sensor
• In case of device unplug
• The KinectSensor instance remain valid
• No more frames are sent/received
• The sensor IsAvailable property become false
11
Sensor Sources Reader
Frame
Ref
Frame
Sources
• The sensor exhibit a source for every functionality
• Color source
• Depth source
• Infrared source
• Body Index source
• Body source (skeleton, hand tracking, lean…)
• Audio source
12
Sensor Sources Reader
Frame
Ref
Frame
Readers
• Give access to frames
• Events
• Polling
• Multiple readers can be created for each source
• Reader can be paused
13
Sensor Sources Reader
Frame
Ref
Frame
Frame References
• Access current frame through AcquireFrame() method
• Frame contains metadata (i.e., for the color: format, width, height)
• MUST be managed quickly and then released (if a frame is not released other frames
shouldn’t arrive)
14
Sensor Sources Reader
Frame
Ref
Frame
Frame
• Access frame data
• Access raw buffer directly
• Take a local copy
15
Sensor Sources Reader
Frame
Ref
Frame
MultiSourceFrameReader
• Allows to get a matched set of frames from multiple sources on a single event
• Delivers frames at the lowest FPS of the selected sources
16
MultiSourceFrameReader MultiReader =
Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
FrameSourceTypes.BodyIndex |
FrameSourceTypes.Body);
var frame = args.FrameReference.AcquireFrame();
if (frame != null) {
using (colorFrame = frame.ColorFrameReference.AcquireFrame())
using (bodyFrame = frame.BodyFrameReference.AcquireFrame())
using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame())
{
//
}
}
Demo
Getting Started with Kinect 2 SDK
17
Data Sources 18
Kinect Data Sources – Color
• 1920 x 1080 array of color pixels
• 30 or 15 fps, based on lighting conditions
• Elaborated Image Format:
• RGBA, BGRA, YUY2, …
• Raw Format: YUY2
• Frame data can be:
• Used in raw format
• Converted to other formats (with a
computational cost)
• The Buffer is a byte array.
• The number of bytes per pixel depends on
raw format (now is 4 bytes per pixel).
19
Kinect Data Sources – Infrared
• 512 x 424 pixel @ 30 fps
• Same physical sensor of the depth source
• Two sources:
• Infrared: single infrared frame
• LongExposureInfrared: overlapping of 3
frames
(better ratio signal/noise but images with
blurry effect)
• Every pixel is composed by 2 byte (16-bit)
and represent the IR intensity value
• Ambient light removed: the SDK get only
the reflection of the infrared light,
projected by the device
20
Kinect Data Sources – Depth
• 512 x 424 pixel @ 30 fps
• Range: 0.5 – 4.5 meters (Extended Depth to
8m)
• Every pixel is composed by 2 byte (16-bit)
and contain the distance in millimeters from
the sensor’s focal plane
• Player index not present
21
Demo
Color, Infrared and Depth sources
22
Kinect Data Sources – Body Index
• 512 x 424 @ 30 fps
• Every pixel is composed by 1 byte
• Pixel Data
• 0 to 5: Index of the corresponding body,
as tracked by the body source
• > 5: No tracked body at that pixel
23
Kinect Data Sources – Body
• Range is 0.5-4.5 meters
• 30fps
• Frame data is a collection of Body objects
• Each body has
• 25 joints (each joint has position in 3D space
and orientation)
• Hand tracking (open, close, “lazo”)
• Face tracking and expressions
• Bones’ orientation
• Up to 6 simultaneous bodies
• Hand State on 2 bodies
24
Body information
• The Body class contains useful properties:
• ClippedEdges: edges of the Field of View that clip the body
• HandState [Left/Right]: { Unknown, NotTracked, Closed, Open, Lasso }
• HandConfidence [Left/Right]: { High, Low }
• IsRestricted
• IsTracked
• TrackingId: 64-bit unique id
• Joints: position in the space of each joint
• JointOrientations: orientation in the space of the articulation
• Lean: inclination vector of the body
• LeanTrackingState: { Inferred, NotTracked, Tracked }
• Up to 6 bodies simultaneously
• Up to 2 players’ hands simultaneously
25
Skeleton VS Body 26
Kinect 1 Kinect 2
Demo
Body source
27
Kinect Data Sources – Audio
• Frame data is an Audio Beam
• Readers and event as previous sources
• Acquire frames through AcquireBeamFrames() method
28
Coordinate System
• ColorSpace (Coordinate System of the Color Image)
• … Color
• DepthSpace (Coordinate System of the Depth Data)
• … Depth, Infrared, BodyIndex
• CameraSpace (Coordinate System with the origin located to the sensor)
• … Body (Joint)
29
Coordinate Mapper
• Three coordinate systems
• Coordinate mapper provides conversions between each system
• Convert single or multiple points
30
Name Applies to Dimensions Units Range Origin
ColorSpacePoint Color 2 pixels 1920x1080 Top left corner
DepthSpacePoint Depth,
Infrared,
Body index
2 pixels 512x424 Top left corner
CameraSpacePoint Body 3 meters – Infrared/depth
camera
Interaction 31
Hand Pointer Gestures 32
Engagement Targeting Press/Pan/Zoom
Hand Pointer States 33
Kinect Region & User Controls
• The KinectRegion user control define a part of the user interface (XAML) where the
user can interact with an hand pointer
• The region must be connected to the sensor instance
• Available gestures (“out-of-the-box”) usable into a KinectRegion:
• Click
• Grab
• Pan
• Zoom
• KinectUserViewer gives a visual feedback related to the tracked state of the users
• Re-use default user controls
34
Demo
User Interaction
35
Kinect Studio 36
Recording, Playback, and Gesture Recognition 37
Recordable Data Sources 38
Infrared
13 MB/s
Depth
13 MB/s
BodyFrame
BodyIndex
Color
120 MB/s
Audio
32 KB/s
Legend
Record/Play
Record Only
Gesture Recognition 39
Gesture Recognition 40
• Gesture is a coding problem
• Quick to do simple gestures/poses
(hand over head)
• ML can also be useful to find good
signals for Heuristic approach
• Gesture is a data problem
• Signals which may not be easily
human understandable (progress in a
baseball swing)
• Large investment for production
• Danger of over-fitting, causes you to
be too specific – eliminating
recognition of generic cases
Heuristic Machine Learning (ML) with G.B.
Visual Gesture Builder (1)
• New tool integrated with v2 SDK
• Organize data using projects and solutions
• Give meaning to data by tagging gestures
• Build gestures using machine learning technology
• Adaptive Boosting (AdaBoost) Trigger
• Determines if player is performing gesture
• Random Forest Regression (RFR) Progress
• Determines the progress of the gesture performed by player
• Analyze / test the results of gesture detection
• Live preview of results
41
Visual Gesture Builder (2) 42
Your Application
Resources
• General Info & Blog  https://dev.windows.com/en-us/Kinect
• Purchase Sensor  http://goo.gl/ZsMtBx
• Developer Forums  https://goo.gl/bpptyq
• Twitter Account  @KinectWindows
• A Facebook Group  http://on.fb.me/1LSflbX
• A LinkedIn Group  http://linkd.in/1J9gFcY
• A Twitter Account  @KinectDevelop
• A Google Plus Page  http://bit.ly/1SHtduT
43

More Related Content

What's hot

Kinect Sensors as Natural User Interfaces
Kinect Sensors as Natural User InterfacesKinect Sensors as Natural User Interfaces
Kinect Sensors as Natural User InterfacesRumen Filkov
 
How Augment your Reality: Different perspective on the Reality / Virtuality C...
How Augment your Reality: Different perspective on the Reality / Virtuality C...How Augment your Reality: Different perspective on the Reality / Virtuality C...
How Augment your Reality: Different perspective on the Reality / Virtuality C...Matteo Valoriani
 
Kinect Hacks for Dummies
Kinect Hacks for DummiesKinect Hacks for Dummies
Kinect Hacks for DummiesTomoto Washio
 
Kinect v1+Processing workshot fabcafe_taipei
Kinect v1+Processing workshot fabcafe_taipeiKinect v1+Processing workshot fabcafe_taipei
Kinect v1+Processing workshot fabcafe_taipeiMao Wu
 
Odessa .NET User Group - Kinect v2
Odessa .NET User Group - Kinect v2Odessa .NET User Group - Kinect v2
Odessa .NET User Group - Kinect v2Dmytro Mindra
 
3 track kinect@Bicocca - sdk e camere
3   track kinect@Bicocca - sdk e camere3   track kinect@Bicocca - sdk e camere
3 track kinect@Bicocca - sdk e camereMatteo Valoriani
 
Develop store apps with kinect for windows v2
Develop store apps with kinect for windows v2Develop store apps with kinect for windows v2
Develop store apps with kinect for windows v2Matteo Valoriani
 
Programming with RealSense using .NET
Programming with RealSense using .NETProgramming with RealSense using .NET
Programming with RealSense using .NETMatteo Valoriani
 
2 track kinect@Bicocca - hardware e funzinamento
2   track kinect@Bicocca - hardware e funzinamento2   track kinect@Bicocca - hardware e funzinamento
2 track kinect@Bicocca - hardware e funzinamentoMatteo Valoriani
 
Kinectic vision looking deep into depth
Kinectic vision   looking deep into depthKinectic vision   looking deep into depth
Kinectic vision looking deep into depthppd1961
 
Introduction to development
Introduction to developmentIntroduction to development
Introduction to developmentMatteo Valoriani
 
The power of Kinect in 10 minutes
The power of Kinect in 10 minutesThe power of Kinect in 10 minutes
The power of Kinect in 10 minutesTom Kerkhove
 
Introduction to Kinect v2
Introduction to Kinect v2Introduction to Kinect v2
Introduction to Kinect v2Tsukasa Sugiura
 
Kinect for Xbox 360: the world's first viral 3D technology
Kinect for Xbox 360: the world's first viral 3D technologyKinect for Xbox 360: the world's first viral 3D technology
Kinect for Xbox 360: the world's first viral 3D technologykamutef
 
Final_draft_Practice_School_II_report
Final_draft_Practice_School_II_reportFinal_draft_Practice_School_II_report
Final_draft_Practice_School_II_reportRishikesh Bagwe
 
“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...
“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...
“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...Edge AI and Vision Alliance
 
Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3
Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3
Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3Umbra
 

What's hot (20)

Kinect Sensors as Natural User Interfaces
Kinect Sensors as Natural User InterfacesKinect Sensors as Natural User Interfaces
Kinect Sensors as Natural User Interfaces
 
How Augment your Reality: Different perspective on the Reality / Virtuality C...
How Augment your Reality: Different perspective on the Reality / Virtuality C...How Augment your Reality: Different perspective on the Reality / Virtuality C...
How Augment your Reality: Different perspective on the Reality / Virtuality C...
 
Kinect Hacks for Dummies
Kinect Hacks for DummiesKinect Hacks for Dummies
Kinect Hacks for Dummies
 
Kinect v1+Processing workshot fabcafe_taipei
Kinect v1+Processing workshot fabcafe_taipeiKinect v1+Processing workshot fabcafe_taipei
Kinect v1+Processing workshot fabcafe_taipei
 
Odessa .NET User Group - Kinect v2
Odessa .NET User Group - Kinect v2Odessa .NET User Group - Kinect v2
Odessa .NET User Group - Kinect v2
 
3 track kinect@Bicocca - sdk e camere
3   track kinect@Bicocca - sdk e camere3   track kinect@Bicocca - sdk e camere
3 track kinect@Bicocca - sdk e camere
 
Develop store apps with kinect for windows v2
Develop store apps with kinect for windows v2Develop store apps with kinect for windows v2
Develop store apps with kinect for windows v2
 
Programming with RealSense using .NET
Programming with RealSense using .NETProgramming with RealSense using .NET
Programming with RealSense using .NET
 
2 track kinect@Bicocca - hardware e funzinamento
2   track kinect@Bicocca - hardware e funzinamento2   track kinect@Bicocca - hardware e funzinamento
2 track kinect@Bicocca - hardware e funzinamento
 
Kinect connect
Kinect connectKinect connect
Kinect connect
 
Kinectic vision looking deep into depth
Kinectic vision   looking deep into depthKinectic vision   looking deep into depth
Kinectic vision looking deep into depth
 
Introduction to development
Introduction to developmentIntroduction to development
Introduction to development
 
The power of Kinect in 10 minutes
The power of Kinect in 10 minutesThe power of Kinect in 10 minutes
The power of Kinect in 10 minutes
 
Introduction to Kinect v2
Introduction to Kinect v2Introduction to Kinect v2
Introduction to Kinect v2
 
Getmoving as3kinect
Getmoving as3kinectGetmoving as3kinect
Getmoving as3kinect
 
Kinect for Xbox 360: the world's first viral 3D technology
Kinect for Xbox 360: the world's first viral 3D technologyKinect for Xbox 360: the world's first viral 3D technology
Kinect for Xbox 360: the world's first viral 3D technology
 
Final_draft_Practice_School_II_report
Final_draft_Practice_School_II_reportFinal_draft_Practice_School_II_report
Final_draft_Practice_School_II_report
 
Communitydays2015
Communitydays2015Communitydays2015
Communitydays2015
 
“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...
“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...
“Case Study: Facial Detection and Recognition for Always-On Applications,” a ...
 
Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3
Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3
Solving Visibility and Streaming in the The Witcher 3: Wild Hunt with Umbra 3
 

Viewers also liked

Kinect 2.0 Programming (1)
Kinect 2.0 Programming (1)Kinect 2.0 Programming (1)
Kinect 2.0 Programming (1)IngChyuan Wu
 
Kinect V2: NUI for dummies!!
Kinect V2: NUI for dummies!!Kinect V2: NUI for dummies!!
Kinect V2: NUI for dummies!!Massimo Bonanni
 
MSDN LATAM Kinect SDK V2
MSDN LATAM Kinect SDK V2MSDN LATAM Kinect SDK V2
MSDN LATAM Kinect SDK V2Bruno Capuano
 
Kinect演習#01課題編
Kinect演習#01課題編Kinect演習#01課題編
Kinect演習#01課題編Akihiko Shirai
 
Visug: Say Hello to my little friend: a session on Kinect
Visug: Say Hello to my little friend: a session on KinectVisug: Say Hello to my little friend: a session on Kinect
Visug: Say Hello to my little friend: a session on KinectVisug
 
Unity and WebSockets
Unity and WebSocketsUnity and WebSockets
Unity and WebSocketsJosh Glover
 
Isadora Duncan - Mother of Modern Dance
Isadora Duncan - Mother of Modern DanceIsadora Duncan - Mother of Modern Dance
Isadora Duncan - Mother of Modern DanceErin Ford
 
Isadora pitch
Isadora pitchIsadora pitch
Isadora pitchJD8307
 
Overview of The Virtual Dressing Room Market - Augmented World Expo
Overview of The Virtual Dressing Room Market - Augmented World ExpoOverview of The Virtual Dressing Room Market - Augmented World Expo
Overview of The Virtual Dressing Room Market - Augmented World ExpoMatthew Szymczyk
 
Modern dance philosophy
Modern dance philosophyModern dance philosophy
Modern dance philosophyIvana Rankovic
 
Artificial Intelligence Presentation
Artificial Intelligence PresentationArtificial Intelligence Presentation
Artificial Intelligence Presentationlpaviglianiti
 
Artificial Intelligence
Artificial IntelligenceArtificial Intelligence
Artificial Intelligenceu053675
 

Viewers also liked (12)

Kinect 2.0 Programming (1)
Kinect 2.0 Programming (1)Kinect 2.0 Programming (1)
Kinect 2.0 Programming (1)
 
Kinect V2: NUI for dummies!!
Kinect V2: NUI for dummies!!Kinect V2: NUI for dummies!!
Kinect V2: NUI for dummies!!
 
MSDN LATAM Kinect SDK V2
MSDN LATAM Kinect SDK V2MSDN LATAM Kinect SDK V2
MSDN LATAM Kinect SDK V2
 
Kinect演習#01課題編
Kinect演習#01課題編Kinect演習#01課題編
Kinect演習#01課題編
 
Visug: Say Hello to my little friend: a session on Kinect
Visug: Say Hello to my little friend: a session on KinectVisug: Say Hello to my little friend: a session on Kinect
Visug: Say Hello to my little friend: a session on Kinect
 
Unity and WebSockets
Unity and WebSocketsUnity and WebSockets
Unity and WebSockets
 
Isadora Duncan - Mother of Modern Dance
Isadora Duncan - Mother of Modern DanceIsadora Duncan - Mother of Modern Dance
Isadora Duncan - Mother of Modern Dance
 
Isadora pitch
Isadora pitchIsadora pitch
Isadora pitch
 
Overview of The Virtual Dressing Room Market - Augmented World Expo
Overview of The Virtual Dressing Room Market - Augmented World ExpoOverview of The Virtual Dressing Room Market - Augmented World Expo
Overview of The Virtual Dressing Room Market - Augmented World Expo
 
Modern dance philosophy
Modern dance philosophyModern dance philosophy
Modern dance philosophy
 
Artificial Intelligence Presentation
Artificial Intelligence PresentationArtificial Intelligence Presentation
Artificial Intelligence Presentation
 
Artificial Intelligence
Artificial IntelligenceArtificial Intelligence
Artificial Intelligence
 

Similar to Kinect2 hands on

Kinect krishna kumar-itkan
Kinect krishna kumar-itkanKinect krishna kumar-itkan
Kinect krishna kumar-itkanPat Maher
 
Kinect krishna kumar-itkan
Kinect krishna kumar-itkanKinect krishna kumar-itkan
Kinect krishna kumar-itkanPat Maher
 
Developing For Kinect For Windows
Developing For Kinect For WindowsDeveloping For Kinect For Windows
Developing For Kinect For WindowsPrashant Tiwari
 
Xbox 360 Kinect
Xbox 360 Kinect  Xbox 360 Kinect
Xbox 360 Kinect Saif Pathan
 
Community Day 2013 - The Power of Kinect
Community Day 2013 - The Power of KinectCommunity Day 2013 - The Power of Kinect
Community Day 2013 - The Power of KinectTom Kerkhove
 
Gam02 kinect1, kinect2
Gam02   kinect1, kinect2Gam02   kinect1, kinect2
Gam02 kinect1, kinect2DotNetCampus
 
Becoming a kinect hacker innovator v2
Becoming a kinect hacker innovator v2Becoming a kinect hacker innovator v2
Becoming a kinect hacker innovator v2Jeff Sipko
 
PyKinect: Body Iteration Application Development Using Python
PyKinect: Body Iteration Application Development Using PythonPyKinect: Body Iteration Application Development Using Python
PyKinect: Body Iteration Application Development Using Pythonpycontw
 
Intro+Imaging.ppt
Intro+Imaging.pptIntro+Imaging.ppt
Intro+Imaging.pptshohel rana
 
Writing applications using the Microsoft Kinect Sensor
Writing applications using the Microsoft Kinect SensorWriting applications using the Microsoft Kinect Sensor
Writing applications using the Microsoft Kinect Sensorphildenoncourt
 
Deep Learning AtoC with Image Perspective
Deep Learning AtoC with Image PerspectiveDeep Learning AtoC with Image Perspective
Deep Learning AtoC with Image PerspectiveDong Heon Cho
 
Lidnug Presentation - Kinect - The How, Were and When of developing with it
Lidnug Presentation - Kinect - The How, Were and When of developing with itLidnug Presentation - Kinect - The How, Were and When of developing with it
Lidnug Presentation - Kinect - The How, Were and When of developing with itPhilip Wheat
 
HMM based Automatic Arabic Sign Language Translator using
HMM based Automatic Arabic Sign Language Translator usingHMM based Automatic Arabic Sign Language Translator using
HMM based Automatic Arabic Sign Language Translator usingعمر أمين
 
Mobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research DirectionsMobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research DirectionsMark Billinghurst
 

Similar to Kinect2 hands on (20)

Kinect krishna kumar-itkan
Kinect krishna kumar-itkanKinect krishna kumar-itkan
Kinect krishna kumar-itkan
 
Kinect krishna kumar-itkan
Kinect krishna kumar-itkanKinect krishna kumar-itkan
Kinect krishna kumar-itkan
 
Kinect sensor
Kinect sensorKinect sensor
Kinect sensor
 
Developing For Kinect For Windows
Developing For Kinect For WindowsDeveloping For Kinect For Windows
Developing For Kinect For Windows
 
Xbox 360 Kinect
Xbox 360 Kinect  Xbox 360 Kinect
Xbox 360 Kinect
 
Community Day 2013 - The Power of Kinect
Community Day 2013 - The Power of KinectCommunity Day 2013 - The Power of Kinect
Community Day 2013 - The Power of Kinect
 
Gam02 kinect1, kinect2
Gam02   kinect1, kinect2Gam02   kinect1, kinect2
Gam02 kinect1, kinect2
 
Becoming a kinect hacker innovator v2
Becoming a kinect hacker innovator v2Becoming a kinect hacker innovator v2
Becoming a kinect hacker innovator v2
 
PyKinect: Body Iteration Application Development Using Python
PyKinect: Body Iteration Application Development Using PythonPyKinect: Body Iteration Application Development Using Python
PyKinect: Body Iteration Application Development Using Python
 
3D scanner using kinect
3D scanner using kinect3D scanner using kinect
3D scanner using kinect
 
Intro+Imaging.ppt
Intro+Imaging.pptIntro+Imaging.ppt
Intro+Imaging.ppt
 
Kinect
KinectKinect
Kinect
 
Writing applications using the Microsoft Kinect Sensor
Writing applications using the Microsoft Kinect SensorWriting applications using the Microsoft Kinect Sensor
Writing applications using the Microsoft Kinect Sensor
 
Kinect
KinectKinect
Kinect
 
Cocos2d programming
Cocos2d programmingCocos2d programming
Cocos2d programming
 
Deep Learning AtoC with Image Perspective
Deep Learning AtoC with Image PerspectiveDeep Learning AtoC with Image Perspective
Deep Learning AtoC with Image Perspective
 
You suck at Memory Analysis
You suck at Memory AnalysisYou suck at Memory Analysis
You suck at Memory Analysis
 
Lidnug Presentation - Kinect - The How, Were and When of developing with it
Lidnug Presentation - Kinect - The How, Were and When of developing with itLidnug Presentation - Kinect - The How, Were and When of developing with it
Lidnug Presentation - Kinect - The How, Were and When of developing with it
 
HMM based Automatic Arabic Sign Language Translator using
HMM based Automatic Arabic Sign Language Translator usingHMM based Automatic Arabic Sign Language Translator using
HMM based Automatic Arabic Sign Language Translator using
 
Mobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research DirectionsMobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research Directions
 

Recently uploaded

Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 

Recently uploaded (20)

Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 

Kinect2 hands on

  • 1. Kinect 2 Hands On Luigi Oliveto Researcher, Developer, IT Consultant Email: luigi.oliveto@gmail.com Twitter: @LuigiOliveto LinkedIn: https://it.linkedin.com/in/luigioliveto
  • 2. Agenda • The Sensor • System Requirements • Architecture • Data Sources • Kinect Studio • Gesture Recognition 2
  • 4. Kinect 2 Sensor Depth resolution: 512×424 pixels RGB resolution: 1920×1080 pixels (16:9) Frame rate: 30 FPS Mic frequecy: 48 kHz Range: from 0.5 to 4.5 m 4 USB hub Power supply 3D DEPTH SENSOR RGB CAMERA MULTI-ARRAY MIC Sensor
  • 5. Kinect 1 VS Kinect 2 5 Feature Kinect for Windows 1 Kinect for Windows 2 Color Camera 640 x 480 @ 30 fps 1920 x 1080 @ 30 fps Depth Camera 320 x 240 512 x 424 Max Depth Distance ~4.0 M ~4.5 M Min Depth Distance 80cm (40 cm in near mode) 50 cm Horizontal Field of View 57 degrees 70 degrees Vertical Field of View 43 degrees 60 degrees Tilt Motor yes no Skeleton Joints Defined 20 joints 25 joints Full Skeletons Tracked 2 6 USB Standard 2.0 3.0 Supported OS Win 7, Win 8 Win 8-8.1 (WSA) Price (sensor + adapter) ~ €160 ~ €200
  • 7. System Requirements • Operating System • Windows 8/8.1 (x64) • Windows 8/8.1 Embedded Standard (x64) • Hardware • 64 bit processor (x64) i7 3.1Ghz (or higher) • 4 GB memory (or more) • Built-in USB 3.0 host controller • DirectX11 capable graphics adapter: ATI Radeon (HD 5400 series, HD 6570, HD 7800), NVidia Quadro (600, K1000M), NVidia GeForce (GT 640, GTX 660), Intel HD 4400 • Kinect v2 sensor (with power supply and USB hub) • Software • .NET Framework 4.5 • Visual Studio 2012 or higher • Microsoft Speech Platform Software Development Kit (Version 11) • Kinect for Windows SDK http://www.microsoft.com/en- us/download/details.aspx?id=44561 • Applications • Windows Presentation Foundation (WPF) • Windows Store App • Programming languages • C++, C#, VB.NET, … 7 https://dev.windows.com/en-us/kinect
  • 9. Architecture (1) • Multiple Kinect-enabled applications can run simultaneously 9
  • 10. Architecture (2) • The sensor is a resource  many applications can access it simultaneously • The sensor gives a set of sources (functionalities) • From every source it is possible to start readers • Every reader gives events to acquire references to the device’s frames. • From every frame it is possible to get data about the specific source (e.g. color image, body data, etc…) 10 Sensor Sources Reader Frame Ref Frame
  • 11. Sensor • Sensor usage • Get an instance of KinectSensor • Open the sensor • Use the sensor • Close the sensor • In case of device unplug • The KinectSensor instance remain valid • No more frames are sent/received • The sensor IsAvailable property become false 11 Sensor Sources Reader Frame Ref Frame
  • 12. Sources • The sensor exhibit a source for every functionality • Color source • Depth source • Infrared source • Body Index source • Body source (skeleton, hand tracking, lean…) • Audio source 12 Sensor Sources Reader Frame Ref Frame
  • 13. Readers • Give access to frames • Events • Polling • Multiple readers can be created for each source • Reader can be paused 13 Sensor Sources Reader Frame Ref Frame
  • 14. Frame References • Access current frame through AcquireFrame() method • Frame contains metadata (i.e., for the color: format, width, height) • MUST be managed quickly and then released (if a frame is not released other frames shouldn’t arrive) 14 Sensor Sources Reader Frame Ref Frame
  • 15. Frame • Access frame data • Access raw buffer directly • Take a local copy 15 Sensor Sources Reader Frame Ref Frame
  • 16. MultiSourceFrameReader • Allows to get a matched set of frames from multiple sources on a single event • Delivers frames at the lowest FPS of the selected sources 16 MultiSourceFrameReader MultiReader = Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color | FrameSourceTypes.BodyIndex | FrameSourceTypes.Body); var frame = args.FrameReference.AcquireFrame(); if (frame != null) { using (colorFrame = frame.ColorFrameReference.AcquireFrame()) using (bodyFrame = frame.BodyFrameReference.AcquireFrame()) using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()) { // } }
  • 17. Demo Getting Started with Kinect 2 SDK 17
  • 19. Kinect Data Sources – Color • 1920 x 1080 array of color pixels • 30 or 15 fps, based on lighting conditions • Elaborated Image Format: • RGBA, BGRA, YUY2, … • Raw Format: YUY2 • Frame data can be: • Used in raw format • Converted to other formats (with a computational cost) • The Buffer is a byte array. • The number of bytes per pixel depends on raw format (now is 4 bytes per pixel). 19
  • 20. Kinect Data Sources – Infrared • 512 x 424 pixel @ 30 fps • Same physical sensor of the depth source • Two sources: • Infrared: single infrared frame • LongExposureInfrared: overlapping of 3 frames (better ratio signal/noise but images with blurry effect) • Every pixel is composed by 2 byte (16-bit) and represent the IR intensity value • Ambient light removed: the SDK get only the reflection of the infrared light, projected by the device 20
  • 21. Kinect Data Sources – Depth • 512 x 424 pixel @ 30 fps • Range: 0.5 – 4.5 meters (Extended Depth to 8m) • Every pixel is composed by 2 byte (16-bit) and contain the distance in millimeters from the sensor’s focal plane • Player index not present 21
  • 22. Demo Color, Infrared and Depth sources 22
  • 23. Kinect Data Sources – Body Index • 512 x 424 @ 30 fps • Every pixel is composed by 1 byte • Pixel Data • 0 to 5: Index of the corresponding body, as tracked by the body source • > 5: No tracked body at that pixel 23
  • 24. Kinect Data Sources – Body • Range is 0.5-4.5 meters • 30fps • Frame data is a collection of Body objects • Each body has • 25 joints (each joint has position in 3D space and orientation) • Hand tracking (open, close, “lazo”) • Face tracking and expressions • Bones’ orientation • Up to 6 simultaneous bodies • Hand State on 2 bodies 24
  • 25. Body information • The Body class contains useful properties: • ClippedEdges: edges of the Field of View that clip the body • HandState [Left/Right]: { Unknown, NotTracked, Closed, Open, Lasso } • HandConfidence [Left/Right]: { High, Low } • IsRestricted • IsTracked • TrackingId: 64-bit unique id • Joints: position in the space of each joint • JointOrientations: orientation in the space of the articulation • Lean: inclination vector of the body • LeanTrackingState: { Inferred, NotTracked, Tracked } • Up to 6 bodies simultaneously • Up to 2 players’ hands simultaneously 25
  • 26. Skeleton VS Body 26 Kinect 1 Kinect 2
  • 28. Kinect Data Sources – Audio • Frame data is an Audio Beam • Readers and event as previous sources • Acquire frames through AcquireBeamFrames() method 28
  • 29. Coordinate System • ColorSpace (Coordinate System of the Color Image) • … Color • DepthSpace (Coordinate System of the Depth Data) • … Depth, Infrared, BodyIndex • CameraSpace (Coordinate System with the origin located to the sensor) • … Body (Joint) 29
  • 30. Coordinate Mapper • Three coordinate systems • Coordinate mapper provides conversions between each system • Convert single or multiple points 30 Name Applies to Dimensions Units Range Origin ColorSpacePoint Color 2 pixels 1920x1080 Top left corner DepthSpacePoint Depth, Infrared, Body index 2 pixels 512x424 Top left corner CameraSpacePoint Body 3 meters – Infrared/depth camera
  • 32. Hand Pointer Gestures 32 Engagement Targeting Press/Pan/Zoom
  • 34. Kinect Region & User Controls • The KinectRegion user control define a part of the user interface (XAML) where the user can interact with an hand pointer • The region must be connected to the sensor instance • Available gestures (“out-of-the-box”) usable into a KinectRegion: • Click • Grab • Pan • Zoom • KinectUserViewer gives a visual feedback related to the tracked state of the users • Re-use default user controls 34
  • 37. Recording, Playback, and Gesture Recognition 37
  • 38. Recordable Data Sources 38 Infrared 13 MB/s Depth 13 MB/s BodyFrame BodyIndex Color 120 MB/s Audio 32 KB/s Legend Record/Play Record Only
  • 40. Gesture Recognition 40 • Gesture is a coding problem • Quick to do simple gestures/poses (hand over head) • ML can also be useful to find good signals for Heuristic approach • Gesture is a data problem • Signals which may not be easily human understandable (progress in a baseball swing) • Large investment for production • Danger of over-fitting, causes you to be too specific – eliminating recognition of generic cases Heuristic Machine Learning (ML) with G.B.
  • 41. Visual Gesture Builder (1) • New tool integrated with v2 SDK • Organize data using projects and solutions • Give meaning to data by tagging gestures • Build gestures using machine learning technology • Adaptive Boosting (AdaBoost) Trigger • Determines if player is performing gesture • Random Forest Regression (RFR) Progress • Determines the progress of the gesture performed by player • Analyze / test the results of gesture detection • Live preview of results 41
  • 42. Visual Gesture Builder (2) 42 Your Application
  • 43. Resources • General Info & Blog  https://dev.windows.com/en-us/Kinect • Purchase Sensor  http://goo.gl/ZsMtBx • Developer Forums  https://goo.gl/bpptyq • Twitter Account  @KinectWindows • A Facebook Group  http://on.fb.me/1LSflbX • A LinkedIn Group  http://linkd.in/1J9gFcY • A Twitter Account  @KinectDevelop • A Google Plus Page  http://bit.ly/1SHtduT 43

Editor's Notes

  1. Improved body tracking The enhanced fidelity of the depth camera, combined with improvements in the software, have led to a number of body tracking developments. The latest sensor tracks as many as six complete skeletons (compared to two with the original sensor), and 25 joints per person (compared to 20 with the original sensor). The tracked positions are more anatomically correct and stable and the range of tracking is broader. In interactive scenarios, your avatars will be more stable—with more accurate body position evaluation and crisper interactions—and you have the potential for bystanders to participate.
  2. Ability to interact with your application Includes control to visualize user Re-use default XAML controls
  3. Ability to interact with your application Includes control to visualize user Re-use default XAML controls