第38回 名古屋CV・PRML勉強会 「Kinect v2本の紹介とPCLの概要」 http://nagoyacv.connpass.com/event/21139/

1. [PR] Kinect v2本の紹介とPCLの概要
Tsukasa Sugiura
@UnaNancyOwen

2. Self-Introduction
Tsukasa Sugiura
Microsoft MVP for Kinect for Windows
 @UnaNancyOwen
 http://UnaNancyOwen.com
 t.sugiura0204@gmail.com

3. Agenda
 Kinect v2の概要
 Kinect v2本の紹介
 Kinect v2本のプレゼント
 Kinect v2とPCL

4. Kinect for Windows v1 Sensor
MULTI-ARRAY MIC MOTORIZED TILT
3D DEPTH SENSORS
RGB CAMERA

5. Kinect for Windows v2 Sensor
MULTI-ARRAY MIC
3D DEPTH SENSOR
( IR Camera + IR Emitters )
RGB CAMERA

6. Kinect for Windows v2 Sensor
Image by iFixit
IR EMITTERS
IR CAMERA

7. Specifications
Kinect for Windows v1 Kinect for Windows v2
Color 640×480 @ 30fps 1920×1080 @ 30fps
Depth 320×240 @ 30fps 512×424 @ 30fps
Sensor Structured Light
(PrimeSense Light Coding)
Time of Flight
(ToF)
Range of View 0.8~4.0 m 0.5~8.0 m
Range of Detection 0.8~4.0 m 0.5~4.5 m
Angle of View
Horizontal / Vertical
57 / 43 degree 70 / 60 degree
Microphone Array ◯ ◯

8. Specifications
Kinect for Windows v1 Kinect for Windows v2
BodyIndex 6 people 6 people
Body 2 people 6 people
Joint 20 joint/people 25 joint/people
Hand State Open / Closed Open / Closed / Lasso
Gesture ☓ ◯
Face ◯
Speech / Beamforming ◯ ◯

9. System / Software Requirements
OS * Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM 4GB (or more)
GPU * DirectX 11 supported
USB * USB 3.0 (Intel or Renesas Host Controller)
Compiler * Visual Studio 2012, 2013 (Supported Express)
Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)
Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)

10. Basic Features
 Color
 1920×1080@30fps / 15fps (Lighting Condition)
 RGBA, YUV, BGRA, Bayer, YUY2

11. Basic Features
 Depth
 512×424@30fps
 500~8000[mm]
 ToF (Time of Flight)

12. Basic Features
 Infrared / LongExposureInfrared
 512×424@30fps
 16bit (higher 8 bits)

13. Basic Features
 BodyIndex
 512×424@30fps
 500~4500[mm]
 6 people
 Body Area : 0~5, Other Area : 255 (5 < Index)
255
0 1

14. Basic Features
 Body
 500~4500[mm]
 6 people
 25 joint / people (Add Tip, Thumb, Neck)
 Orientation (Quaternion)
 Hand Type (Right, Left),Hand State (Open, Closed, Lasso), Lean (-1.0f~1.0f)

15. Basic Features
 Audio
 Beamforming (+/-50 degree)
 Speaker Estimation
 Speech Recognition

16. Application Features
 Gesture
 Gesture Recognition using Machine Learning
 Discrete (detected true/false), Continuous (progress 0.0f~1.0f)
 Learning Classifier Tool “Visual Gesture Builder”
Video by http://youtu.be/-XYoblrnDpg

17. Application Features
 Face
 Bounding Box, Rotation, Points (Eye, Nose, Mouth Corner)
 Activity, Appearance, Expression
 Activity … Eye Closed, Mouth Open / Moved, Looking Away
 Appearance … Wearing Glasses
 Expression … Happy

18. Application Features
 HDFace
 For Creating 3D Face Model
 Points (1347), Triangles (2340), Hair Color, Skin Color
 Fitting Face Model

19. Application Features
 Other
 Kinect Fusion (3D Shape Reconstruction)
 Controls (Assist in implementation of NUI)

20. Kinect v2 Book
Kinect for Windows SDK プログラミング
Kinect for Windows v2センサー対応版
中村薫, 杉浦司, 高田智広, 上田智章
秀和システム
480ページ
3,400円+税
Publisher : http://www.shuwasystem.co.jp/products/7980html/4395.html
Sample : https://github.com/K4W2-Book/K4W2-Book
Reference : https://github.com/K4W2-Book/Docs

21. Contents
 Basic Streams
 Color, Depth, Infrared, BodyIndex, Body, Audio
 Application Futures
 Speech, Fusion, Gesture, Control, Face, HDFace

22. Contents
 NUI Tools
 Kinect Studio, Visual Gesture Builder, Configuration Verifier
 3rdParty Library, Tools, Math., Sensing
 Unity, Point Cloud Library, Vector

23. Review Copy

24. Point Cloud Library
 About PCL(Point Cloud Library)
 3次元点群のためのオープンソースのライブラリ
 様々な点群処理が含まれている
http://pointclouds.org/

25. Point Cloud Library
 PCL Features

26. Point Cloud Library
 PCL All-in-one Installer
 ビルド済みのPCLと依存ライブラリを一括でインストールする
 ただし、公式サイトでは旧い開発環境向けのパッケージしか配布していない
(PCL 1.6.0 All-in-one Installer MSVC 2008/2010)
Download : PCL 1.7.2 All-in-one Installer MSVC 2012/2013/2015
 Self-Build
 PCLと依存ライブラリをビルドする
Reference : Building PCL with Visual Studio

27. PCL All-in-one Installer
 Point Cloud Library
 PCL (1.7.2)
 Dependencies Library
 Boost (1.57.0) … C++準標準ライブラリ
 Eigen (3.2.4) … 行列ライブラリ
 FLANN (1.8.4) … 最近傍探索ライブラリ
 Qhull (2012.1) … 計算幾何ライブラリ
 VTK (6.2.0) … 可視化ライブラリ
 (OpenNI / OpenNI2) … RGB-Dセンサーライブラリ

28. PCL Grabber
 OpenNI/OpenNI2
 オープンソースのRGB-Dセンサーライブラリ
 ただし、OpenNIはVisual Studio 2010までしか利用できない
 PCL 1.7.2からOpenNI2がサポートされた
openni_grabber / openni2_grabber
 Kinect for Windows SDK v1/v2
 Microsoft公式のKinectセンサーのソフトウェア開発キット
 最新のVisual Studioで利用できる
kinect_grabber / kinect2_grabber

29. PCL Project
 Create Project
 Visaul C++のプロジェクトを作成、プロパティシートを読み込む
PCL.props
 CMakeでVisual C++のプロジェクトを自動生成する
CMakeLists.txt

30. PCL Project
 CMake
 CMakeLists.txt
cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
project(solution)
find_package(PCL 1.7 REQUIRED)
include_directories(${PCL_INCLUDE_DIRS})
include_directories($ENV{KINECTSDK20_DIR}/inc)
link_directories(${PCL_LIBRARY_DIRS})
link_directories($ENV{KINECTSDK20_DIR}/Lib/$(PlatformTarget))
add_definitions(${PCL_DEFINITIONS})
set(HEADER kinect2_grabber.h)
set(SOURCE main.cpp)
add_executable(project ${HEADER} ${SOURCE})
target_link_libraries(project ${PCL_LIBRARIES})
target_link_libraries(project Kinect20.lib)

31. Basic Usage Example
 Source Code
 main.cpp
#include "kinect2_grabber.h"
#include <pcl/visualization/pcl_visualizer.h>
typedef pcl::PointXYZRGB PointType;
int main( int argc, char* argv[] )
{
// PCL Visualizer
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(
new pcl::visualization::PCLVisualizer( "Point Cloud Viewer" ) );
// Point Cloud
pcl::PointCloud<PointType>::ConstPtr cloud;
// Retrieved Point Cloud Function
boost::mutex mutex;
boost::function<void( const pcl::PointCloud<PointType>::ConstPtr& )> function =
[&cloud, &mutex]( const pcl::PointCloud<PointType>::ConstPtr& ptr ){
boost::mutex::scoped_lock lock( mutex );
cloud = ptr;
};

32. Basic Usage Example
 Source Code
// Kinect2Grabber
pcl::Grabber* grabber = new pcl::Kinect2Grabber();
// Register Callback Function
boost::signals2::connection connection = grabber->registerCallback( function );
// Start Grabber
grabber->start();
while( !viewer->wasStopped() ){
// Update Viewer
viewer->spinOnce();
boost::mutex::scoped_try_lock lock( mutex );
if( cloud && lock.owns_lock() ){
if( cloud->size() != 0 ){
/* Processing Point Cloud */
// Update Point Cloud
if( !viewer->updatePointCloud( cloud, "cloud" ) ){
viewer->addPointCloud( cloud, "cloud" );
viewer->resetCameraViewpoint( "cloud" );
}
}
}
}

33. Basic Usage Example
 Source Code
// Stop Grabber
grabber->stop();
return 0;
}
 Kinect2Grabber Supported Point Types
 pcl::PointXYZRGB … 3次元位置(XYZ)と色(RGB)
 pcl::PointXYZ … 3次元位置(XYZ)

34. Down Sampling
 VoxcelGrid
#include <pcl/filters/voxel_grid.h>
// Create Object
pcl::VoxelGrid<PointType> grid;
// Setting
grid.setLeafSize( 0.01f, 0.01f, 0.01f );
// Input
grid.setInputCloud( *input );
// Processing
pcl::PointCloud<PointType>::Ptr output( new pcl::PointCloud<PointType> );
grid.filter( *output );
 Point CloudをVoxcel Gridフィルターでダウンサンプルする
 Voxcel Gridの間隔を0.01[m]に設定
http://pointclouds.org/documentation/tutorials/voxel_grid.php

35. Down Sampling
 Result

36. Segmentation
 SACSegmentation
#include <pcl/segmentation/sac_segmentation.h>
// Create Object
pcl::SACSegmentation<PointType> seg;
// Setting
seg.setOptimizeCoefficients( true );
seg.setModelType( pcl::SACMODEL_PLANE );
seg.setMethodType( pcl::SAC_RANSAC );
seg.setMaxIterations( 1000 );
seg.setDistanceThreshold( 0.01 );
// Input
seg.setInputCloud( *input );
// Processing
pcl::PointIndices::Ptr inliers( new pcl::PointIndices );
pcl::ModelCoefficients::Ptr coefficients( new pcl::ModelCoefficients );
seg.segment( *inliers, *coefficients );
 Point Cloudから平面(SACMODEL_PLANE)を検出する
 最大反復回数を1000回、距離閾値を0.01[m]に設定
http://pointclouds.org/documentation/tutorials/planar_segmentation.php

37. Segmentation
 Draw Segmentation Result
// Drawing Detection Area
pcl::PointCloud<PointType>::Ptr output( new pcl::PointCloud<PointType> );
output = input;
for( size_t i = 0; i < inliers->indices.size(); i++ ){
int index = inliers->indices[i];
output->points[index].r = 255;
output->points[index].g = 0;
output->points[index].b = 0;
}

38. Segmentation
 Result

39. Extracting
 ExtractIndices
#include <pcl/filters/extract_indices.h>
// Create Object
pcl::ExtractIndices<PointType> extract;
// Setting
extract.setIndices( inliers );
// Input
extract.setInputCloud( *input );
// Processing
pcl::PointCloud<PointType>::Ptr output_positive( new pcl::PointCloud<PointType> );
extract.setNegative( false );
extract.filter( *output_positive );
pcl::PointCloud<PointType>::Ptr output_negative( new pcl::PointCloud<PointType> );
Extract.setNegative( true );
extract.filter( *output_negative );
 Point Cloudから検出した平面を抽出する
 Positiveは検出領域、Negativeは非検出領域
http://pointclouds.org/documentation/tutorials/extract_indices.php

40. Extracting
 Result

41. Extracting
 Result

42. Tips
 Convert ConstPtr tp Ptr
// Convert pcl::PointCloud<T>::ConstPtr pcl::PointCloud<T>::Ptr
pcl::PointCloud<PointType>::ConstPtr ConstPtr( new pcl::PointCloud<PointType> );
pcl::PointCloud<PointType>::Ptr Ptr;
Ptr.reset( new pcl::PointCloud<PointType>( *ConstPtr ) );
 Save/Load Point Cloud
#include <pcl/io/pcd_io.h>
// Save Point Cloud
pcl::PointCloud<PointType>::Ptr cloud( new pcl::PointCloud<PointType> );
pcl::io::savePCDFileBinary( "pointcloud.pcd", *cloud );
// Load Point Cloud
pcl::PointCloud<PointType>::Ptr cloud( new pcl::PointCloud<PointType> );
pcl::io::loadPCDFile( "pointcloud.pcd", *cloud );

43. Tips
 CloudCompare
CloudCompare http://www.danielgm.net/cc/

44. Reference
 PCL Official
 Documentation - Tutorials
 Documentation - API Reference
 Users Forum
 Japanese
 DERiVE - PCLを触ってみよう！
 第13回 名古屋CV・PRML勉強会 - PCL (Point Cloud Library)
 PCL Development
 GitHub - PointCloudLibrary/pcl (issues/pull requests)

45. PCL Next All-in-one Installer
 Point Cloud Library
 PCL (1.8.0)
 Dependencies Library
 Boost (1.59.0) … C++準標準ライブラリ
 Eigen (3.2.6) … 行列ライブラリ
 FLANN (1.8.4) … 最近傍探索ライブラリ
 Qhull (2012.1) … 計算幾何ライブラリ
 VTK (6.3.0) … 可視化ライブラリ
 OpenNI2 … RGB-Dセンサーライブラリ
 RealSense … Intel RealSense SDK