機器學習應用於蔬果辨識，佈署於樹莓派並使用Intel Movidius NCS提升運算效能。 2018 11 28 人工智慧學校 台中分校

1. 機器學習
應用於蔬果偵測
Kobe Yu

2. 投影片授權聲明
投影片內容採 CC BY-SA-NC 3.0TW(姓名標示-相同方式分享-非商業性 3.0 台灣)。
完整授權內容 : https://creativecommons.org/licenses/by-nc-sa/3.0/tw/

3. About Me
● 清大資工博士班學生
○ 從系統軟體的角度提升電腦視覺與類神經網路的運算效能。
● 8+軟體開發經驗
○ 電腦視覺/多媒體串流/iOS App開發/攝影機模組驅動與調校。
● 開源專案
○ Farmharvest bot (農場採收機器人) FB: FarmBot Taiwan User Group-FBTUG
○ Agrino (農業感測) http://bit.ly/2MDaAdy

4. Outline
● 農場採果機器人(FarmHarvestbot)專案簡介
● ML開發流程以蔬果辨識為例
● Demo
● Q&A

5. FarmBot Taiwan User Group-FBTUG
● 專案緣起
○ 希望藉由開源農業機器的推廣與貢獻，給友善農業一點幫助。
● 社群成員
○ 發起人： 哈爸
○ 成員：~2400 (各領域專家或有興趣的大眾 )
○ 與採果專案有關的幾位成員
■ 志弘大哥 硬體與CNC專家
■ CH大 ML(YOLO) geek
■ Jack AR/VR、電腦視覺專家
■ Eric 埔里小農(一點都不小的小農)
■ Olin Dataset協調人
■ …. 默默協助的伙伴們

6. FarmHarvestbot
● 農場採果機器人
○ 開源專案
■ 軟體/硬體/機構/資料集
○ Why
■ 農業缺工問題
■ 採收時效性
■ 知識與經驗共享
○ How
■ Core/Support群組
■ 開放討論
FB: FarmBot Taiwan User Group-FBTUG
Doc: https://paper.dropbox.com/doc/FBTUG-FarmHarvestBot-w2FKkhc4ZTlj6knhOK43p

7. Machine Learning + Computer Vision
● 目標
○ 蔬果辨識與定位
■ 種類
● 小果番茄/小黃瓜/彩椒
■ ML
● 辨識蔬果
■ CV
● 影像前處理:擷取影像+Resize
● 座標位置(x,y,z)
○ 蔬果中心位置
○ 切割點位置
○ 座標轉換
■ 夾具移動向量

8. Machine Learning + Computer Vision
● 建立自己的YOLO辨識模型 – 以柑橘辨識為例 - CH大大
source https://projectplus.cc/Projects/yolo-201810/

9. Machine Learning Pipeline
Phase 1.
Problem
Understanding
Phase 3.
Model
Processing
Phase 2.
Data
Processing
Phase 4.
Deploy

10. Problem Understanding
● 目標
○ 專案目標：蔬果採收機器人
■ NN+Vision：辨識可採收的蔬果位置 (x,y,z) + 座標轉換(camera <-> arm)
■ 即時處理
● 系統架構
○ 軟體 Raspbian + ROS
○ 硬體 RPi
○ 機構 (Arm/Camera/autonomous vehicles)
● 作物種類
○ 小果番茄/彩椒/小黃瓜
○ 採收標準/採收方法
○ 夾具
● 場域
○ 溫室為主，一個溫室基本上只種一種作物
Phase 1 Phase 2 Phase 3 Phase 4

11. Problem Understanding
● 目標物種
○ 小果番茄 - 玉女番茄
● 場域與栽種方式
○ 溫室 / 掛網式 (走道寬度/目標物件高度)
● USB Camera
○ 解析度 / F0V(影響目標物件在影像中的成像大小 )
● 判斷標準(採收標準/採收方式)
● 運行平台
○ RPi+Intel Movidius NCS
source: https://paper.dropbox.com/doc/FBTUG--ASl58MS7PkuY4SIYoHLxJXHoAg-QdMWCaBBoyudOTNuvhd4c
Phase 1 Phase 2 Phase 3 Phase 4

12. Data Processing
● 資料特性
● 蒐集照片
● 資料標記
● 資料轉換
Phase 1 Phase 2 Phase 3 Phase 4

13. Data Processing
● 資料特性
○ 目標物種
■ 小果番茄 - 玉女番茄
○ 標記類別
■ 0_pepper_flower 花朵
■ 1_pepper_young 尚未成熟果實
■ 2_pepper_matured 成熟果實可摘採
■ 3_pepper_covered 成熟果實需撥葉摘採
○ 判斷標準與注意事項(from domain expert)
Phase 1 Phase 2 Phase 3 Phase 4
source https://paper.dropbox.com/doc/FBTUG--ASl58MS7PkuY4SIYoHLxJXHoAg-QdMWCaBBoyudOTNuvhd4c

14. Data Processing
● 蒐集照片
○ 活動紀錄
■ 基本資訊(人事時地物)
■ 作物資料
■ 相機資料
■ 拍攝方式
source
活動紀錄 https://paper.dropbox.com/doc/FBTUG-FarmHarvestBot--ASkl43eqEeSkI3g~dcTtYHVWAg-XHZpHOCR4Tw6amCxMY0E0
資料蒐集紀錄表https://paper.dropbox.com/doc/JsCHkjEGzSIAvT65J0fc4
Phase 1 Phase 2 Phase 3 Phase 4

15. Data Processing
● 資料標記
○ 工具
■ ImageLabel (offline)
■ Datatrunk (online)
○ 輸出格式
■ PASCAL VOC Format(XML)
source
https://drive.google.com/file/d/1WfdacRUEJ1HgIlQsimbMVLVe2GlKsKJS/view
https://dataturks.com/projects/kobeyu/FarmHarvestBot-tomato
Phase 1 Phase 2 Phase 3 Phase 4

16. Pascal VOC format.
<annotation>
<folder>GeneratedData_Train</folder>
<filename>000001.png</filename>
<path>/my/path/GeneratedData_Train/000001.png</path>
<source>
<database>Unknown</database>
</source>
<size>
<width>224</width>
<height>224</height>
<depth>3</depth>
</size>
<segmented>0</segmented>
<object>
<name>21</name>
<pose>Frontal</pose>
<truncated>0</truncated>
<difficult>0</difficult>
<occluded>0</occluded>
<bndbox>
<xmin>82</xmin>
<xmax>172</xmax>
<ymin>88</ymin>
<ymax>146</ymax>
</bndbox>
</object>
</annotation>
Phase 1 Phase 2 Phase 3 Phase 4

17. Data Processing
● 資料轉換
○ VOC to YOLO
○ VOC to tfrecord
■ create_pascal_tf_record.py
source
https://github.com/tensorflow/models/blob/master/research/object_detection/dataset_tools/create_pascal_tf_record.
py
Phase 1 Phase 2 Phase 3 Phase 4

18. Model Processing
● 定義問題
● 選擇網路架構
● 訓練
● 驗證
Phase 1 Phase 2 Phase 3 Phase 4

19. Classification/Detection/Segmentation
source https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852

20. Object Detection
● R-CNN
○ Fast R-CNN
○ Faster R-CNN
● YOLO
○ YOLO
● SSD
○ SSD+Mobilenet
Phase 1 Phase 2 Phase 3 Phase 4

21. Object Detection Dataset
● PASCAL VOC
● MSCOCO
● ImageNet Det
Phase 1 Phase 2 Phase 3 Phase 4

22. Model Processing
● 選擇網路
○ 準確度 v.s 執行速度
● 回顧Phase1
○ 運行平台
■ RPi + Intel Movidius NCS
○ 實測結果
■ SSD-Mobilenet(5.5 FPS)
■ Tiny-yolo v2(0.7 FPS)
reference
https://github.com/movidius/ncappzoo
https://www.youtube.com/watch?v=llBhBSgoWPs
Phase 1 Phase 2 Phase 3 Phase 4

23. Model Processing
● 效能指標
○ mAP
○ IoU
Phase 1 Phase 2 Phase 3 Phase 4
source https://github.com/Cartucho/mAP

24. Mobilenet
● Separate convolutions
○ Depthwise + pointwise convolution
● Multiplier
○ α：Width Multiplier (0.25、0.5、0.75、1)
○ ρ：Resolution Multiplier (128、160、192、224)
source:
https://yinguobing.com/bottlenecks-block-in-mobilenetv2/
https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md
https://arxiv.org/abs/1704.04861
Phase 1 Phase 2 Phase 3 Phase 4

25. Mobilenet
輸入資料: W_in * H_in * Nch，Kernel Map: k * k * Nk，輸出資料: W_out * H_out * Nk
一般卷積 計算量為 W_in * H_in * Nch* k * k * Nk
Depthwise separable convolution計算量為 (Depthwise convolution計算量+Pointwise convolution計算量)
Depthwise convolution計算量為 W_in * H_in * Nch* k * k
Pointwise convolution計算量為 Nch*Nk * W_in * H_in
source
https://medium.com/@chih.sheng.huang821/%E6%B7%B1%E5%BA%A6%E5%AD%B8%E7%BF%92-mobilenet
-depthwise-separable-convolution-f1ed016b3467

26. Mobilenet
Million MACs Million Parameters Top-1 Accuracy Top-5 Accuracy
MobileNet_v1_1.0_224 569 4.24 70.9 89.9
MobileNet_v1_1.0_192 418 4.24 70.0 89.2
MobileNet_v1_1.0_160 291 4.24 68.0 87.7
MobileNet_v1_1.0_128 186 4.24 65.2 85.8
MobileNet_v1_0.75_224 317 2.59 68.4 88.2
MobileNet_v1_0.50_224 150 1.34 63.3 84.9
MobileNet_v1_0.25_224 41 0.47 49.8 74.2
MobileNet_v1_1.0_224_quant 569 4.24 70.1 88.9
MobileNet_v1_0.25_128_quant 14 0.47 39.5 64.4
Phase 1 Phase 2 Phase 3 Phase 4

27. MobilenetV2
● Classification
○ MobilenetV2
● Detection
○ MobileNetV2 + SSDLite
● Segmentation
○ MobileNetV2 + DeepLabV3
source https://ai.googleblog.com/2018/04/mobilenetv2-next-generation-of-on.html

28. MobilenetV1 v.s MobilenetV2

29. Model Processing
● Model
○ SSD+Mobilenetv1
○ SSDLite+Mobilenetv2
○ yolo
● Dataset (Image + PASCAL VOC)
○ Farmharvestbot
■ 小果番茄
■ 彩椒
○ MS COCO
Phase 1 Phase 2 Phase 3 Phase 4

30. Model Training
● Model
○ SSD-Mobilenetv1
● Dataset
○ MS COCO
● Training framework
○ Tensorflow + Tensorflow/Models/research/object_detection
● Script
python object_detection/model_main.py
--pipeline_config_path=./object_detection/ssd_mobilenetv1/model/ssd_mobilenet_v1_pets.config
--model_dir=./object_detection/ssd_mobilenetv1/model/
--num_train_steps=200000
--num_eval_steps=8000
--alsologtostderr
source https://github.com/kobe0308/models.git (branch develop)
Phase 1 Phase 2 Phase 3 Phase 4

31. Model Processing
● Model
○ SSD+Mobilenetv1
● Dataset
○ MS COCO
● Training parameters
○ batch size:24
○ steps: 200K
○ classes:20
Phase 1 Phase 2 Phase 3 Phase 4

32. Model Processing
● Model
○ SSD+Mobilenetv1
● Dataset
○ Farmharvestbot-tomato
● Training parameters
○ batch size:24
○ steps: 200K
○ classes:4
Phase 1 Phase 2 Phase 3 Phase 4

33. Model Processing
● Model
○ SSD+Mobilenetv1
● Dataset
○ Farmharvestbot-pepper
● Training parameters
○ batch size:24
○ steps: 200K
○ classes:4
Phase 1 Phase 2 Phase 3 Phase 4

34. Deploy
● Edge compute (compare with client/server)
● Platform(RPi+Neural Compute Stick)
■ SBC(RPi 3 Model B)
● 1.4GHz Quad-Core ARM Cortex-A53 (64Bit)
■ NCS
● Intel Movidus NCS
○ 12 Shave Core(VLIW Arch)
○ 1 TOPs
● Intel Movidus NCS2
○ 16 Shave Core(VLIW Arch)
○ Dedicated Neural Compute Engine
○ 4 TOPs
● Orange Pi AI Stick 2801
○ Matrix Processing Engine
○ 9.3 TOps
source https://qconsf.com/sf2017/system/files/presentation-slides/qconf_presentation.pdf
Phase 1 Phase 2 Phase 3 Phase 4

35. NN+Vision Pipeline
● Recap problem understanding
○ Realtime!
○ Capture image
■ resolution @30fps
○ Acceleactor
■ NN inference - NCS
■ Image Processing - CPU(Multi-thread + SIMD/ARM compute library)
○ Programming language
■ Python or C++
Image
Preprocess
Capture
Image
NN
Inference
Image
Postprocess
source https://www.anandtech.com/show/7591/answered-by-the-experts-arms-cortex-a53-lead-architect-peter-greenhalgh
Phase 1 Phase 2 Phase 3 Phase 4

36. Movidius Toolkit
NCSDK
● Compiler (from trained nn to inference engine)
● Checker
● Profiler (profile performance in each nn layer)
● NCSAPI（C++)
source https://qconsf.com/sf2017/system/files/presentation-slides/qconf_presentation.pdf
Phase 1 Phase 2 Phase 3 Phase 4

37. Movidius Toolkit
● Command
○ mvNCCompile -s <number of shaves> <name of the model>_frozen.pb -in=input
-on=<name of the output node>
● Support training framework
○ Tensorflow
○ Caffe
Phase 1 Phase 2 Phase 3 Phase 4
source https://movidius.github.io/ncsdk/tf_modelzoo.html

38. Movidius Architecture
● SHAVE core
○ 8 parallel function units
■ predicated execution unit (PEU)
■ branch and repeat unit (BRU)
■ two 64-bit load-store units (LSU0/LSU1)
■ 32-bit integer arithmetic unit (IAU)
■ 32-bit scalar arithmetic unit (SAU)
■ 128-bit vector arithmetic unit(VAU)
■ 128-bit compare move unit (CMU)
source https://en.wikichip.org/wiki/movidius/microarchitectures/shave_v2.0
Phase 1 Phase 2 Phase 3 Phase 4

39. Single Issue RISC v.s VLIW
source https://slideplayer.com/slide/4896590/
Phase 1 Phase 2 Phase 3 Phase 4

40. Demo
● Platform
○ RPi3+Intel Movidius NCS
● Model
○ SSD+Mobilenet
● FPS ~5
Phase 1 Phase 2 Phase 3 Phase 4

41. Robot Arm and Camera Calibration
source https://slideplayer.com/slide/7491989/

42. TODO...
● 坑坑相連到天邊...
● Join us!

43. Farmharvestbot
FB: FarmBot Taiwan User Group-FBTUG
Doc: https://paper.dropbox.com/doc/FBTUG-FarmHarvestBot--ASlFVrNQa1XWOCWxktW_~JkgAg-w2FKkhc4ZTlj6knhOK43p
專案成員: https://paper.dropbox.com/doc/FBTUG-FarmHarvestBot--ASkpEA1~UFD9CQDm4hhPaWKMAg-9q9yQmaWjhkgh1EdB4EBc

44. Q & A