SSII2020TS: Event-Based Camera の基礎とニューラルネットワークによる信号処理〜生き物のように「変化」を捉えるビジョンセンサーと「変化」を処理するニューラルネットワーク〜

イベントカメラの研究動向と
ニューラルネットワークによる処理
2
関川雄介
（デンソーアイティーラボラトリ）

Features of
Event-Based
Camera
3
High
speed
1µs
No
motion
blur
Low Low
Power
Sparse
High
dynamic
range
130dB

6
d-itlab.co.jp
Nagata et.al, SSII2019
6

7
Video from Falanga et.al, How Fast is Too Fast? The Role of Perception Latency in High-Speed Sense and Avoid, RAL2019
7

88
Video from Event-based, 6-DOF Camera Tracking from Photometric Depth Maps, Gallego et.al., PAMI2017

99
Video from Event-based, 6-DOF Camera Tracking from Photometric Depth Maps, Gallego et.al., PAMI2017

Event-Based
Vision
10
• Fast&Sparce,
No-Blur,
HDR,...
Event-Based
Camera
• Process only
for changes
Event-Based
Processing

ニューラルネットワークによる処理, SSII2020
お伝えしたい内容
11
どうやって
処理するの︖
フレームベースカメラ:
輝度の画像を取得するカメラ
イベントベースカメラ:
輝度の差分を取得するカメラ
なにが嬉しいの︖
Video from, Inivation
Y.Sekikawa, イベントカメラの研究動向と,
イベントデータ︓スパースで⾮同期な時系列情報

⾃⼰紹介
§2004- 経済産業省特許庁
➢特許審査 (移動体通信)
§2008- オリンパスイメージング
➢無線ファームウェア開発
➢カメラ商品企画
§2012- デンソーアイティーラボラトリ
➢MIT Media Lab Tangible Media
➢Computational Photography
➢画像テンプレートマッチング
➢⽣成モデル学習（GAN）
➢Event-Basedカメラの信号処理
§2020 PhD@慶應⼤斎藤研 Event Cameraのテーマ
12Y.Sekikawa, イベントカメラの研究動向と,

もくじ
§フレームベースビジョンとその課題 (5min)
§イベントベースカメラ (10min)
§特徴／原理／難しさ
§イベントカメラの研究動向（アルゴリズム＋嬉しさ） (50min)
§モデルベース (20min)
§トラッキング，輝度復元，VO／SLAM，３次元復元
§機械学習ベース (30min)
§フレームに変換
§そのまま処理（SNN※，我々のアプローチ）
§ まとめ
※Spiking Neural Network

Frame-based Vision
概要と課題
14

Frame-based Vision: Sensing & Processing
15
Image from Wikipedia / Video from Inivation
Frame-based Sensing Frame-based Processing
frame
CMOS sensor

Motion blur

Limited Dynamic Range
18
Image from expertphotography.com

Speed vs Power/Data Rate/Price tradeoff
19
Fram
e-based
cam
eraSpeed[fps]
Datarate[bps]
Image from ix-camera
EnergyComsumption[W]
Price[$]

Frame-based Processing
20
106FPS
Tracking, Recognition, …

Biological Vision
“Retina is sensitive to temporal brightness gradients”
“Retina is blind to static scenes in absence of eye movements ”
22
Receptive fields of single neurons in the cat’s striate cortex
David H Hubel et.al.,1959, Nobel prize 1981

Event-based Vision
23
Sensing
Retina (Event Camera)
Processing
Brain (CPU, GPU, SNN-Proc.)

History
24
2010
・ 1991 Mahowald et.al
1990 20202000
・2020 Gen.4
1280 x 720 w/ SONY
・2018 Celexl V
1280 x 960
2015-
2014-
・2020 DVXplorer
640 x 480
2012-
・2018 Samsung Gen.3
Event-Based Sencing Device
2017- ・2020 GaAI One
・2018 DyNap CNN2017-
・2014 IBM TrueNorth
・2018 Intel Loihi
・2018 Stanford Braindrop
Event-Based Processing Device
・2009 Lichtsteiner et.al
128x128

Comparison between different event cameras
25
Prophesee(Chronocam) iniVation(iniLabs) Samsung Celepixel(Hillhouse)
Latest version ATIS-Gen4 DAVIS346 DVS Gen.4 CeleX-V
Resolution CD : 1280 x 720 CD+EM : ? 346x260 1280 x 960 1280 x 800
Pixel pitch CD : 4.86μm CD+EM : ? 18.5μm 4.95μm 9.8μm
Intensity information
EM: Exposure Measurement
130dB
Event resets a capacitor to a
high voltage. Brighter →faster
discharges
APS: Active pixel sensor
56.7dB
Similar to standard frame
N/A
Intensity at event-rate
Other feature
/ Info
IMARGO industrial Camera
Joint dev. with SONY
Sony acquires Insightness
DVXplorer
(640 x 480 no intensity)
In-home monitoring camera
Event-wise optical flow
Commercial product for
DSM in China

Consumer/Industrial products
26
Images from Samsung (left)/ Prophesee.ai (right)
• Low-power
• HDR
• High-speed
• HDR

Bio-inspired Retina: Event-based Cameras driven by intensity changes
Event-Based: Asynchronous
Time
LogIntensity
Sensor array
Frame-Based: Synchronous
Exposure time
Intensity Time
frame

28Video from, Hanme Kim, Event-Based Camera vs Standard Camera)

31
Time
LogIntensity
31

Event generation Model
§Each pixel asynchronously report intensity changes
Image from Kim et.al., et.al, Simultaneous mosaicing and tracking with an event camera

33
Time
LogIntensity
!
""
§ Intensity difference Δ" > C ⟶ trigger event
Δ" #$, %$ ≐ " #$, %$ − " #$, %$ − Δ%$
Δ" #$, %$ = '$(,
Frame
Δ""
$"

34
Δ" = $!% ≈ −("/(* ⋅ ,Δ-
v Temporal relation
v Spatial relation (optical flow constraint)
) #$, %$ − ) #$, %$%& ≈ ,
$∈(
('$
Note: No event when image gradient is perpecdicular to motion −("/(*
,
Time
LogIntensity
!
""""#$
34

Event-Based Camera
35
• High speed (1µs)
• Low data rate/Sparse (0-30Mbps)
• No motion blur
• High dynamic range (130dB)
✔
✔
Fram
e-based
cam
era
Speed[fps]
EnergyComsumption[W]
Event-based camera
Datarate[bps]
Price[$]
✔

Difficulties when dealing with event data
§Sparse data representation: Frame-based alg. cannot be applied
Image from Gehrig et.al, Asynchronous, Photometric Feature Tracking using Events and Frames
§Motion dependent data: Association in SLAM / Generalization in ML
Frame:
Motion Independent
Event (Histogram):
Motion Dependent

Wide Range of Usage
Algorithm
§Tracking
§Optical Flow
§Visual odometry
§SLAM
§Image Reconstruction
§Stereo depth estimation
§3D measurement with SL
§Object Recognition
§Etc..
37
Applications
§Surveillance at Home
§Obstacle avoidance
§UAV, automotive
§Bin-picking
§Gesture recognition
§Etc..

イベントカメラの研究動向
第1部 Model-Based
うれしさ
技術ポイント
38

ML-Based (End-to-end neural-network for event processing)
Algorithm
• Feature Tracking
• Optical Flow (OF)
• Visual Odometry (VO)
• Simultaneous Localization and Mapping (SLAM)
• 3D Reconstruction
• Intensity Reconstruction (IR)
Model-based Processing
Setup
Image from Gallego et.al., Event-based, 6-DOF Camera Tracking from Photometric Depth Maps
Geometry
Planer/Known/3D? Texture
Known/Estimate?
Known/Estimate?
Rotation/SE(2)/SE(3)?
Environment
Static/Dynamic
Use image as
Proxy/Direct?
Ext. sensor
/Reconstruct ?
Algorithm
Input

Speed Invariant Time Surface
for Learning to Detect Corner Points with Event-Based Cameras
Manderscheid et.al., CVPR2019
Very fast/robust corner tracking in challenging illumination conditions
Corner detetion using SI time surface
Event to Time Surface
SI:Speed Invariant
40
Corner tracking
Event only
Image (on the bottom) from Alzugaray, et.al., Asynchronous Corner Detection and Tracking for Event Cameras in Real Time
Video from Alzugaray et.al., Asynchronous Corner Detection and Tracking for Event Cameras
Tracking by simple Nearest Neighbor association

EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames
Gehrig et.al., Depts. Informatics and Neuro informatics@ETH, ECCV2018 IJCV2019
159
! = arg min
!
( − )
Very fast/rubust feature tracking in challenging illumination conditions
Compare intensity increment from event with prediction from frame
41
Feature tracking
Event + Frame

Simultaneous Optical Flow and Intensity Estimation from an Event Camera
Bardow et.al., Imperial College Dyson Lab., CVPR2016
43
32
Data term
"#
Optical flow
Intensity & OF estimation at high rate in challenging illumination conditions
Joint optimization using optical flow constrains
IR+OF
Event

Continuous-time Intensity Estimation Using Event Cameras
Scheerlinck et.al., ACCV2018
37
Real-time & high rate intensity estimation in challenging inllumination conditions
Complementary fusion of frame and event
44
IR
Event+Frame

Simultaneous Mosaicing and Tracking with an Event Camera
§Kim et.al, Imperial College London, BMVC2014
46
Localization (PF)
39
Mapping (EKF based IR)
SLAM (rotation only) in challenging illumination conditions
Mapping by intensity reconstructing (Pioneering work for event-based SLAM)
Intencity MAP
SLAM(SO(3))
Event

Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera
Kim et.al, Imperial College London, ECCV2016 (Best Paper)
48
47
Full 6DOF SLAM in challenging illumination conditions
Extension of Kim2016 to SE(3) by incorporation depth estimation
SLAM(SO(3))
Event

Focus Is All You Need: Loss Functions For Event-based Vision
Guillermo et.al., UZH@ETH, CVPR2018, CVPP2019
49
120
Efficient motion (OF) estimation in challenging illumination conditions
OF estimation w/o intensity. Novel focus-based loss
OF
Event

51
Motion segmentation
Video from: Event-Based Motion Segmentation by Motion Compensation (ICCV'19) 51

EMVS: Event-based Multi-View Stereo
Henri et.al, RPG@ETH, BMVC 2016
52
195
Simple/Fast/Easy 3D reconstruction in challenging illumination conditions
Vote events into DSI using know trajectory
DSI: Disparity Space Image (DSI)
Mapping
Event
max

EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking
and Mapping in Real-time
Rebecq et.al., UZH@ETH, RAL2016
54
48
Very fast SLAM (500Hz on CPU) in challenging illumination conditions
Utilize DSI (No IR, edge-map alignment suffice)
SLAM(SE(3))
Event

Ultimate SLAM? combining events, images, and IMU
for robust visual SLAM in HDR and high speed scenarios
Vidal et.al., RPG@ETH, ROBOTICS AND AUTOMATION LETTERS 2017
56
50
Efficient SLAM in challenging illumination conditions
Utilize all available sensors for computational efficiently and robustness
SLAM(SE(3))
Event+Frame+Gyro

CameraProjector
MC3D: Motion Contrast 3D Scanning Nathan
Nathan et.al., Evanston, ICCP2015
58
28
Real-time & precise 3D reconstruction
Utilize precise event-time stamp for easy & robust correspondence
3D Rec
Event+Projector
(%!, '!, (!, )!)
(%"!, '", (", )")

Event-Based Structured Light for Depth Reconstruction using Frequency Tagged
Light Patterns
Leroux, et.al., University of Pittburgh&CMU&Sorbonne&Universitasx&Prpphesee, arXiv 2018
61
Light weight & real-time 3D reconstruction w/o synchronization
Encode code for temporal dimension → decode by simple pixel-wise correlation
3D Rec
Event+Projector

イベントカメラの研究動向
第2部 ML-Based
うれしさ
技術ポイント
62

63
Event camera
Pros: Can be utilize exiting architecture !! (e.g., CNN)
Cons: Inefficient, Slow
Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing
Pros: Efficient (Process only for changed pixels)
Cons: No established method like CNN
Event to Frame "(⋅)
fS
(e) = y

64
Event camera
!! ≔ x, y, p, t "
fD
(g(e)) = y
fS
(e) = y

Event-based Vision meets Deep Learning
on Steering Prediction for Self-driving Cars
Maqueda et.al., Dept. of Informatics and Neuroinformatics@ETH, CVPR2018
65
126
Sophisticated CNN can be used
Convert sparse events to dense frame

Industrial DVS Design: Key Features and Applications
Ryu et.al., Samsung, CVPR2019WS
66

Learning an event sequence embedding for dense event-based deep stereo
Tulyakov et.al., EPLF, ICCV2019
67
Event camera Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event to Frame "(⋅) &
Better than hand crafted conversion
Learn to convert (temporal kernel ) sparse events to dense frame
'&

End-to-End Learning of Representations for Asynchronous Event-Based Data
Gehrig et.al., RPG@ETH, ICCV2019
68
!±[#", %#, &$]
= (# ∗ %±)('", )#*$)
= ∑
!(∈ℰ±
#±(%%, '%, (%)*(%& − %%, '' − '%, (( − (%)
132
Trainable Kernel

Matrix-LSTM: a Differentiable Recurrent Surface for Asynchronous Event-Based Data
Cannici et.al, arXiv 2020
69

Constant Velocity 3D Convolution
Sekikawa et.al., 3DV IEEE Access 2018
70
[Background] 3D Convolution: Common strategy for capturing spatiotemporal feature
§ Problem: Computationally intensive
!
!
% = ' ⊛ )
Time-surface representation of stream of events
Red: Newer, Blue: Older
[Key observation] Spatiotemporal event ≈ piece-wise linear movements of 2D feature
Efficient 3D convolution to capture spatiotemporal features
Decompose constant velocity 3D kernel into 2D conv+sum
#
#
2D kernel +
linear motion

71
constant velocity 3d kernel
3dconv
>1,000x less MAP※
※multiply–accumulate operation
decompose cv3dconv(ours)
=∑*
71
#

High Speed and High Dynamic Range Video with an Event Camera
Rebecq et.al., UZH@ETH, CVPR 2019 PAMI 2019
73
Existing alg. can be readily applicable for challenging applications
Leaning to convert sparse event to intensity frame

75
Event camera
!! ≔ x, y, p, t "
fD
(g(e)) = y
fS
(e) = y

76
Event camera
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing (Spike)
Cons: Requires Neuromorphic H/W (less neurons), Difficult to train
Event-based processing (Continuous)
fS
(e) = y

SNN (Spikingk Neural Network):
3rd generation of neural network: Spiking Neural Network
77
Leaky and Integrate and Fire (LIF)
Charge → Fire(10ms)→ Refractory(100ms)→
"
#
Activation: Non-differentiable Spike (ANN: Relu, Sigmoid)
Asynchronous: MP※ > threshold → Fire (Similar to Event Camera)
MP%(')
Non-differentiable
*+
*,!
=
*+
*,"
⋅ …
*,#
*,!
Chain Rule
※MP: membrane potential

SNN Hardware
TrueNorth DYNAP Loihi Braindrop
Manufacture IBM aiCTX Intel Stanford
Type of neurons Digital LIF Analog LIF Digital LIF Analog
Neurons per chip 1,0000,000 4096x4 130,000x8 4096
Year 2014 2017 2018 2018
Programing Corelet, Eedn libcaer /cAER in C/C++ Nengo/Brain/PyNN Nengo
Training Outside chip On chip On chip Outside chip
For more detailed review see Young et.al. A Review of Spiking Neuromorphic Hardware Communication Systems, IEEE Access 2019
78

Categorization of training SNN
Supervised
Rewarded-STDP ANN (Back-propagation) to SNN
Unsupervised
STDP
Back-propagation
• Continuous relaxation (Approximate gradient / Inefficient)
• Temporal Coding (Exact / Dead neuron)
• Random back propagation
79

Synaptic Modifications in Cultured Hippocampal Neurons:
Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type
Bi et.al., Journal of Neuroscience, 1998
Images from arc-instruments
Weight update
Simple unsupervised training for non-differentiable spike. Neuroplausible
Hebb rule: “who fire together, wire together”
80

81
Image Diehl et.al, l. Unsupervised learning of digit recognition using spike-timing-dependent plasticity
Simple mapping yealds 95% accuracy
81

A Low Power, Fully Event-Based Gesture Recognition System
Amir et.al., IBM Research+UZH-ETH, CVPR2017
Realized efficient gesture recognition using real SNN H/W (TrueNorth)
Convert trained ANN to SNN
82

Training Deep Spiking Neural Networks Using Backpropagation
Lee et.al, Institute of Neuroinformatics@ETH, Frontiers in Neuroscience 2016
84
Events by emulation saccade
"
# 0
9
Non-differentiable
0
9
Error
Est Ref
44
E2E training of SNN
Approximate non-differentiable spike using differentiable low-passed spike

85
"
# 0
9
Spike rate
Approximate with differentiable continuous function
Spike rate GradientLow pass

Random synaptic feedback weights support error backpropagation
for deep learning
Lillicrap et.al., Univ.Oxford, Nature2016
Symmetric Backpropagation
(Chain Rule on ANN)
Random- Backpropagation
Direct feedback
For more detail see http://www.cs.toronto.edu/~tingwuwang/2546.pdf
Asymmetric Backpropagation
(Mammal neuron)
Neuroplausible training
DNN can be trained using Random matrix $ instead of symmetric weight
Direct error feedback
86

Event-Driven Random Back-Propagation:
Enabling Neuromorphic Deep Learning Machines
Neftci et.al., Univ.California+Intel, Frontiers in Neuroscience 2016
Weight update
Enables on-chip training & layer-by-layer parallelization
Apply RBP to SNN.
87

Event camera
!! ≔ x, y, p, t "
fD
(g(e)) = y
88
fS
(e) = y

EventNet: Asynchronous recursive event processing
Sekikawa et.al, CVPR 2019
89
Event camera
!! ≔ x, y, p, t "
fD
(g(e)) = y
133
Real-time event-wise inference on CPU
Recursive formulation & LUT to drastically reduce computational complexity
fS
(e) = y

Problem Statement: Asynchronously Model Event Stream
90
Requirements
§Sparse Event-based Processing (No densification)
§Recursive Processing (Real time processing)
§Local Permutation Invariance (Order may change)
e : (x, y, p, t)<latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit>
yj = f(ej) ⇡ g(max(h(<latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit>
yj =<latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit>
tj<latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit><latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit><latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit><latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit>
tj n(j)+1<latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit><latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit><latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit><latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit>
t<latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit><latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit><latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit><latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit>
⌧<latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit><latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit><latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit><latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit>
ej := {ei|i = j n(j) + 1, ..., j}<latexit sha1_base64="u2rFRK6vv1UnGQ6yiXMxNqGWQ7A=">AAACwnicbVFba9RAFJ6Nl9Z42+qjL4OLsOIaEluoCIVSffCxgtsWmiVMJifbSeeSzkxsl2n64A/w1/iq/8N/42S7iNvtgYGP75xvzuXLa86MjeM/veDO3Xv319YfhA8fPX7ytL/x7MCoRlMYU8WVPsqJAc4kjC2zHI5qDUTkHA7z049d/vAbaMOU/GpnNUwEmUpWMkqsp7L+MBXEnuSlgzZzVfthJ3WQsUu2U72Vw+r1m2QURdGoStusP4ijeB54FSQLMECL2M82et/TQtFGgLSUE2OOk7i2E0e0ZZRDG6aNgZrQUzKFYw8lEWAmbr5Si195psCl0v5Ji+fs/wpHhDEzkfvKbgFzM9eRt+ZEwy3T6nypv8s1nMl2tdD3HeHcH7ewcGHbcHloW76fOCbrxoKk1zOXDcdW4e7UuGAaqOUzDwjVzP+G6QnRhFpvSJhKOKdKCCILlxI9FeSidXM3VO1SLbDnrjoy5Uwwa9pVBZO3KDz5TxF615KbHq2Cg3dRshnFX7YGu3sL/9bRC/QSDVGCttEu+oz20RhR9AP9RL/Q7+BTUAVngbkuDXoLzXO0FMHlX+Fi4MQ=</latexit>

[Related work] PointNet: Deep Learning on Point Sets for 3D Classification and
Segmentation
§Qi et.al., Stanford Univ., CVPR2016
{&!, . . . , &"}
input points
MLP: Multi-layer perceptron
#*+,(⋅)
( = *({-/, . . . , -0}) = 1(23#({4/, . . . , 40}))
'(⋅)
Point Feature Embedding: 1$ = ℎ%&'(4$)
mlp-e
Nx3
#
!=
(&,(,))
NxK
max
global feature !
mlp-c
1xK
outputs
embedded features "
shared
(64)
shared
(64)
shared
(64)
shared
(128)
shared
(1024)
-+
91
xs
xy
z
[,, &, .]
Direct Point Processing (Efficient ITO. Memory & Computation)
Realize permutation-invariance using symmetric function

92
6(7(#))
Pentagon
Star
8,9({. . . })
{&!, . . . , &"
(,)
}
{&!, . . . , &"
(!)
}
ℎ*+,(⋅)
). = ℎ*+,(+.)
6(7(*))
7(#) = 8,9({1#, . . . , 1+})
7(*)

Idea 1: Recursive computation by temporal coding (t-code)
MLP ℎ
nx1024
shared global
feature
MLP g
Batch-based synchronous architecture (PointNet) Requirements
✓ Sparse
✓ Recursive
✓ PI※
※ Permutation
invariant
max
"($) events
(+ ms)
/(1-) = 4(567({ℎ(9-./ - 0!), . . . ℎ(9-)}))
t-code -
Requirements
✓ Sparse
✓ Recursive
✓ PI
t-code -
MLP ℎ MLP gmax
Event-based asynchronous architecture (EventNet)
global
feature
1x1024
! events (+ ms)
/(1-) = 4(567({:(;-.!, <=-), ℎ(9-)}))
e : (x, y, p, t)<latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit>
93

mlp (64,64,64,128,1024)
Idea 2: LUT※ Realization of MLP ℎ
※Look-up table
>
?
> ?
,(-@) = '(./0({2(3@AB, 56@), ℎ(7@)}))
94
Inputs to ℎ: discrete → Precompute MLP on LUT → 45× faster than MLP

95
Triangle other
Setup Output
95

Event-based Asynchronous Sparse Convolutional Networks
Messikommer et.al, arXiv2020
Synchronous training = Asynchronous event-wise inference. 10x less FLOPS than dense conv
Derived recursive alg. based on SSC※
※Submanifold Sparse Convolutional Networks” (CVPR2018)
96
Image from github/btgraham
CONV
Sparcity is constant acrross layers
=Fixed # of anctive site
(spatial potision whicn contained none zero entry)
SSC

まとめイベントカメラって︖
§明るさの変化を観測するカメラ
§センサーとして良い特徴
§ HDR・⾼速・ブラーレス・コンパクトデータ
§ うまく使えば難しい環境で動作する低計算量で⾼速レスポンスな⼿法が実現︕
99

まとめどうやって使うの︖
§データの形式や特性がフレーム画像と違うので，“フレーム画像処理“がそのまま使えない
§⾮同期＆スパース
§動きで⾒えが変わる
§イベントの特性を⽣かした処理でフレームカメラの適⽤が困難なシーンにも
§HD環境での⾼速トラッキング
§逐次型NNによる⾼速な認識
§将来
§Event型センサ＆処理とフレーム型センサ＆処理のハイブリッド
§スパース性を活かした逐次型NNはまだ黎明期発展が楽しみな分野
100

Reference
• Gallego et.al., Event-based Vision: A Survey
• Gallego et.al., Event-based Vision Resources
• Scaramuzza et.al., Event-based Vision and Smart Cameras (CVPR2019 Workshop)
101

SSII2020TS: Event-Based Camera の基礎とニューラルネットワークによる信号処理〜生き物のように「変化」を捉えるビジョンセンサーと「変化」を処理するニューラルネットワーク〜

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to SSII2020TS: Event-Based Camera の基礎とニューラルネットワークによる信号処理〜生き物のように「変化」を捉えるビジョンセンサーと「変化」を処理するニューラルネットワーク〜

Similar to SSII2020TS: Event-Based Camera の基礎とニューラルネットワークによる信号処理〜生き物のように「変化」を捉えるビジョンセンサーと「変化」を処理するニューラルネットワーク〜 (20)

More from SSII

More from SSII (20)

Recently uploaded

Recently uploaded (20)