"Digital Gimbal: Rock-steady Video Stabilization without Extra Weight!," a Presentation from FotoNation

Copyright © 2016 FotoNation 1
DIGITAL GIMBAL
Rock-Steady Video Stabilization Without Extra Weight!
Dr. Petronel Bigioi
May 3, 2016

IMAGING
Wide Expertise
Strong Innovation
Top Quality
SOFTWARE
Always Real Time
Platform Agnostic
Full Flexibility
HARDWARE
Proven
Low Power
High Performance
QA
Advanced Data Acquisition
Multi-million Image Database
Testing Automation Tools
FotoNation in a Nutshell

FotoNation in Numbers
550+ 1000+ 60% 2.5+ 450+
imaging patents man years of experience high end smartphone penetration billion units shipped million devices / year
understand ● enhance ● accelerate

DIGITAL GIMBAL
A state-of-the-art, energy-efficient image stabilization solution
for high frequency vibration by FotoNation

Problem Statement
High quality videos 4K @ 60 FPS are accepted
as norm in cameras on the fly and move
Low frequency motion is not the only problem
anymore
HIGH and VERY HIGH frequency motion due to
motors and activities need to be addressed
including motions with horizon lock

Mechanical Gimbal Solution — Current State-of-the-Art to
Attenuate Motors Vibration Effects
ADVANTAGES
DISADVANTAGES
WHAT IS IT?
• Mechanical assembly built to counteract vibration effects
• Uses gyros in close loop to control motors
• Have large range of movement
• Cost!!!
• Large & inconvenient
• Heavy
• Consume power
• Reduce flight time due to extra weight
• Slow reaction time

Digital Gimbal Solution — FotoNation’s Technology to
Deal with Vibrations (Low and High Frequency)
ADVANTAGES
DISADVANTAGES
WHAT IS IT?
• Algorithms removing the need for mechanical assembly; uses gyro to measure high
frequency vibration and digital image processing for frame to frame registration to reverse
the effects in the image/frames
• Low cost
• No extra weight
• Low power
• Ultra fast reaction time
• Advanced image stabilisation
• Rolling shutter correction
• Limited range of movement

Image Stabilization Requirements
Image stabilization has always been a subject of great interest especially when
associated with camera modules on the move; today’s solutions are influenced
by system requirements (cost) as well as jitter specific problems (usage).
PerformanceSystem requirements
good sampling quality
stable frame rate
small error estimations
low power consumption
future proof (2K, 4K, 8K)
roll (x-y rotation)
panning
out of plane rotations
rolling shutter distortions
large moving objects

The Rolling Shutter Problem
Hand jitter vibration patterns Drone/motorbike jitter vibration patterns
Rolling shutter effects — caused by slow
camera motion during line by line exposure
and frame readout
Rolling shutter effects — caused by rapid
camera motion during line by line exposure
and frame readout
Have low frequency (up to 15Hz) Have high frequency (hundreds of Hz)

ISP
Inertial
Sensor
Motion
Filtering
Camera
projection
model
Motion
Correction
Image
Corrected
Image
Correction
Grid
Motion
Estimation
Image
Sensor
Legend:
Memory Buffer
Digital Gimbal Components
HW Module
App Processor & DDR
256 independent
motion vectors;
IMU samples
acquisition and sync;
sensor fusion
(IMU + frame to frame)
Corrects lens distortion,
roll, pan & complex
rolling shutter in 1 pass
Adaptive motion filtering
& rolling
shutter effects
estimation
System Architecture and Workflow

Motion Estimation — Synchronization
• Accurate synchronization of inertial and video data is essential
• Hardware core used to read inertial data (IMU) and assign precise timestamps,
with same time source for frames and IMU samples
• Two possible implementations
• Pure HW implementation — ensures minimum latency and highest timing
accuracy, low power consumption
• Hybrid implementation — more flexible, but higher power. Requires high
sampling frequency to minimize motion measurement errors

SOF
Interrupt
SOF
InterruptSample with
FSYNC marker
Sample with
FSYNC marker
unknown time
periods
Maximum unknown period is equal to IMU sample interval. Very inaccurate for low sampling frequencies
Hybrid Synchronization Timing Accuracy Limit

Bias — a constant value added to the measured signal
Motion Filtering — IMU calibration

Sampled IMU data needs to be filtered (4x rule) in order to avoid aliasing
Motion Filtering — Data Filtering

Use sensor fusion to track camera
orientation over time
Using gyroscope alone to track camera
orientation leads to error accumulation
Acceleration and magnetic field
lack short-term accuracy but
provide good long-term
reference frame
Gyroscope provides short-
term accurate changes in
the camera orientation
Motion Filtering — Sensor Fusion

• The real camera orientation is tracked using quaternion arithmetic
• The required correction is the difference between the real and the estimated
trajectory
• Our motion filters adapts to the camera motion in order to produce naturally
smooth experience and react fast to the intentional motion
Motion Filtering

• Stabilisation margin is limited by the sensor size. Sudden camera movements can cause
optimal correction to exceed available correction margin. Excessive motion needs to be
limited to the available correction margin
• Limits imposed on 2D projection must control 3D rotation represented as 4D quaternion —
difficult task. Problem gets even more complicated for highly distorted rectilinear lenses
(left) or fisheye lenses (right)
Motion Filtering — Motion Limitation

time
reference line
• Orientation of the camera in the first line is the
reference point
• Each consecutive line is transformed to counteract
camera movement up to the given line
• Inertial samples and the video frames must be
precisely synchronized
• Camera projection must be accurately modeled
Motion Correction — Rolling Shutter Correction

Camera frame
reference
vector
Gravity vector
from
accelerometer
Correction
amount
After correction
vectors are
aligned
The horizon locking feature keeps the horizon line in the middle of the
frame and horizontal regardless of the drone orientation.
Motion Correction — Horizon Locking Option

Motion Correction With “Follow Me” Feature /
Owner Tracking

FUTURE
PROOF
FAST TO
DEPLOY
EFFICIENTLY
DESIGNED
In a seamless way that is simultaneously
A DEDICATED
HARDWARE — IPU
Low gate count implementation
to offload heavy image and video
computation. Most difficult tasks
completed locally and then passing
processed data to host CPU
How Do We Deliver This?

OBJECT DETECTION ENGINE
~ 1M gates, 240 kB Sram, 54 mW
Multi-Core CPU
GPU DSP ISP
3G/4G Baseband
Memory & I/OsIPU
Video Encoder/Decoder
MOTION PROCESSING ENGINE
~ 125K gates, 9.8 kB Sram, 2 mW
DISTORTION CORRECTION ENGINE
~ 390K gates, 48 kB Sram, 18 mW
4K@60FPS, 28nm technology Typical AP
IPU Components Overview

• Precise lens modeling allows for lens distortion correction and re-projection.
• Any type of correction is possible (rectilinear to perspective, fisheye to
perspective, distorted fisheye to perfect fisheye, fisheye to cylindrical, arbitrary
input projection to arbitrary output projection and freeform image warping).
• All distortion corrections are done at the same time with stabilisation and high
frequency rolling shutter correction (single pass).
• Adding distortion correction has no impact on system performance.
• Frames are resampled only once to ensure maximum image quality.
• Lowest possible power consumption (18mW for 4k@60fps correction in 28n
technology).
Motion Correction — Uses FotoNation’s DCE
(Distortion Correction Engine)

DCE GPU
Designed to work with
rectangular texture mapping
for 2D image transformations
Native support for
bicubic resampling
Scalable with increasing input
size and frame rata
Small gate count
Low power consumption
Low bandwidth thanks to
specific cache design
Designed to support generic
triangular texture mapping
for 3D scenes rendering
Bi-cubic resampling requires
extra processing power &
time
Designed to work at display
resolution
Large gate count
High power consumption
High bandwith
Motion Correction — Done DCE (Distortion Correction
Engine) Part of FotoNation’s IPU (Image Processing Unit)

INPUT OUTPUT
Correction — DCE example
Corrected by FotoNation DCE

Correction — DCE example
INPUT OUTPUT
Corrected by FotoNation DCE

FotoNation EIS / High Frequency Stabilization

• Digital Gimbal components (synchronization, motion filtering and
correction) are part of FotoNation’s IPU (Image Processing Unit) for
optimum low power high performance implementation.
• IPU incorporates additional units to enable more features such as owner
recognition and tracking combined with image stabilization (e.g. high
performance object detection and tracking, high performance face
detection and tracking and face recognition).
Conclusion

Thank You

FotoNation EIS / A No Trade-off User Experience

"Digital Gimbal: Rock-steady Video Stabilization without Extra Weight!," a Presentation from FotoNation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to "Digital Gimbal: Rock-steady Video Stabilization without Extra Weight!," a Presentation from FotoNation

Similar to "Digital Gimbal: Rock-steady Video Stabilization without Extra Weight!," a Presentation from FotoNation (20)

More from Edge AI and Vision Alliance

More from Edge AI and Vision Alliance (20)

Recently uploaded

Recently uploaded (20)

"Digital Gimbal: Rock-steady Video Stabilization without Extra Weight!," a Presentation from FotoNation