Android Builder's Summit 2013

1. Camera 2.0
The New Camera Hardware Interface in Android 4.2
Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging
Ashutosh Gupta, Principal Software Engineer, Aptina Imaging
Android Builder’s Summit, San Francisco, CA
2.18.2013
© 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or
specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to
scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners.
1 | © 2013 Aptina Imaging Corporation

2. Agenda
• Camera Use Cases
‣ Prominent Use Cases
‣ Limitations of existing APIs
‣ Overview of android.hardware.Camera
• Android Framework : Camera Service
‣ Native Camera service
‣ New - Under the hood – Camera 2.0
• It’s all about the Camera hardware !
‣ Camera Hardware Abstraction Layer
‣ Camera Device Driver
• Challenges in Camera HAL development
• Emerging Trends
• Q&A
2 | © 2013 Aptina Imaging Corporation

3. Section I
Android Camera APIs
3 | © 2013 Aptina Imaging Corporation

4. Prominent Camera Use Cases
• Main Use Cases
‣ Live Preview of Camera Stream
• Live Preview + copy of the Frame returned to the application
‣ Capture a frame
‣ Video Recording of a Camera Stream
• Secondary Use Cases
‣ Configuring the Camera
‣ Snapshot during video recording
‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved
• Information Related Use cases
• Receiving more than an image back . e.g. face detection data
• Meta Data of an Image
4 | © 2013 Aptina Imaging Corporation

5. Limitations in Camera API
• No support for Burst Mode Photography
• No support for Panoramic Shots
• Very limited support for frame metadata
• No per-frame control of the camera or the image
processing pipeline
• No access to control subsystems within the camera e.g.
Focus, Flash, Image Sensor
5 | © 2013 Aptina Imaging Corporation

6. Overview of android.hardware.Camera
6 Classes 8 Callback Interfaces
• Camera.AutoFocusCallback
• Camera
• Camera.ErrorCallback
• Camera.CameraInfo
• Camera.FaceDetectionListener
• Camera.Parameters
• Camera.OnZoomChangeListener
• Camera.Size • Camera.PictureCallback
• Camera.Face • Camera.PreviewCallback
• Camera.Area • Camera.ShutterCallback
• Camera.AutoFocusMoveCallback
6 | © 2013 Aptina Imaging Corporation

7. Camera class
Contains all the methods for the Camera Lifecycle
• Open & Release
• Access to the Camera Controls
• Preview
‣ Direct Live Preview to the display or a texture
‣ Get Preview Frame in a Callback
• Capture
‣ Callbacks: Shutter, JPEG, RAW, “Postview”
• Lock & Unlock
• Actions: startAutoFocus, startSmoothZoom & startFaceDetection
7 | © 2013 Aptina Imaging Corporation

8. Camera.Parameters
Class for Camera Controls
① Mandatory Feature Set
‣ getSupportedPreviewSizes + set/get
② Optional Feature Set
‣ isVideoStabilizationSupported + set/get
③ Custom Feature Set
‣ Camera.Parameters class provides a “dumb” pipe to the
hardware for custom controls
‣ set (“your_param_string”, value); get(“your_param_string”);
Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes,
Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding
8 | © 2013 Aptina Imaging Corporation

9. … the rest of the Camera Classes
• Camera.CameraInfo
‣ For each camera, front or back facing, orientation of the camera
image
• Camera.Size
‣ width and height of the image
• Camera.Face
‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of
the face
• Camera.Area
‣ Rectangular bounds with a weight
‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto
Exposure
9 | © 2013 Aptina Imaging Corporation

10. Camera 2 Internals
Released as part of Android 4.2
Camera 1 => android.hardware.Camera
Camera 2 => android.hardware.ProCamera
New Camera HAL implementation based on
Camera 2.0 – Samsung exynos5 based
No application that actually uses Camera 2.0
Disclaimer: All reference to Camera2 in this slide deck is based
on reverse engineering AOSP code !
10 | © 2013 Aptina Imaging Corporation

11. Section II
Android Framework - Camera
11 | © 2013 Aptina Imaging Corporation

12. High Level Architecture
12 | © 2013 Aptina Imaging Corporation

13. Android High Level Architecture
Source: Android Anatomy and Physiology, Google IO 2008
13 | © 2013 Aptina Imaging Corporation

14. Hardware Abstraction Layer
Camera
Source: Android Anatomy and Physiology, Google IO 2008
14 | © 2013 Aptina Imaging Corporation

15. Camera Subsystem
Application
Application framework Hardware Independent
Camera Service
Camera HAL
Implementation
Camera Device Driver Hardware Dependent
Camera Hardware
HAL = Hardware Abstraction Layer
15 | © 2013 Aptina Imaging Corporation

16. Process View
App App App
Binder IPC
ICamera
Camera Service
Back Facing Front Facing
Binder IPC
Surfaceflinger ISurface Camera Hardware Camera Hardware
Object Object
Media server
System Call System Call
Kernel Driver Kernel Driver
16 | © 2013 Aptina Imaging Corporation

17. Inside the Camera App
Application Code
android framework code Camera.java
JNI
android_hardware_Camera.cpp
IBinder Interfaces
Camera Service
libcameraservice.so
Media server
17 | © 2013 Aptina Imaging Corporation

18. JNI Layer
18 | © 2013 Aptina Imaging Corporation

19. android_hardware_Camera
• Allocates Memory from the Java memory heap for JPEG
images.
• If a Copy of the Preview Frame is requested by the app,
then the copy from native to java buffers is done here.
• Creates a persistent context for callbacks from native
code to Java (JNICameraContext)
• Holds references to the Java Camera, Face and Area
objects.
19 | © 2013 Aptina Imaging Corporation

20. Camera Service
- with Camera 2.0
20 | © 2013 Aptina Imaging Corporation

21. Camera Service
Application Code
android framework code Camera.java
JNI
android_hardware_Camera.cpp
ICameraClient | ICameraService | ICamera
Camera Service
libcameraservice.so
camera.h
Camera HAL
implementation
Media server
21 | © 2013 Aptina Imaging Corporation

22. Camera Service with Camera 2
Application Code
android framework code Camera.java
JNI
android_hardware_Camera.cpp
ICameraClient | ICameraService | ICamera
Camera2Client
Camera Service libcameraservice.so
Camera2Device
camera.h camera2.h
Camera HAL Camera2 HAL
implementation implementation
Media server
22 | © 2013 Aptina Imaging Corporation

23. Camera Service
• Runs in the media server process
• It is a shared library libcameraservice.so
• Queries System for Camera HAL Api version - accordingly creates a
CameraClient or Camera2Client for the application
• For Camera 1, libcameraservice does:
Permission check android.permission.CAMERA
Ensures only one Client connects to a Camera Hardware Object
Ensures each Process connects to a single Camera Hardware Object
Redirects callbacks back to the app layer
Accessed over IBinder Interface
Number of Cameras Available
CameraInfo Details
23 | © 2013 Aptina Imaging Corporation

24. Camera2Client
• Implements android.hardware.camera API on top of camera device HAL version 2
• Has processors that run in their own threads
Recording
Manages Preview & Recording Streams Streaming Processor Preview Stream
Stream
Still Image capture output processing JPEG Processor Capture Stream
Processes Callback Callback Processor Callback Stream
ZSL Queue Processing ZSL Processor ZSL Stream
Metadata processing for output frames Frame Processor
Manages Capture Sequences for ZSL Capture Sequencer
regular and Burst modes
24 | © 2013 Aptina Imaging Corporation

25. Camera2Device
• Manages Input and Output Streams between the Camera HAL
and the Service
• Functions
‣ Retrieves Static Metadata Information of the Camera
‣ Sends requests for Capture and Streaming
‣ Manages input and output streams including re-processing streams
and metadata buffers
‣ StreamAdaptor between ANativeWindow Interface and the
Camera HAL stream ops
• ANativeWindows is Android Native Window interface
‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and
AWB)
25 | © 2013 Aptina Imaging Corporation

26. Metadata
• Camera2 has a lot of emphasis on retrieving metadata from the camera and
making it available to the application
• Two kinds of Metadata
‣ Static & Frame-Based
‣ Static does not change and is available without opening the camera device (Camera Info
Class)
‣ *_INFO is the static metadata
ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO,
ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO,
ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO,
ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC,
ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP ,
ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER,
ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS,
ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO,
ANDROID_QUIRKS_INFO etc.
• Extensible - there is provision for Vendor Specific Tags
26 | © 2013 Aptina Imaging Corporation

27. Android Open Source Project (AOSP) Structure
• Android Framework
‣ Java: frameworks/base/core/java/android/hardware
‣ JNI: frameworks/base/core/jni
• Camera Service
‣ frameworks/av/services/camera/libcameraservice/
• IBinder Interfaces
‣ frameworks/av/include/camera/ICamera.h etc.
• IBinder Implementation
‣ frameworks/av/camera/ICamera.cpp etc.
• Camera HAL Interface
‣ hardware/libhardware/include/hardware/camera2.h etc.
• Camera HAL Implementation
‣ hardware/<vendor>/camera (typically)
• Camera Metadata
‣ system/media/camera/include/system/camera_metadata_tags.h
27 | © 2013 Aptina Imaging Corporation

28. Section III
Its all about the Camera Hardware !
28 | © 2013 Aptina Imaging Corporation

29. Camera Hardware Abstraction Layer
Review of a Typical Implementation
29 | © 2013 Aptina Imaging Corporation

30. Camera Stack – Camera HAL
Upper Camera Stack …
Camera
Hardware Abstraction Layer
SurfaceFlinger (HAL)
/Overlay Buffers User
Vendor Specific HAL
Implementation
Camera Driver Kernel
Image Sensor Image Sensor Hardware
Processer
30 | © 2013 Aptina Imaging Corporation

31. Android CameraHAL Library
• The Camera Hardware Abstraction Layer (HAL) is a library that is
specific to the camera hardware platform
‣ Written by hardware vendors (Qualcomm, Samsung, TI, others)
• CameraHAL maps Android Camera Service calls to driver functions
‣ Ice Cream Sandwich (ICS) uses camera.h
‣ Jelly Bean (and above) use camera2.h
• CameraHAL low level interface communicates with the kernel level
driver
‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax
(OMX)
‣ Communicates with the driver through file I/O calls (open, close,
input/output controls (IOCTL), etc)
31 | © 2013 Aptina Imaging Corporation

32. Sample ICS CameraHAL
Functional Diagram
Camera Service I/F
Memory Manager
Display Surface
Manager
CameraHAL
Event
Notification
Manager
Camera Manager
User
Camera Driver Kernel
Source: TI OMAP4 /dev/videoX
git.omapzoom.org
32 | © 2013 Aptina Imaging Corporation

33. Sample JB CameraHAL
Functional Diagram
Camera Service I/F
Stream Manager
CameraHAL
Operations Handler Metadata Handler
Reprocess Stream
Management
User
Camera Driver Kernel
Source: Samsung Exynos 5 /dev/videoX
33 | © 2013 Aptina Imaging Corporation

34. JB CameraHAL Block Diagram Discussion
• Block Functions
‣ CameraHAL
• Initialization, thread creation, function dispatch
‣ Stream Manager
• Handle stream events, coordinate buffer and stream usage,
manage state machine
‣ Metadata Handler
• Acquire per shot metadata, convert to Android format
‣ Reprocess Stream Management
• Setup and manage reprocess streams
34 | © 2013 Aptina Imaging Corporation

35. JB CameraHAL Changes
• For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is
moved to libcameraservice
• CameraHAL 2.0 targeted at HW specific functionality
• New CameraHAL functionality currently not passed up to
applications
• Image metadata has more importance
• Reprocessing introduced – Process an already captured image
stream
• Stream based rather than stream function based
‣ CameraHAL 1.0 has methods for specific stream types (e.g.,
start_preview(), take_picture())
‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream())
35 | © 2013 Aptina Imaging Corporation

36. Camera Device Driver
36 | © 2013 Aptina Imaging Corporation

37. Camera Stack – Camera Driver
Upper Camera Stack …
Camera
Hardware Abstraction Layer
SurfaceFlinger (HAL)
/Overlay Buffers User
Vendor Specific HAL
Implementation
Camera Driver Kernel
Image Sensor Image Sensor Hardware
Processer
37 | © 2013 Aptina Imaging Corporation

38. Android Kernel Camera Driver
• The kernel driver presents a standard interface for
different types of camera hardware
‣ Camera hardware specific attributes are handled by the low
level kernel driver
‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor -
differences are handled at the driver level
• For Android, Video for Linux 2 (V4L2) is used in many
implementations
‣ V4L2 has been in existence for many years
‣ OpenMax (OMX) is also used for a low level driver interface by
some vendors.
38 | © 2013 Aptina Imaging Corporation

39. V4L2 Kernel Driver Block Diagram
V4L2 Driver Interface
- IOCTL support/dispatch
- V4L2 driver infrastructure
Controlling Interface
- Support for different device configurations
- Control device flow
Buffer/Memory Management Camera HW Management
- Memory allocation (if needed) - One of these blocks for each camera
- Buffer management type
- Buffer queue/de-queue - Device discovery
- Device initialization
- Power management
- Set/get device specific parameter
- Enable/disable image streaming
39 | © 2013 Aptina Imaging Corporation

40. V4L2 Kernel Driver Resources
• Memory
‣ Memory can be either driver-allocated or user-provided
‣ The image transfers from the camera to memory through hardware Direct Memory Access
(DMA)
‣ Hardware memory management may be used to avoid contiguous memory requirement
• Interrupts
‣ Camera ports support for interrupts on events such as frame start, finish, focus events,
etc.
• Camera Control: I2C/SPI
‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers
‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C
• Control Signals/GPIO
‣ All controlled by the low level driver
• Power
‣ Sensor power management is critical to embedded device operation
‣ Sensors support standby mode where settings are maintained while power usage is reduced
40 | © 2013 Aptina Imaging Corporation

41. V4L2 Driver Buffer Management
• One or more buffers are supported
• User buffers or kernel-allocated buffers
are supported Buff_0
• Buffers are treated the same for preview,
capture, video (output resolution does not Buff_x Buff_1
matter)
• Buffers are queued to a circular list
Buff_3 Buff_2
• Buffer filling starts when the V4L2
Stream_On command is executed
• Once filled, the CameraHAL de-queues a
buffer, processes the buffer, then re-
queues the buffer
• The Stream_Off command causes all
buffer to be released
41 | © 2013 Aptina Imaging Corporation

42. Typical V4L2 Preview Sequence (1)
• V4L2 preview start up sequence is given below
V4L2 Call Driver Events Hardware Events
VIDIOC_S_FMT – set format Set image format and size Set both resolution and output
pixel format
VIDIOC_G_PARM – get Get a camera driver or hardware Read camera parameter
parameter parameter
VIDIOC_S_PARM – set Set a camera driver or hardware Write camera parameter
parameter parameter
VIDIOC_CROPCAP – get Return camera cropping None
cropping capabilities capabilities
VIDIOC_S_CROP – set Set cropping rectangle Set camera cropping rectangle
cropping
VIDIOC_REQBUFS – request Request buffer support from the None
camera buffer driver (user vs. kernel)
Loop: VIDIOC_QUERYBUF For kernel allocated buffers, return None
– query buffer caps buffer characteristics
V4L2_MMAP – map buffers For kernel allocated buffers, None
to user space memory map to user space
42 | © 2013 Aptina Imaging Corporation

43. Typical V4L2 Preview Sequence (2)
• V4L2 preview start up sequence (cont)
V4L2 Call Driver Events Hardware Events
Loop: VIDIOC_QBUF – Queue buffers in the circular queue none
queue buffers
VIDIOC_STREAM_ON – start Start image capture state Enable image output
streaming
• V4L preview shut-down sequence
V4L2 Call Driver Events Hardware Events
VIDIOC_STREAM_OFF – stop Stop streaming, deallocate Disable image output
streaming buffer
43 | © 2013 Aptina Imaging Corporation

44. V4L2 Media Controller Architecture
• Designed to support dynamically re-connectable
hardware blocks.
• Allows for greater programmer control
• Introduces the notion of entities, pads and links
44 | © 2013 Aptina Imaging Corporation

45. V4L2 Media Controller Architecture(Cont.)
45 | © 2013 Aptina Imaging Corporation

46. Section IV
Challenges in Camera HAL Development
46 | © 2013 Aptina Imaging Corporation

47. Challenges in Camera HAL Development
• Memory Management
• Various Implementation of Camera Driver
• Color Format Conversion
• Buffer Synchronization between Camera & Display
• Support for Advanced Features
47 | © 2013 Aptina Imaging Corporation

48. Are all Camera HALs equal ?
No
• Supported Features depend on hardware capabilities
• Supported Features depend on the implementation in
the Camera HAL
• Performance
• Reliability
• Extensions to the standard Android Feature Set
48 | © 2013 Aptina Imaging Corporation

49. A Peek into the Future
49 | © 2013 Aptina Imaging Corporation

50. Emerging Trends
• Computer Vision Applications go mainstream
‣ APIs on Object Tracking, Gesture Recognition become more
common place
• Computation Photography application
‣ Developers get fine grained control of flash and camera
• High Dynamic Range
‣ Ability to capture larger exposure range
• 3D Imaging
‣ Use of 2 cameras to generate a 3D image
50 | © 2013 Aptina Imaging Corporation

51. What’s coming ?
51 | © 2013 Aptina Imaging Corporation

52. Q&A
52 | © 2013 Aptina Imaging Corporation

53. References
• http://developer.android.com/
• http://www.codeaurora.org
• http://omappedia.org
• http://source.android.com
• http://stackoverflow.com/questions/10775942/android
-sdk-get-raw-preview-camera-image-without-
displaying-it
• http://www.cjontechnology.com/blog/?p=14
• http://graphics.stanford.edu/projects/camera-2.0/
53 | © 2013 Aptina Imaging Corporation

54. About
Aptina enables imaging everywhere.
A leading innovator of CMOS imaging technology, Aptina
delivers excellent pixel performance, sensor functionality and camera
system capability to a world going visual.
Aptina Camera Software Stack (Access) is Aptina’s
implementation of the Android Camera HAL with some custom
extensions.
Contact Information
Balwinder Kaur bkaur@aptina.com
Ashutosh Gupta ashutosh@aptina.com
54 | © 2013 Aptina Imaging Corporation