"Android's Multimedia Framework" presentation

1. 1
Android's Multimedia
Framework
September 2014
Karim Yaghmour
@karimyaghmour
karim.yaghmour@opersys.com

2. 2
These slides are made available to you under a Creative Commons Share-
Alike 3.0 license. The full terms of this license are here:
https://creativecommons.org/licenses/by-sa/3.0/
Attribution requirements and misc., PLEASE READ:
● This slide must remain as-is in this specific location (slide #2), everything
else you are free to change; including the logo :-)
● Use of figures in other documents must feature the below “Originals at”
URL immediately under that figure and the below copyright notice where
appropriate.
● You are free to fill in the “Delivered and/or customized by” space on the
right as you see fit.
● You are FORBIDEN from using the default “About” slide as-is or any of its
contents.
●
You are FORBIDEN from using any content provided by 3rd
parties without
the EXPLICIT consent from those parties.
(C) Copyright 2013-2014, Opersys inc.
These slides created by: Karim Yaghmour
Originals at: www.opersys.com/community/docs
Delivered and/or customized by

3. 3
About
● Author of:
● Introduced Linux Trace Toolkit in 1999
● Originated Adeos and relayfs (kernel/relay.c)
● Training, Custom Dev, Consulting, ...

4. 4
Agenda
● Architecture Basics
● Display Core
● Multimedia Architecture
● App Dev Model
● Kernel Drivers
● HAL definitions
● HAL modules
● System Services
● Stagefright
● Walkthrough
● Relevant Apps
● References and Pointers

5. 5
1. Architecture Basics
● Hardware used to run Android
● AOSP
● Binder
● System Services
● HAL

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11
/frameworks/base/services/java/...
/frameworks/base/services/jni/
/hardware/libhardware/
/device/[MANUF.]/[DEVICE]
/sdk/emulator/
Kernel or module
/frameworks/base/core/...
AOSP-provided
ASL
Manuf.-provided
Manuf. license
Manuf.-provided
GPL-license

12. 12
2. Display Core
● Display Hardware
● Display stack in Android
● Kernel driver
● HAL definition
● HAL module
● Surface Flinger
● Window Manager

13. 13
2.1. Display Hardware
MMU
IOMMU

14. 14
2.2. Display stack in Android

15. 15

16. 16
From source.android.com - CC-BY 2.5

17. 17
From source.android.com - CC-BY 2.5

18. 18
2.3. Kernel driver
●
Video memory management
●
Mode setting
● Checking of parameters
● Motorola Xoom:
●
/dev/nvhdcp1
● /dev/nvhost-ctrl
● /dev/nvhost-display
● /dev/nvhost-dsi
●
/dev/nvhost-gr2d
● /dev/nvhost-gr3d
●
/dev/nvhost-isp
●
/dev/nvhost-mpe
● /dev/nvhost-vi
● /dev/nvmap
● /dev/tegra-crypto
●
/dev/tegra_avp
●
/dev/tegra_camera
●
/dev/tegra_fuse
● /dev/tegra_rpc
● /dev/tegra_sema
● ... whatever hides in hwcomposer HAL module

19. 19
2.3. HAL Definition
● /hardware/libhardware/include/hardware/hwcomposer.h
● struct hwc_procs:
● invalidate()
● vsync()
● struct hwc_composer_device:
● prepare()
● set()
● dump()
● registerProcs()
● query()
● *()

20. 20
2.5. HAL module
● Skeleton /hardware/libhardware/modules/hwcomposer.cpp
● /system/lib/hw/hwcomposer.BOARD.so
● /system/lib/hw/gralloc.BOARD.so
● Ex. - Mot Xoom:
● hwcomposer.tegra.so
● gralloc.tegra.so
● Surface Flinger hook:
● /frameworks/native/services/surfaceflinger/DisplayHardware
– HWComposer.cpp
– Provides fake vsync if none is provided in HW

21. 21
2.6. Surface Flinger
● Actual server:
● /frameworks/native/services/surfaceflinger
● Client side:
● /frameworks/native/libs/gui
● Client / Server interface:
● ISurfaceComposerClient.cpp
● ISurfaceComposer.cpp
● This is NOT an aidl'ed service
● All communication is manually
marshalled/unmarshalled

22. 22
2.7. Window Manager
● Server side:
● /frameworks/base/services/java/com/android/server/wm/
– WindowManagerService.java
– Session.java
● Client side:
● /frameworks/base/core/java/android/view/
– WindowManager.java
– WindowManagerImpl.java
– ViewRootImpl.java
● Interfaces:
● IWindowManager.aidl
● IWindowSession.aidl
● Parameters (incl. z-order):
● See WindowManager.java

23. 23
3. Multimedia Architecture
● OpenMAX
● Media Playback
● Audio
● Camera

24. 24
3.1. OpenMAX
From Khronos website

25. 25
3.2. Media Playback
From source.android.com - CC-BY 2.5

26. 26
3.3. Audio
From source.android.com - CC-BY 2.5

27. 27
3.4. Camera
From source.android.com - CC-BY 2.5

28. 28
From source.android.com - CC-BY 2.5

29. 29
From source.android.com - CC-BY 2.5

30. 30
4. App Dev Model
● Media Playback
● Audio Record
● Camera

31. 31
4.1. Media Playback
● developer.android.com sample:
Uri myUri = ....; // initialize Uri here
MediaPlayer mediaPlayer = new MediaPlayer();
mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
mediaPlayer.setDataSource(getApplicationContext(), myUri);
mediaPlayer.prepare();
mediaPlayer.start();

32. 32
From developer.android.com - ASL

33. 33
4.2. Audio Record
● developer.android.com steps:
● Create a new instance of android.media.MediaRecorder.
● Set the audio source using MediaRecorder.setAudioSource(). You will probably
want to use MediaRecorder.AudioSource.MIC.
● Set output file format using MediaRecorder.setOutputFormat().
● Set output file name using MediaRecorder.setOutputFile().
● Set the audio encoder using MediaRecorder.setAudioEncoder().
● Call MediaRecorder.prepare() on the MediaRecorder instance.
● To start audio capture, call MediaRecorder.start().
● To stop audio capture, call MediaRecorder.stop().
● When you are done with the MediaRecorder instance, call
MediaRecorder.release() on it. Calling MediaRecorder.release() is always
recommended to free the resource immediately.

34. 34
From developer.android.com - ASL

35. 35
5. Kernel drivers
● Possibly V4Linux
● Possibly ALSA
● ... whatever hides in multimedia HAL modules

36. 36
6. HAL Definition
● /hardware/libhardware/include/hardware/:
● audio_effect.h
● audio.h
● audio_policy.h
● camera.h
● camera2.h
● camera3.h
● camera_common.h

37. 37
7. HAL module
● Audio skeletons in /hardware/libhardware/modules/audio
● USB audio in /hardware/libhardware/modules/usbaudio
● Default implementation in /hardware/libhardware/modules/camera
● /system/lib/hw/:
● audio.a2dp.default.so
● audio_policy.default.so
● audio_policy.mpq8064.so
● audio_policy.msm8960.so
● audio.primary.default.so
● audio.primary.mpq8064.so
● audio.primary.msm8960.so
● audio.usb.default.so
● camera.msm8960.so

38. 38
8. System Services
● Player
● Audio Flinger
● Audio Policy
● Camera

39. 39
8.1. Player
● Actual server:
● frameworks/av/media/libmediaplayerservice
● Client side:
● frameworks/av/media/libmedia
● Client / Server interface:
● IMediaPlayerClient.cpp
● IMediaPlayer.cpp
● IMediaPlayerService.cpp
● IMediaRecorderClient.cpp
● IMediaRecorder.cpp

40. 40
8.2. Audio Flinger
● Actual Server:
● frameworks/av/services/audioflinger
● Client side:
● frameworks/av/media/libmedia
● Client / Server interface:
● IAudioFlingerClient.cpp
● IAudioFlinger.cpp
● IAudioRecord.cpp
● iAudioTrack.cpp

41. 41
8.3. Audio Policy
● Actual Server:
● frameworks/av/services/audioflinger
● Client side:
● frameworks/av/media/libmedia
● Client / Server interface:
● IAudioPolicyService.cpp

42. 42
8.4. Camera
●
Actual Server:
● frameworks/av/services/camera/libcameraservice
● Client Side:
● frameworks/av/camera
● Client / Server interface:
● ICameraClient.cpp
● ICamera.cpp
● ICameraRecordingProxy.cpp
● ICameraRecordingProxyListener.cpp
● ICameraService.cpp
● ICameraServiceListener.cpp
● IProCameraCallbacks.cpp
● IProCameraUser.cpp

43. 43
8.5. Wrapper process
● All media system services run within
“mediaserver” system process.
● Started by init
● Sources:
● frameworks/av/media/mediaserver

44. 44
9. Stagefright
● Multimedia framework
● Houses AwesomePlayer
● Location:
● frameworks/av/media/libstagefright
● Support loadable HW codecs:
● libstagefrighthw.so

45. 45
10. Walkthrough

46. 46
10. Relevant Apps
● Music
● MusicFX
● Gallery2
● SpeechRecorder
● VideoEditor

47. 47
11. References and Pointers
● Google's “official” doumentation:
source.android.com/devices/media.html
source.android.com/devices/audio.html
source.android.com/devices/camera/camera.html
● “Use the source, Luke”
●
Developer doc:
● https://developer.android.com/guide/topics/media/mediaplayer.html
● https://developer.android.com/reference/android/media/MediaPlayer.html
● https://developer.android.com/guide/topics/media/audio-capture.html
● “Android Multimedia Framework”, picker - slideshare
●
“The Android media framework”, Bert Van Dam & Poornachandra Kallare, Android Builders Summit
2014
●
“Android’s Stagefright Media Player Architecture”, Ivo Krka,
http://quandarypeak.com/2013/08/androids-stagefright-media-player-architecture/
●
“Mastering the Android Media Framework”, Google I/O 2009
● https://www.khronos.org/openmax/

48. 48
Thank you ...
karim.yaghmour@opersys.com

49. 49
Acknowledgements:
● Some figures and snippets taken from Google's
Android documentation at
developer.android.com and source.android.com
distributed under the Apache 2.0 license and/or
CC-BY-SA 2.5.