How to image capture for Unity3D OpenGL ES screen in Android NDK.

1. Android OpenGL ES Game
ImageGrabber Final Report
Jungsoo Nam
2015/03/26
namjungsoo@gmail.com

2. Agenda
• Development environment setup
• Development tools setup
• Build guide
• Source code review
• Java classes/methods
• C functions
• ELF architecture with flow chart
• Linux hooking mechanism

3. Development environment setup (1/2)
• Development tools setup
• Eclipse with ADT(Android Developer Tools) for Mac
• http://dl.google.com/android/adt/adt-bundle-mac-x86_64-20140702.zip
• SDK updates
• Requires android-18(4.4)
• NDK setup
• android-ndk-r10d-darwin-x86_64.bin
• Bash Profile setup
• vi .bash_profile
• export PATH=$PATH:/Users/yegam400/Downloads/android-ndk-r10d

4. Development environment setup (2/2)
• Build guide
• Download source codes(link to download)
• AngryBots.zip
• MediaWebEncoder.zip
• Import projects
• File – Import – Android – Existing Android Code into workspace
• NDK build
• Open terminal
• cd MediaWebEncoder
• ndk-build => libmediaweb.so
• In Eclipse, Right click at MediaWebEncoder project, click “Refresh (F5)”
• Run Unity3D app(UnityPlayerNativeActivity(=AngryBots))
• In Eclipse, Check project reference with MediaWebEncoder

5. Source code review – Java (1/3)
• com.medaweb.Grabber.VideoGrabber
• public
• startEncoder() – starts video recording
• stopEncoder() – stops video recording
• getRecordingEnabled() – gets recording enabled
• initialize() – initialize grabber
• private
• begin/endClear() – called when glClear() -> deprecated
• begin/endSwapBuffers() – called when eglSwapBuffer()
• prepareFrameBuffer() – called when glBindFrameBuffer()

6. Source code review – Java (2/3)
• com.android.grafika.gles – Reuse Grafika sources
• Texture2dProgram (modified for our implementation)
• draw2() – similar to draw()
• backupStates() – backup GLSL, arraybuffer and texture stage states
• setupStates() – same as draw()’s one
• restoreStates() – restore GLSL, arraybuffer and texture stage states
• To backup rendering states are the states which are changed by
setupState():
• 1. active texture unit
• 2. multi texture binding
• 3. current program binding
• 4. array buffer binding
• 5. vertex shader attributes
• 6. blend enabled
• 7. vertex shader attributes' pointer arrays

7. Source code review – Java (3/3)
• Library copy(not needed when using project reference linking)
• mediawebencoder.jar -> libs
• libmediaweb.so -> libs/armeabi-v7a
• User code
• initialize()
• startEncoder()
• stopEncoder()

8. Source code review – C (1/5)
• Android.mk
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := mediaweb
LOCAL_LDLIBS:= -llog -landroid -lEGL -lGLESv2
LOCAL_SRC_FILES := mediaweb.c hook.c
LOCAL_CFLAGS := -w
include $(BUILD_SHARED_LIBRARY)
• Application.mk
APP_ABI := armeabi-v7a
APP_PLATFORM := android-14

9. Source code review – C (2/5)
• mediaweb.c
• Called by VideoGrabber
• Called by Texture2dProgram
public class VideoGrabber {
static {
System.loadLibrary("mediaweb");
}
public static native void setInputSurface(long surfaceNativeHandle);
public static native int getWidth();
public static native int getHeight();
public static native void makeCurrent();
public static native void startNativeEncoder(int captureFbo);
public static native void stopNativeEncoder();
public static native void backupVertexAttribPointer(int index);
public static native void restoreVertexAttribPointer(int index, int size, int
type, int normalized, int stride);

10. Source code review – C (3/5)
• mediaweb.c
• public
• setInputSurface() - set MediaCodec input surface handle
• makeCurrent() – call eglMakeCurrent() with original surfaces
• backupVertexAttribPointer() - backup by glGetVertexAttribPointerv()
• restoreVertexAttribPointer() - restore by glVertexAttribPointer()
• startNativeEncoder() – start native side encoder with custom fbo handle
• stopNativeEncoder() – stop native side encoder
• private
• setupViewport() - calls prepareFramebuffer()
• do_hook2() - do_hook() wrapper. Hook a function
• so_entry() - hooks OpenGL ES functions
• JNI_OnLoad() - called when JNI initializing
These are not
supported by Java
OpenGL ES Wrapper

11. Source code review – C (4/5)
• mediaweb.c
• private
• hooked_glViewport()
• when first called, calls setupViewport() -> prepareFramebuffer()
• hooked_glClear()
• hooked glClear() function. rendering entry point -> deprecated.
• hooked_glBindFramebuffer()
• rendering entry point.
• If framebuffer == 0, then this is main fbo, so it changes mainfbo to capturefbo for capture when stateRecording
== 1.
• hooked_eglSwapBuffers()
• rendering end point.
• If stateRecording == 1, then calls beginSwapBuffers() -> orig_eglSwapBuffers() -> endSwapBuffers()
• hooked_eglMakeCurrent()
• when first called, remember main surfaces
• Unity3D callstack
• glClear()
• glBindFramebuffer()
• eglSwapBuffers()
4 times

12. Source code review – C (5/5)
• hook.c
• private
• get_module_base() – get certain module(*.so) base memory address from
/proc/self/maps
• find_got_entry_address_() – load certain module and find functions address
offset.(according to ELF file format)
• public
• do_hook() – entry of function hooking
• Replace entry_addr to hook_func
• Entry address = base address + offset address

13. ELF architecture with flow chart
• Document
• http://www.cs.bgu.ac.il/~caspl142/wiki.files/lab7/elf.pdf

14. Linux hooking mechanism
• dlsym() vs Elf32_Ehdr

15. Thank you