Тема нативних движків, що спрощують роботу з 3D графікою для Android, є головним болем багатьох розробників. У той час як для iOS існує SceneKit, розробники під Android були змушені або йти в крос-платформені рішення (Unity3d, LibGDX і т.п.) або використовувати самописні/опенсорсні рішення з обмеженим функціоналом. Поява Sceneform SDK вирішує не тільки цю проблему але й інкапсулює в собі роботу з ArCore SDK, відкриваючи для розробників можливість нативної реалізації AR рішень, не вдаючись до стороннього SDK. В рамках доповіді будуть розглянуті практичні можливості та обмеження Sceneform SDK стосовно проектів з доповненою реальністю (AR). Так само будуть розглянуті такі теми як SLAM, ArCore SDK, розпізнавання та трекінг маркерів на основі зображень, Cloud Anchors та існуючі альтернативи на ринку.

1. Sceneform SDK
In practice
Illya Rodin, CTO
ROAR AR Platform

2. Get started
Use cases:
1. Drawing 3D
scene
2. Item 1 + SLAM
(require
support ARCore)
Requirements:
1.Android SDK
version >=24
2.Java 8
3.*Sceneform Gradle
Plugin
4.*Google Sceneform
Tools for Android
Studio
5.*com.google.ar.sc
eneform:core
Support:
1. Shapes (sphere,
cylinder, cube)
2. 2D Android View
3. 3D models
(OBJ/FBX/GLTF/G
LB)
4. *Android
Property
animations or
FBX 2016/17+)

3. Scene and Nodes
Scene
Sun Camera AnchorNode AugmentedFaceNode
SkeletonNode
Special nodes
TransformableNode

4. World Vs Local Coord
World Coord.
Anchor A Coord.
(Local Coord.)
Anchor B Coord.
(Local Coord.)

5. Node Transformation
● World/LocalPosition
● World/LocalRotation
● World/LocalScale
● Quaternion(float x, float y, float z, float w)
● sqrt(x*x + y*y + z*z + w*w) = 1
● Quaternion.axisAngle(Vector3 axis, float degrees)

6. Renderable:shapes
Sphere
ShapeFactory.makeSphere(0.1f,
Vector3(0.0f, 0.15f,
0.0f),material))
Cylinder
ShapeFactory.makeCylinder(0.1f,
0.3f,Vector3(0.0f, 0.15f,
0.0f),material))
Cube
ShapeFactory.makeCube(Vector3(0.
2f, 0.2f, 0.2f),Vector3(0.0f,
0.15f, 0.0f),,material))

7. Renderable:ViewRendarable
ViewRenderable.builder()
.setView(this, R.layout.view)
.build()
.thenAccept(...);
1 meter = 250 dp
ViewSizer:
● DpToMetersViewSizer
● FixedWidthViewSizer
● FixedHeightViewSizer

8. Sceneform Tools Plugin / Google Sceneform Tools(I)
Import 3D Assets
3D Model Assets:
● OBJ
● FBX
● GLTF/GLB
Dependencies:
● MTL
● BIN
● PNG/JPG/e.t.c.

9. Sceneform Tools Plugin / Google Sceneform Tools(II)
sceneform.asset('sampledata/models/a
ndy_dance.fbx','default',
'sampledata/models/andy_dance.sfa',
'src/main/res/raw/andy_dance',
['sampledata/models/andy_wave_r.fbx'
,
'sampledata/models/andy_wave_l.fbx']
)
3D Model + Dependencies
Sceneform Asset Definition
(.sfa)
Sceneform Binary Asset
(.sfb)

10. Renderable:ModelRendarable
From samples folder
ModelRenderable.builder()
.setSource(this, Uri.parse("andy.sfb"))
//.setSource(this, R.raw.andy)
.build()
.thenAccept(....)
.exceptionally(...);
In runtime (only gltf/glb)
<uses-permission android:name="android.permission.INTERNET"/>
ModelRenderable.builder()
.setSource(this, RenderableSource.builder().setSource(this,
Uri.parse(GLTF_ASSET),RenderableSource.SourceType.GLTF2)
.build()).setRegistryId(GLTF_ASSET)
.build()
.thenAccept(...)
.exceptionally(...);

11. Sceneform Asset Definition
materials[].parameters
Default:
● OBJ
● FBX
● glTF
Custom: *.mat
materials[].source
model.attributes
model.file
model.scale
model.recenter
samplers[]

12. Sceneform Material (I)
A material defines the visual appearance of a surface.
To completely describe and render a surface, a material
provides the following information:
● Material model
● Set of use-controllable named parameters
● Raster state (blending mode, backface culling,
etc.)
● Vertex shader code
● Fragment shader code
material {
// material properties
}
vertex {
// vertex shader, optional
}
fragment {
// fragment shader
}

13. Sceneform Material (II)
Shading model:
● Lit
● Cloth
● Unlit:
material {
name : Skybox,
parameters : [
{
type : sampler2d,
name : skybox
}
],
variables : [ eyeDirection ],
vertexDomain : device,
depthWrite : false,
shadingModel : unlit
}
fragment {
void material(inout MaterialInputs material) {
prepareMaterial(material);
float theta = acos(variable_eyeDirection.y);
float phi = atan(variable_eyeDirection.z / variable_eyeDirection.x)
+
(variable_eyeDirection.x > 0.0 ? 0.0 : PI);
material.baseColor = texture(materialParams_skybox,
vec2((phi + PI / 2.0) / (2.0 * PI), theta / PI));
}
}
vertex {
void materialVertex(inout MaterialVertexInputs material) {
float3 p = getPosition().xyz;
float3 u = mulMat4x4Float3(getViewFromClipMatrix(), p).xyz;
material.eyeDirection.xyz =
mulMat3x3Float3(getWorldFromViewMatrix(), u);
}
}

14. Textures
val texture: CompletableFuture<Texture> = Texture.builder()
.setSource(context, Uri.parse(uri))
.setUsage(Texture.Usage.DATA)
.setSampler(
Texture.Sampler.builder()
.setMagFilter(Sampler.MagFilter.LINEAR)
.setMinFilter(Sampler.MinFilter.LINEAR_MIPMAP_LINEAR)
.build()
).build()
//--------------------------------------------------------------------------------------------------------------------------------------------------//
renderable.material.setTexture(“parameterName”, texture)

15. Filament
Filament is a real-time physically
based rendering engine for Android,
iOS, Linux, macOS, Windows, and WebGL.
It is designed to be as small as
possible and as efficient as possible
on Android.
● OpenGL 4.1+ for Linux, macOS and Windows
● OpenGL ES 3.0+ for Android and iOS
● Metal for macOS and iOS
● Vulkan 1.0 for Android, Linux, macOS and iOS (with
MoltenVk), and Windows
● WebGL 2.0 for all platforms

16. Light & Shadow

18. Animation(I)
File format: FBX 2016/2017 or later
Axis conversion: Up axis is Y
Scale factor: Centimeters
Required geometry settings:
Smoothing groups
Smooth mesh
Referenced asset content
Recommended geometry settings:
Tangents and binormals
Triangulate
NURBS
Animation: enabled
Deformed models settings:
Deformed models
Skins
private void onPlayAnimation() {
if (animator == null || !animator.isRunning()) {
AnimationData data =
andyRenderable.getAnimationData(nextAnimation);
nextAnimation = (nextAnimation + 1) %
andyRenderable.getAnimationDataCount();
animator = new ModelAnimator(data, andyRenderable);
animator.start();
}

19. Animation(II)
private static ObjectAnimator createAnimator(boolean clockwise, float axisTiltDeg) {
Quaternion[] orientations = new Quaternion[4];
//-------------------------------------------------------//
ObjectAnimator orbitAnimation = new ObjectAnimator();
orbitAnimation.setObjectValues((Object[]) orientations);
orbitAnimation.setPropertyName("localRotation");
//------------------------------------------------------/
return orbitAnimation;
}

20. SLAM(Simultaneous localization and mapping)

21. ArCore + SceneformUX
● SceneView
● ArSceneView
● ArFragment
○ Check ARCore version
○ Check permissions
○ Session initialization and
configuration
○ Handle gestures

22. Cloud Anchors

23. Image Recognition
● Up to 1000 reference images.
● Up to 20 images simultaneously in the environment, but it cannot
track multiple instances of the same image.
● The physical image in the environment must be at least 15cm x 15cm
and must be flat
● Once tracked, ARCore provides estimates for position, orientation,
and physical size. These estimates are continuously refined as
ARCore gathers more data.
● ARCore cannot track a moving image, but it can resume tracking
that image after it stops moving.
● All tracking happens on the device, so no internet connection is
required. Reference images can be updated on-device or over the
network without requiring an app update.

24. Augmented Faces

25. ViroCore SDK
● *Free
● Android API 21+
● ReactNative
● AR
● VR
● Support glTF and *FBX
animation
● PBR, Physics,
Portals, Particle
Effects, Views,
e.t.c.

26. Q&A

27. Resources
● https://developers.google.com/ar/develop/java/sceneform/
● https://proandroiddev.com/@harivigneshjayapalan
● https://github.com/google/filament
● https://google.github.io/filament/Filament.md.html
● https://developers.google.com/ar/
● https://virocore.viromedia.com/