Scala is a powerful language; You can build front-end applications with Scala.js, and efficient backend application servers for JVM. In this session, we will learn how to build everything with Scala by using Airframe OSS framework. Airframe is a library designed for maximizing the advantages of Scala as a hybrid of object-oriented and functional programming language. In this session, we will learn how to use Airframe to build REST APIs and RPC (with Finagle or gRPC) services, and how to create frontend applications in Scala.js that interact with the servers using functional interfaces for dynamically updating web pages.

1. 1
Taro L. Saito
Treasure Data
October 17th, 2020
Scala Matsuri 2020
Scala For Everything
From Frontend to Backend Applications
Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ

2. Copyright 1995-2020 Treasure Data. All rights reserved.
About Me: Taro L. Saito
2
● Treasure Data
○ Principal Software Engineer
○ Living in US for 5 years
● Ph.D. of Computer Science
○ Database systems, genome science.
● OSS:
○ Airframe, snappy-java (used in
Parquet, Spark, etc.), sbt-sonatype,
sbt-pack, etc.
● Book:
自己紹介

3. Copyright 1995-2020 Treasure Data. All rights reserved.
Today’s Topic
● There are a lot of libraries and frameworks for backend and frontend applications
● Airframe
○ An OSS library collection that enables Scala-oriented development
● Goal: Forget everything other than Scala
3
Scala無双：Scala以外を全て駆逐する
Airframe
Scala.js Web Browsers
JSON

4. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe: Core Scala Modules of Treasure Data
● Airframe (GitHub: wvlet/airframe)
○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019)
■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc.
○ Airframe RPC, Airframe Rx (Scala Matsuri 2020)
○ AirSpec
■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021)
4
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming OSS Outcome
Airframe
AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群

5. Copyright 1995-2020 Treasure Data. All rights reserved.
Treasure Data: A Ready-to-Use Cloud Data Platform
5
Logs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS
Data
トレジャーデータはすぐに使えるクラウドデータプラットフォーム

6. Copyright 1995-2020 Treasure Data. All rights reserved.
Scala.js Application: Presto Service Monitor
● Presto Usage: More than 1,000,000 queries / day (2020), 4 regions
6
Scala.jsで開発されたアプリケーションの例

7. Copyright 1995-2020 Treasure Data. All rights reserved.
Server APIs and Library for Spark
Airframe DI
DataFrame MPC1
airframe-codec
airframe-msgpack
Plazma Public API
airframe-http
airframe-finagle
Airframe DI
Airframe RPC
airframe-fluentd
Master Worker
DesignSparkContext
TDSparkContext TDSparkService
MPC1 Reader/Writer IO Manager
Airframe DI
airframe-http
airframe-config
airframe-launcher
airframe-jmx
airframe-metrics
airframe-control
airframe-metrics
td-spark.jarairframe-log
airframe-log
airframe-codec
airframe-json
Airframe
7
Sparkをサポートするために、ScalaとAirframeで作られたサービスが使われている
● MPC1: MessagePack Columnar Format
● See also td-spark internals (Spark Meetup Tokyo
2020)

8. Copyright 1995-2020 Treasure Data. All rights reserved.
Scala is Versatile
● Backend
○ Can leverage JVM ecosystem (e.g.,
netty, gRPC, etc.)
● Frontend
○ Scala.js:
■ 1.0.0 is released in 2020
● The current version is 1.3.0
■ Compile Scala code into
Javascripts
○ Out of scope of this talk
■ iOS/iPhone native applications
● How can we bridge server (Scala JVM)
and client (Scala.js) applications?
Scalaはバックエンド、フロントエンド開発に使える言語に。では両者をどう繋ぐ？
Scala.js
Web Browsers
Serve-Client
Interaction
8

9. Copyright 1995-2020 Treasure Data. All rights reserved.
Bridging Scala and Scala.js with RPC
Program Function
Call
Return
Local Function Call
(ideal)
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
ローカル関数呼び出しと違い、実際にはネットワーク経由でのRPCが必要になる
Remote Procedure Call
(reality)
Network
Client
Call
Return
Server
Call
Return
9
Scala.js
Scala.js

10. Copyright 1995-2020 Treasure Data. All rights reserved.
Existing Approaches for Implementing RPC
● REST
○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.)
○ REST web frameworks for Scala
■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc.
● gRPC
○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language
○ Generate server and client code stubs from proto files
○ Scala wrappers:
■ ScalaPB, muScala, akka-grpc, etc.
RPCを実装するには、RESTやgRPCなどが使われている
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Return
Server
Call
Return
10

11. Copyright 1995-2020 Treasure Data. All rights reserved.
API Design War
https://cloud.google.com/blog/products/api-management/
understanding-grpc-openapi-and-rest-and-when-to-use-th
em
11
● Which API design should we choose?
● REST API
○ Designing good APIs over HTTP
requires experiences and significant
amount of time
● gRPC
○ Requires special software like code
generators for using the ecosystem
around ProtocolBuffers
● And also, there are too many choices for
web frameworks
しかし、APIデザイン、フレームワーク選びでの争いは付きない

12. Copyright 1995-2020 Treasure Data. All rights reserved.
Why Not Using Scala for RPC Interface?
● Scala is a functional and statically-typed object-oriented programming language
○ Scala functions = RPC methods
○ Scala objects (statically typed) = RPC request/response types
● Scala is a perfect fit for RPC!
12
関数型で静的型付きオブジェクト指向言語のScalaはRPCインターフェースに適している

13. Copyright 1995-2020 Treasure Data. All rights reserved.
Scala-First Approach: Airframe RPC
RPC InterfaceScala.js Client RPC Web Server
Generates
Scala.js
Web Application
API Documentation
13
Airframe RPCはScalaをサーバー・クライアント共通インターフェースとして使う新手法
RPC Call RPC Call
Implements
● Forget about REST or ProtocolBuffers!
● Use Scala as a unified interface between server and clients Airframe

14. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe RPC Backend Is Pluggable
RPC Interface RPC Web Server
Generates API Documentation
14
Airframe RPCのServer, Clientは差し替え可能
Airframe
RPC Call RPC Call
Implements
● Backend server
○ Finagle (HTTP/1), gRPC (HTTP/2)
● Backend client
○ Finagle, gRPC, okhttp, Java URLConnection client etc.
RPC Clients

15. Copyright 1995-2020 Treasure Data. All rights reserved.
Building Blocks of RPC Frameworks
● 1. Message serializer and deserializer
● 2. Network data format
○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC)
● 3. RPC interface language
○ REST API, Protobuf, or Scala
● 4. HTTP client and server implementation
○ Code generator
15
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Return
Server
Call
Return
RPCフレームワークを作るのに必要な構成要素

16. Copyright 1995-2020 Treasure Data. All rights reserved.
MessagePack-Based Object Serialization
● MessagePack
○ A compact binary format compatible
with JSON
Object Object
Pack Unpack
PackUnpack
Server SideClient Side
16
Scala.js
JSON
Airframe
MessagePackを使ってクライアントとサーバー間通信を行う

17. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-codec: Pack/Unpack Interface
● Pack
○ Converting Data to MessagePack
● Unpack
○ Converting MessagePack to Data
Input Output
Pack Unpack
PackUnpack
17
Airframe
MessageCodec MessageCodec
MessagePackを通したデータ変換インターフェースをScalaに適用

18. Copyright 1995-2020 Treasure Data. All rights reserved.
Pre-defined Codecs in airframe-codec
● Primitive Codecs
○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
○ FloatCodec, DoubleCodec
○ StringCodec
○ BooleanCodec
○ TimeStampCodec
● Collection Codec
○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc.
○ OptionCodec
● Java-specific Codec
○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec
○ JDBCResultSetCodec
● etc.
18
Scalaで必要なほぼ全てのデータ型へのマッピングをサポート
Airframe

19. Copyright 1995-2020 Treasure Data. All rights reserved.
Serializing Complex Objects
Pack
Unpack
IntCodec
StringCodec
DoubleCodec
MessageCodec.of[A]
19
オブジェクトの型に合わせてCodecを合成できる
Serialize
Deserialize
JSON

20. Copyright 1995-2020 Treasure Data. All rights reserved.
Defining RPC Interface with Scala
● Add ＠RPC annotation to a trait (= RPC interface)
● Define model classes (= data structures) using case classes
● Use the same interface and model classes between servers and clients
RPC InterfaceをScalaで記述する
20

21. Copyright 1995-2020 Treasure Data. All rights reserved.
Serializing Function Call Request
● Function call arguments
○ A sequence of argument name -> value, ….
● Example:
○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”)
○ Serialize this Map into MessagePack with airframe-codec
● HTTP Mapping
○ HTTP method: POST
○ Path: /(package name)/hello
■ e.g., /hello.api.v1/hello
○ Content body: serialized function call argument data
21
関数呼び出しの引数もシリアライズする

22. Copyright 1995-2020 Treasure Data. All rights reserved.
Implementing RPC Servers
● Extending the RPC interface trait
Serverの実装はRPC traitの継承
22

23. Copyright 1995-2020 Treasure Data. All rights reserved.
● Finagle: Twitter’s HTTP server written in Scala
○ HTTP/1 based. Accessible from web browsers running Scala.js code
airframe-http: Starting A Finagle Backend RPC Server
23
Airframe
Scala.js
airframe-rx-html
airframe-http: FinagleバックエンドのHTTPサーバーを起動する
airframe-http-finagle

24. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-http: gRPC Backend
● Suited to internal microservices
● HTTP/2 based protocol
○ Multiplexing multiple RPC
requests within a single
connection
● No .proto file is required
24
Airframe
airframe-http: gRPCバックエンドサーバーを起動する。ProtoBuf定義は必要ない

25. Copyright 1995-2020 Treasure Data. All rights reserved.
Advanced Topic: Extending gRPC
● gRPC is data-format agonistic
framework
○ To support Scala interface and
MessagePack, we have two
extension points in grpc-java.
● MethodDescriptor
○ Define gRPC endpoints
corresponding to Scala
functions
○ Register marshallers
● Request/ResponseMarshaller
○ Define how to encode/decode
RPC request/response data
25
Airframe RPCはgrpc-javaを拡張してMessagePackデータの受け渡しをしている

26. Copyright 1995-2020 Treasure Data. All rights reserved.
RPC Performance Comparison (Greeter Service)
● Airframe RPC
○ serde: MessagePack (airframe-codec) <->
Scala case classes
○ Finagle (HTTP1) or gRPC (HTTP2)
● ScalaPB
○ serde: Protobuf <-> Scala case classes
○ gRPC (HTTP2)
● grpc-java
○ serde: Protobuf <-> Java classes
○ gRPC (HTTP2)
● Notes
○ Using Finagle with HTTP2 had almost no
benefit over HTTP1
■ Multiplexing RPC requests over
HTTP2 is the key performance factor
○ Overhead of ScalaPB
■ Scala Future (10% overhead)
■ Mapping Protobuf to Scala case
classes (10% overhead)
26
gRPCはリクエスト多重送信により高速。MessagePackによるオーバーヘッドは少ない
HTTP/1 HTTP/2 (gRPC)

27. Copyright 1995-2020 Treasure Data. All rights reserved.
sbt-airframe: Generating RPC Clients
● sbt-airframe plugin
○ Read RPC interface classes
○ Generate HTTP client code for accessing the RPC server
○ OpenAPI schema generation support
sbt-airframe
Code
Generation
RPC Client
Scala.js
27
Scala.js Client
HTTP/gRPC Client
Open API Spec
Cross-Language
RPC Client
RPCクライアントを生成するsbt-airframeプラグイン

28. Copyright 1995-2020 Treasure Data. All rights reserved.
sbt-airframe: Generating Open API Schema
● RPC Interface -> Open API schema YAML file
● Generating RPC Clients & Swagger Documentation
28
ScalaのインターフェースからOpenAPIのYAMLを生成する
RPC Interface
API Documentation
Open API Spec
(YAML)
Cross-Language
RPC Client
Generate
sbt-airframe
API Documentation

29. Copyright 1995-2020 Treasure Data. All rights reserved.
Making An RPC Call
RPCを実行する。サーバー、クライアント共にScalaで動いている
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Response
Server
Call
Response
AirframeScala.js
29

30. Copyright 1995-2020 Treasure Data. All rights reserved.
Scala.js Programming Tips
● Scala.js: Compiling Scala -> JavaScript
● Use Pure Scala Code
○ [NG] Java code
■ Dependent libraries also need to be pure Scala
● Airframe is carefully designed to minimize non-Scala dependencies
○ [NG] Annotation, runtime-reflection
○ [OK] Scala Macros (compile-time meta-programming)
● RPC
○ Asynchronous calls using Scala Future
○ Await(Future) is not supported
■ Because JS is a single-threaded model.
■ Need to properly chain Future operators with map, flatMap, rescue, etc.
30
Scala.jsでのプログラミングのコツ
Scala.js

31. Copyright 1995-2020 Treasure Data. All rights reserved.
Debugging Scala.js Code with airframe-log
● Showing logs at the developer console of the browser
31
airframe-logとブラウザのコンソールでScala.jsコードのデバッグ

32. Copyright 1995-2020 Treasure Data. All rights reserved.
Rendering DOM with Scala.js
● DOM element:
○ <div class=”container”> …. </div>
○ scalajs-dom library has functions necessary for generating DOMs
● Previous approach: Converting XML literal into DOM elements
○ monadic-html, Binding.scala
○ But XML literal will be deprecated in Dotty (Scala 3.0)
32
Scala.jsでのDOMレンダリング手法には様々な手法がある

33. Copyright 1995-2020 Treasure Data. All rights reserved.
Rendering DOM
● Current Best Practice
○ Using Scala functions for
building DOM elements
● Option 1: Separation between
DOM elements and attributes
○ ScalaTags, Slinky
○ Too many parentheses!
● Option 2: No separation
between DOM elements and
attributes
○ scalajs-react, Airframe Rx
○ More Scala-friendly syntax
33
DOMのelement, attributeを区別しない方が関数型に適している
ScalaTags
Airframe Rx

34. Copyright 1995-2020 Treasure Data. All rights reserved.
Complex DOM Rendering
● Airframe Rx
● RxElement interface
○ def render(): RxElement
○ Support nesting with apply(...)
■ Embedding Scala functions
and collections
34
airframe-rx-html: Scalaの関数をネストしてDOMをレンダリングする

35. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe Rx
● Rx.variable
○ Re-render DOM if the variable is updated
● Gathering best practices from existing libraries
○ Type-safe DOM elements and attributes
■ ScalaTags, scalajs-react
○ Reactive streaming
■ Monix
■ Scala Collection
■ Scala.rx
○ Cancelable Interface
■ Cleanup DOM and event handlers when
updating DOM
■ Monix, monadic-html
35
既存技術の良いところ取りをしたAirframe Rx

36. Copyright 1995-2020 Treasure Data. All rights reserved.
Reactive Stream Interface: Rx[X]
● Rx[X]
○ ReactiveX stream implementation
○ e.g., Monix, Scala.Rx
● Reactive DOM Rendering
○ Rewriting DOM interactively upon upstream data
change
● Reactive Operators
○ map, flatMap, filter, join, etc.
○ Almost the same with Scala collection APIs
36
Reactive stream interface Rx[X]でインタラクティブなDOMを作成

37. Copyright 1995-2020 Treasure Data. All rights reserved.
gRPC Streaming with Airframe Rx
● Interactive communication between server and clients with reactive stream
interface: Rx[A]
37
gRPC StreamingもRxインターフェースでサポートできる

38. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe Enables Scala-Oriented Development
38
RPC InterfaceScala.js Client RPC Web Server
Generates
Scala.js
Web Application
API Documentation
RPC Call RPC Call
Implements
Airframe
AirframeでScala中心のアプリケーション開発を可能に
● Only need to learn how to write Scala for kick-starting client/server application
development

39. Copyright 1995-2020 Treasure Data. All rights reserved.
Summary
● Scala is a powerful language that can consolidate existing technologies
● Minimizing the learning cost of application development with Scala-First
approach
○ Airframe OSS (GitHub: wvlet/airframe) is designed for this purpose
39
まとめ：既存技術の長所を兼ね備えることで、Scala無双を可能にするAirframe
Airframe
Scala.js Web Browsers
JSON