this presentation is an intro to the finagle ecosystem

1. Exploring Twitter’s
Finagle technology stack
for microservices
Tomasz Kogut / @almendar
XII Tricity Scala User Group Meetup

2. About me:
• Software Engineer at
• Working in the internet advertisement
industry
• I try to create distributed, highly performant
systems that will run on the JVM
• Programming in Scala since ~2011

3. Microservices

4. questionnaire
Who uses μ-services approach?
Who has a system that would like to break down into smaller pieces?

5. What we already know?
• Working on huge monolithic projects is hard
• Complexity and interdependencies become
incomprehensible at some point
• We all fell in our guts that breaking it down
is a way to go
• Single responsibility principle but for
services

6. The plan
1. Create multiple git repositories
2. Create/copy/move code around repos
3. Pour it all with a fair share of
REST/HTTP
4. Extract common code/models into libs
5. Run it!

7. Let’s go to production!

10. So what
happened?

12. Paying for the lunch
• Easy to develop
• Hard to operate
• More fine-grained services boost both
traits

13. Things to address
• A lot of stuff from “Fallacies of distributed computing"
• Tracing request across different services
• Service discovery and tracing service topology
• Message serialization
• Distributing load across nodes
• Measuring performance and finding bottlenecks is not easy
• Interface contracts can introduce hidden coupling
• Duplication of effort

15. • Historically we were a .Net shop
• Think spring-like approach
• Moved to API first approach (although we were
running some kind of μ-service approach to some
extend)
• Splitting one of the business features into small-
chunks using whatever tech stack we want so we
went akka/akka-http

16. • We didn’t want to use akka clustering
• Recreating utils e.g. request retry
• Establishing new project = work upfront before creating
any business value e.g. metrics
• Every project had it’s own structure - mental context
switching
• REST is good but other protocols might be better
• Fighting the “spray DSL” and marshallers 😀

17. • Creating json is annoying and automatic derivation from
classes was useless most of the time
• We had to think how to properly version our apis and hold
backward compatibility
• We had situations where multiple services needed to be
deployed for the solution to work (hidden monolith)
• How do I run this service locally?

19. What is it?
• Protocol agnostic RPC system for JVM
• Based on Netty
• Allows for building servers and clients
• Core platform on which Twitter is build
• "Your Server as a Function" by Marius
Eriksen

20. Foundations
• Futures
• Services
• Filters

21. Future
• NOT the Scala one unfortunately but very close
• Some api differences
- import com.twitter.util.Future
- Future.value == Future.successful
- Future.exception == Future.failed
- Future.collect == Future.sequence
• Usual monadic composition with flatMap and map
• There is also twitter’s Try, Duration and a bunch of others…

22. Service
• Basic building block in Finagle
• Just a function that returns a Future
• Servers implement services to which Finagle dispatches incoming requests
• Finagle furnishes clients with instances of Service representing either virtual or
concrete remote servers
abstract class Service[-Req, +Rep] extends (Req =>
Future[Rep])

23. Example local service (1)
import com.twitter.finagle.Service
import com.twitter.util.Future
val lengthService = Service.mk[String, Int] { req =>
Future.value(req.length)
}
lengthService("Hello TSUG").foreach(println)

24. import com.twitter.finagle.http
import com.twitter.finagle.Service
import com.twitter.util.Future
import com.twitter.finagle.Http
val service = new Service[http.Request, http.Response] {
def apply(req: http.Request): Future[http.Response] =
Future.value(
http.Response(req.version, http.Status.Ok)
)
}
val server = Http.serve(":8080", service)
Example http server (2)

25. import com.twitter.finagle.http.Method.Get
import com.twitter.finagle.{Http, Service}
import com.twitter.finagle.http.{Request, Response}
import com.twitter.finagle.service.FailFastFactory.FailFast
import com.twitter.util.Future
val productServiceHttpClient: Service[Request, Response] =
Http.client.configured(FailFast(false)).newClient("127.0.0.1:8080",
"productService").toService
val response: Future[Response] =
productServiceHttpClient(Request(Get,"/products/22"))
response.foreach{ response =>
println(response.statusCode)
println(response.contentString)
}
Example http client (3)

26. Other protocols client
examples
var memCache = ClientBuilder()
.name("mc")
.codec(Memcached())
.hostConnectionLimit(config.concurrency())
.hosts(config.hosts())
val mySqlClient = Mysql.client
.withCredentials(username(), password())
.withDatabase(dbname())
.newRichClient("%s:%d".format(host().getHostName, host().getPort))

27. Services represents
clients and servers
symmetrically
Please notice that:

28. Filter
• Again a function that returns a Future
• Filters are always attached to Services altering their
behavior
• Used for application agnostic concerns such as: timeouts,
retry policies, service statistics, and authentication
abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn]
extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])

29. Filter
• Filters just like any other functions
compose
• There is a andThen method that chains
different filters with each other and in the
end with service

30. Local service filter
import com.twitter.finagle.{Filter, Service}
import com.twitter.util.Future
val someStringMetrics = Service.mk[String,Int] { req =>
Future.value(req.length)
}
val evenMetricsFilter = new Filter[Float, Boolean, String, Int] {
override def apply(input: Float,
stringMetricsService: Service[String, Int]): Future[Boolean] = {
stringMetricsService(input.toString).map(_ % 2 == 0)
}
}
(evenMetricsFilter andThen someStringMetrics)(1.234f)

31. import com.twitter.finagle.{Http,Service, Filter}
import com.twitter.finagle.http.{Request,Response}
import com.twitter.finagle.service.TimeoutFilter
import com.twitter.util.MockTimer
import com.twitter.conversions.time._
val httpClient: Service[Request, Response] =
Http.client.newService("twitter.com")
val timeoutFilter: Filter[Request, Response, Request, Response] =
new TimeoutFilter[Request, Response](30.seconds, new MockTimer)
val httpClientWithTimeout: Service[Request, Response] =
timeoutFilter andThen httpClient
Filter example http client
WARNING
This is only a showcase, timeouts for http clients are best configured by Http.client object
How to implement
this?

32. Stacking filters
recordHandletime andThen
traceRequest andThen
collectJvmStats andThen
parseRequest andThen
logRequest andThen
recordClientStats andThen
sanitize andThen
respondToHealthCheck andThen
applyTrafficControl andThen
virtualHostServer
Fronted webservers configuration at twitter:

33. Cool client filters
(or wannabe-filters)
• MonitorFilter - handling exceptions
• StatsFilter - exposing metrics
• RetryFilter - retying calls with configured budget and a back-off policy
• TimeoutFilter - mentioned earlier, for different aspects like idle time, request
time, connection time ect.
• LoadBalancing (not really a filter) - distributed load across nodes, just pass
multiple host values
• FailFast (not really a filter) - mark node as dead for a certain period of time

34. • Unfortunately not all functionality can be
expressed with Filters
• Some things are a bit hairy - e.g. request
cancelation
• Beneath implementations is non-trivial but
comprehensible
• But the good part is that it all works out to the
box for free with almost none configuration!

35. There is more in Finagle to
explore:
• Thrift protocol
• Mux - multiplexing RPC
• ZooKeeper support
• Network location naming

36. Twitter
Server
Even more goodies

37. Twitter Server
• A “template” for finagle-based server
• Flags
• Logging
• Metrics
• Admin interface

38. Flags (1)
• Cmd line arguments passed to the
application
• A remedy to “How do I start this thing
and what are the options?”
• Smart with type inference and parsing
capabilities

39. Flags (2)
val addr = flag("bind", new InetSocketAddress(0), "Bind address")
val durations = flag("alarms", (1.second, 5.second), "2 alarm durations")
$ java -jar target/myserver-1.0.0-SNAPSHOT.jar -help
AdvancedServer
-alarm_durations='1.seconds,5.seconds': 2 alarm durations
-help='false': Show this help
-admin.port=':9990': Admin http server port
-bind=':0': Network interface to use
-log.level='INFO': Log level
-log.output='/dev/stderr': Output file
-what='hello': String to return
Option for fail fast when missing a flag

40. Logging
• Twitter server provides a logging trait
• It can be configured with standard flags
• Can be tuned on-the-fly

41. Admin interface loggers
control

42. Metrics
• Defining own statistics
• JVM stats
• Processing and network stats

44. Others
• Linting your server for problems
• Looking at threads
• Profiling

45. Admin interface

46. Finatra

47. What is it?
• Finatra is build atop Finagle and Twitter
Server
• Finatra does all the heavy-lifting that one
needs to do when working only with
Finagle

48. Finatra features
• Dependency injection using Guice
• JSON handling with Jackson
• Mustache support
• Routing
• Integrates it all together nicely

49. package pl.tk.finagle.recommendation.controller
import javax.inject.{Inject, Singleton}
import com.twitter.finagle.http.Request
import com.twitter.finagle.tracing.ClientTracingFilter.TracingFilter
import com.twitter.finatra.http.Controller
import com.twitter.finatra.request._
import com.twitter.inject.Logging
import pl.tk.finagle.recommendation.engine.{RecommendationCmd, RecommendationEngine}
case class RecommendationFromCookieRequest(@Inject request: Request,
@RouteParam cookieId: Int,
@RouteParam eshopId: Int,
@Header `x-auth`: String)
@Singleton
class RecommendationController @Inject()
(recommendationEngine: RecommendationEngine)
extends Controller with Logging {
get("/recommend/:eshop_id/:cookie_id") { r: RecommendationFromCookieRequest =>
infoResult("Hello") {
TracingFilter("RecommendationEngine") andThen
recommendationEngine apply
RecommendationCmd(r.cookieId, r.eshopId)
}
}
}

50. Unified error reporting
➜ finagle-spike git:(master) ✗ curl -v 127.0.0.1:2001/recommend/32/wsf
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 2001 (#0)
> GET /recommend/32/wsf HTTP/1.1
> Host: 127.0.0.1:2001
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=utf-8
< Server: Finatra
< Date: Wed, 30 Mar 2016 07:05:53 +00:00
< Content-Length: 79
<
* Connection #0 to host 127.0.0.1 left intact
{"errors":["cookie_id: 'wsf' is not a valid int","x-auth: header is required"]}%

51. JSON Support
• Build on jackson-module-scala
• Support for case classes (de)serlialization
• Integrated with Joda
• Error accumulation (instead of fail-fast)
• Integration with routing
• Most of the cases just return object

52. Customizing Json
class Server extends HttpServer {
override def jacksonModule = CustomJacksonModule
...
}
object CustomJacksonModule extends FinatraJacksonModule {
override val additionalJacksonModules = Seq(
new SimpleModule {
addSerializer(LocalDateParser)
})
override val serializationInclusion = Include.NON_EMPTY
override val propertyNamingStrategy = CamelCasePropertyNamingStrategy
override def additionalMapperConfiguration(mapper: ObjectMapper) {
mapper.configure(Feature.WRITE_NUMBERS_AS_STRINGS, true)
}
}

53. Validation
import com.twitter.finatra.validation._
import org.joda.time.DateTime
case class GroupRequest(@NotEmpty name: String,
description: Option[String],
tweetIds: Set[Long],
dates: Dates) {
@MethodValidation
def validateName = {
ValidationResult.validate(
name.startsWith("grp-"),
"name must start with 'grp-'")
}
}
case class Dates(@PastTime start: DateTime,
@PastTime end: DateTime)
Also checked at routing time

54. Testing (1)
• Based on ScalaTest
• Feature testing (both black box and white box)
- looking at external interface of our service
• Integration tests - only a subset of modules is
instantiated and tested
• Finatra does not provide anything for unit
testing

55. Testing (2)
• Smart JSON diff
• Integration with DI
• Easy mocking
• Embedded http server with our service

56. last but not
least…

58. What is it?
• Distributed request tracing
• Based on Google Dapper paper
• Helps getting insight on how the services
interact

60. Zipkin
Python pyramid_zipkin
Pyramid Http
(B3)
Http (B3) Kafka | Scribe Yes
py2, py3
support.
Java brave
Jersey,
RestEASY,
JAXRS2,
Apache
HttpClient,
Mysql
Http (B3)
Http, Kafka,
Scribe
Yes
Java 6 or
higher
Scala finagle-zipkin Finagle
Http (B3),
Thrift
Scribe Yes
Ruby zipkin-tracer Rack Http (B3)
Http, Kafka,
Scribe
Yes
lc support.
Ruby 2.0 or
higher
C#
ZipkinTracerM OWIN,
Http (B3) Http Yes
lc support.
Not tied to finagle

61. Zipkin Architecture

62. DEMO APP
Recommendation
ProductsService
UserProfile
1 GetProducts
2 GetPromotedProducts
GetUser
CookieId, Eshop

63. Thank you!