Wix has finally released to open-source its Kafka client SDK wrapper called Greyhound.
Completely re-written using the Scala functional library ZIO.
Greyhound harnesses ZIO's sophisticated async and concurrency features together with its easy composability to provide a superior experience to Kafka's own client SDKs.
It offers rich functionality including:
- Trivial setup of message processing parallelisation,
- Various fault tolerant retry policies (for consumers AND producers),
- Easy plug-ability of metrics publishing and context propagation and much more.
This talk will also show how Greyhound is used by Wix developers in more than 1000 event-driven microservices.
9. @NSilnitsky
Kafka Broker
Service A Service B
Abstract
so that it is easy to
change for everyone
Simplify
APIs, with additional
features
Greyhound
wraps Kafka
Kafka
Consumer
Kafka
Producer
10. @NSilnitsky
Multiple APIs
For Java, Scala and
Wix Devs
Greyhound
wraps Kafka
Scala Future ZIO Java
Kafka
Consumer
Kafka
Producer
Kafka Broker
ZIO Core
Service A Service B
* all logic
45. class OldWorker(capacity: Int, /*...*/) {
private val tasksQueue = new LinkedBlockingDeque(capacity)
start()
private def start() = {
// simplified
val thread = new Thread(new TaskLoop)
thread.start()
}
private class TaskLoop extends Runnable {
override def run() = {
// simplified
while (true) {
val task = tasksQueue.take()
task.run()
}
}
}
...
}
Old Worker
46. class OldWorker(capacity: Int, /*...*/) {
private val tasksQueue = new LinkedBlockingDeque(capacity)
start()
private def start() = {
// simplified
val thread = new Thread(new TaskLoop)
thread.start()
}
private class TaskLoop extends Runnable {
override def run() = {
// simplified
while (true) {
val task = tasksQueue.take()
task.run()
}
}
}
...
}
Old Worker
* resource, context, maxPar
47. Greyhound
wraps Kafka
✔ Simple Consumer API
✔ Composable Record
Handler
✔ Parallel Consumption
+ Retries!
What do we
want it to do?
...what about
Error handling?
59. foreachWhile(blockingBackoffs) { interval =>
handleAndBlockWithPolling(interval, blockingStateRef)
}
Current Solution
Checks
blockingState
between short
sleeps
Allows user
request to
unblock
60. foreachWhile(blockingBackoffs) { interval =>
handleAndBlockWithPolling(interval, blockingStateRef)
}
Current Solution
def foreachWhile[R, E, A](as: Iterable[A])(f: A => ZIO[R, E, Boolean]):
ZIO[R, E, Unit] =
ZIO.effectTotal(as.iterator).flatMap { i =>
def loop: ZIO[R, E, Unit] =
if (i.hasNext) f(i.next).flatMap(result => if(result) loop else
ZIO.unit)
else ZIO.unit
loop
}
61. foreachWhile(blockingBackoffs) { interval =>
handleAndBlockWithPolling(interval, blockingStateRef)
}
Current Solution
def foreachWhile[R, E, A](as: Iterable[A])(f: A => ZIO[R, E, Boolean]):
ZIO[R, E, Unit] =
ZIO.effectTotal(as.iterator).flatMap { i =>
def loop: ZIO[R, E, Unit] =
if (i.hasNext) f(i.next).flatMap(result => if(result) loop else
ZIO.unit)
else ZIO.unit
loop
}
Stream.fromIterable(blockingBackoffs).foreachWhile(...)
62. Greyhound
wraps Kafka
✔ Simple Consumer API
✔ Composable Record
Handler
✔ Parallel Consumption
✔ Retries
+ Resilient Producer
What do we
want it to do?
and when
Kafka brokers
are
unavailable...
63. + Retry
on Error
Kafka Broker
Producer
Use Case:
Guarantee completion
Consumer
Wix
Payments
Service
Subscription
renewal
Job
Scheduler
64. + Retry
on Error
Kafka Broker
Producer
Use Case:
Guarantee completion
Consumer
Wix
Payments
Service
Job
Scheduler
68. Greyhound
wraps Kafka
✔ Simple Consumer API
✔ Composable Record
Handler
✔ Parallel Consumption
✔ Retries
✔ Resilient Producer
+ Context Propagation
What do we
want it to do?
Super cool for
us
78. - Much less boilerplate
- Code that’s easier to understand
- Fun
REWRITING
GREYHOUND IN ZIO
RESULTED IN...
79. ...but ZIO offers lower level abstractions too, like Promise and clock.sleep.
SOMETIMES YOU CAN’T DO
EXACTLY WHAT YOU WANT WITH
HIGH LEVEL ZIO OPERATORS