4. Akka Streams
● An implementation of Reactive Streams
based on Scala and Akka
○ Java API is also available
● Akka HTTP
○ Covers HTTP server and client applications
● Alpakka
○ Provides various Akka Streams connectors
5. Example
// Source
val source = Source(1 to 100)
// Sink
val sink = Sink.foreach { x: Int => println(x) }
// Just connect Source and Sink directly
source.runWith(sink)
// Put Flow between Source and Sink
source.map(x => x * 2).runWith(sink)
7. ElasticsearchSource
val source = ElasticsearchSource.typed[Book](
"source", // Index name
"book", // Type name
"""{"match_all": {}}""", // Query
ElasticsearchSourceSettings(5) // Buffer size
)
8. Implementation of ElasticsearchSource
● Scroll Elasticsearch and buffer documents
● Push docs when pulled from down stream
● Read next window if read buffer is empty
9. ElasticsearchSink
val sink = ElasticsearchSink.typed[Book](
"sink", // Index name
"book", // Type name
ElasticsearchSinkSettings(5) // Buffer size
)
10. Implementation of ElasticsearchSink
● Pull until buffer is full if buffer isn't full
● Indexing by bulk request buffered docs
● Run these processes in parallel
11. Demo
See testcase to know more details:
https://github.com/akka/alpakka/blob/master/elast
icsearch/src/test/scala/akka/stream/alpakka/elasti
csearch/ElasticsearchSpec.scala
12. TODO
● Automatic pull amount tuning
○ By Looking up to metrics of Elasticsearch cluster
● Partial retry
○ Retry failed documents when bulk request is failed
partially
● Sliced scroll
○ Elasticsearch 5.x supports sliced scroll that makes
possible to scroll in parallel
