how continuation-passing style can be expressed in the terms of monadic effects and how to implement generic async/await using dotty metaprogramming techniques in https://github.com/rssh/dotty-cps-async

1. Can be functional concurrent programming be liberated
from the monadic style
ScalaUA - 2020
// ruslan shevchenko: ruslan@shevchenko.kiev.ua //
work: robotsmom [ managing partner ]
zaka.io [R&D]
twitter: @rssh1

2. Problem statement:
cache.query[Property](sql, args: _*).
map(_.map(_.getValue).toList
)
- typical enterprise application usually contains something like this
monads, monads, … monads everywhere
// are we really need this for fetching data ?
// why accidental complexity is so hight ?
or control flow, reduced to linear with control-flow DSL inside for …
for{
x <- doSomething()
y <- ifM(q)(
oneBranch()
)(
pure(x))
}

3. Problem statement:
cache.query[Property](sql, args: _*).
map(_.map(_.getValue).toList
)
await cache.query[Property](sql, args: _*)
Can we fix this ?
- Is this already exists in Scala ?
Make Scala programming great again ?
- Why it’s all unusable now ?
- What will changed in the future?

4. 1. Introduction.
• Monads/Continuations/Concurrency
• async/await in different languages/runtimes
2. Current situation
• existing libraries
• why it not works (?)
3. Dotty-Cps-Async ( https://github.com/rssh/dotty-cps-async )
1. How implement ‘usual async/await’ ?
2. What to do with
• Higher-order functions ?
• Different monads ?
• Behind concurrency.
3. Tasty reflection in a large scale.
PLAN

5. Monads/Concurrency
1993. Koen Claessen. A Poor Man's Concurrency Monad.
// Journal of Functional Programming
https://pdfs.semanticscholar.org/d4e0/a8554588b91f7404a75bd79807c08771da22.pdf
EVALUATOR
INPUT QUEUE
M[X]
— submit to evaluator value.pure : x ⇒ M[X]
—
submit value and function to eval
map(x : M[X])( f : X ⇒ Y) ⇒ M[Y]
—
submit value and function
and resubmit result back
flatMap(x : M[X])( f : X ⇒ M[Y]) ⇒ M[Y]
SIDE EFFECTS
// it is possible to build concurrency primitives:
fork, queue, … etc

6. Monads/Concurrency
1993. Koen Claessen. A Poor Man's Concurrency Monad.
// Journal of Functional Programming
https://pdfs.semanticscholar.org/d4e0/a8554588b91f7404a75bd79807c08771da22.pdf
EVALUATOR
INPUT QUEUE
M[X]
SIDE EFFECTS
Any concurrency type <=> type of evaluator
Monad-like concurrency in Scala:
- evaluate later in one thread ….
- evaluate now in Execution Pool (Future) ….

7. Why we represent concurrency in monads ?
poor-mans concurrency…..
- processes [CTSS ~ 1960]
- isolated data, communication primitives provides by OS.
-
// Context switching and memory mapping is slow, let’s invent something more fine-grained

8. Why we represent concurrency in monads ?
poor-mans concurrency…..
- processes [CTSS ~ 1960]
- isolated data, communication primitives and scheduling provides by OS.
-
// Context switching and memory mapping is slow, let’s invent something more fine-grained
- kernel threads [OS360Task ~ 1971, SunOS Threads ~ 1993]
- shared data, communication primitives provided by library, scheduling - by OS.
-
// Context switching is slow,
// Dealing with shared data is hard
let’s invent something more fine-grained

9. Why we represent concurrency in monads ?
poor-mans concurrency…..
- processes [CTSS ~ 1960]
- isolated data, communication primitives and scheduling provides by OS.
-// Context switching and memory mapping is slow, let’s invent something more fine-grained
- kernel threads [OS360Task ~ 1971, SunOS Threads ~ 1993]
- shared data, communication primitives provided by library, scheduling - by OS.
-
// Context switching is slow,
// Dealing with shared data is hard
let’s invent something more fine-grained
- user-land threads, Fibers [JVM: Green Threads ~ 2000 , Loom ~ 2018 - 2020]
- shared data, communication primitives and scheduling provided by library/runtime
-
// Dealing with shared data is hard
// IO access with user scheduling is hard.
let’s invent something more simple

10. Why we represent concurrency in monads ?
poor-mans concurrency…..
- processes [CTSS ~ 1960]
- isolated data, communication primitives and scheduling provides by OS.
-// Context switching and memory mapping is slow, let’s invent something more fine-grained
- kernel threads [OS360Task ~ 1971, SunOS Threads ~ 1993]
- shared data, communication primitives provided by library, scheduling - by OS.
-
// Context switching is slow,
// Dealing with shared data is hard
let’s invent something more fine-grained
- user-land threads, Fibers [JVM: Green Threads ~ 2000 , Loom ~ 2018 - 2020]
- shared data, communication primitives and scheduling provided by library/runtime
-
// Dealing with shared data is hard
// IO access with user scheduling is hard.
let’s invent something more simple
- Callback hell, [Node.js: ~ 2009 , Reactive manifesto ~ 2019]
- shared data, communications primitives use callbacks to avoid scheduling.
-

11. User-level scheduling….
- Callback hell,
- shared data, communications primitives use callbacks to avoid scheduling.
-
- Functional Programming (Monads),
- communications primitives use hight-order functions instead callbacks
- which was invented to avoid scheduling.
- Concurrency Frameworks for Functional Programming,
- let’s write generic functional scheduler for better organising of hight-order functions
- which are here to avoid callbacks
- which are here to avoid generic user-land scheduling
- which are here to avoid thread context switching using OS scheduler
- which are here to avoid process context switching using OS scheduler
// Programming with callbacks is hard
// Functional Programming is hard

12. Let’s return to our monads….
M[X]
Concurrency (different types)
Distributed programming
(thread-level, fibers, …. etc)
Abstraction [free monad + interpreter]
Unmonad: f : X ⇒ M[X]
await[M] : M[X] → X
async[M] : (f : (X ⇒ Y)) ⇒ (f′ : (X ⇒ M[Y]))
await ⊏ f await /⊏ f′

13. Let’s return to out monads….
Unmonad:
await[M] : M[X] → X
async[M] : (f : (X ⇒ Y)) ⇒ (f′ : (X ⇒ M[Y]))
await /⊏ f′
async/await
Possible solutions:
lift/unlift
runtime: (Project Loom, specific case of concurrency.)
compile-time: (macros or compiler plugin)
effectfully/!
Idris bang notation
reset/shift

14. Let’s return to out monads….
Scala2 attempts:
• Scala-continuations.
• http://infoscience.epfl.ch/record/149136/files/icfp113-rompf.pdf
• Scala-async:
• https://github.com/scala/scala-async
• Storm-enroute coroutines:
• http://storm-enroute.com/coroutines/
• https://drops.dagstuhl.de/opus/volltexte/2018/9208/pdf/LIPIcs-ECOOP-2018-3.pdf
• Thoughtworks DSL.scala:
• https://github.com/ThoughtWorksInc/Dsl.scala
• Monadless.io:
• http://monadless.io/
• Effectfull:
• https://github.com/pelotom/effectful
// different levels of usability, but all solutions are partial

15. Main techniques for transformation: State Machine CPS Transform
def f(x:Int) {
val q = await(doSomething)
val y = await(doOtherthing(x,q))
y + q
}
class f’ {
val q
val y
var state = 0
def apply(x:Int) {
state match {
case 0 => state=1
doSomething.flatMap(r => {q=r; this(x)})
case 1 => state=2
doOtherthing(x,q).flatMap( r => { y=r; this(x) })
case 2 =>
monad.pure(y+q)
}
}
State Machine

16. Main techniques for transformation: State Machine CPS Transform
def f(x:Int):Int {
val q = await(doSomething)
val y = await(doOtherthing(x,q))
y + q
}
def f’(x:Int, cont[A]: Int => CpsMonad[A]) {
doSomething( r =>
val q = r
doOtherthing(x,q, r => {
val y = r
cont(y + q) })
)
}
CPS Transform

17. Dotty / Scala3:
• New
• compile, syntax, macro system
Let’s made ideal async transformer

18. • Dotty-cps-async
• https://github.com/rssh/dotty-cps-async
• optimised monadic CPS transform.
• any monad
• (which implements needed facilities)
• full language
• (essential constructs)
• higher-order functions
• (need to implement helper typeclass)

19. • optimised monadic CPS transform.
def f(x:Int):Int {
val q = await(doSomething)
val y = await(doOtherthing(x,q))
y + q
}
def f’[F:AsyncMonad](x:Int):F[Int] {
summon[F].flatMap(doSomething){ r =>
val q = r
summon[F].map(doOtherthing(x,q)){r =>
val y = r
y + q
}
}
}

20. • optimised monadic CPS transform
• main constructions:
• Constant expression:
• Block(sequential composition)
• sync, sync:
• async, sync:
• async, async:
m.pure(x)x
{x; y} m.pure({x; y})
{await[M](x); y} m.map(x)(_ => y)
{await[M](x);
await[M](y)}
m.flatMap(x)(_ => y)
//Number of maps ~~ Number of awaits…

21. • Local values
• sync expr, sync tail
• async expr, async tail
m.map(vx){ r => val x = r; y }
{val x = vx; …} //unchanged..
{val x = await(vx); y}

22. • Loops
• l
while( ) {
}
conditionasync
bodyasync
def whilefun() = {
m.flatMap(condition){ c =>
if ( )
m.flatMap(body)(_ =>
whilefun())
else
m.pure(())
}
c

23. async[F:AsyncMonad] {
val connection = await(receiveConnection)
try
while {
val command = readComand(connection)
val reply = await(handleOperation(command))
Command != Shutdown
} do ()
finally
connection.close()
}
m.flatMap(openConnection())(a => {
val connection: Connection[F] = a
m.withAction({
def _whilefun(): F[Unit] =
m.flatMap(
m.flatMap(readCommand(connection))((a: Command) => {
val command: Command = a
m.flatMap(handle(command))((a: Reply) => {
val reply: Reply = a
m.flatMap(
if (!reply.isMuted)
connection.send(reply.toBytes)
else
m.pure(())
)( _ => m.pure(!command.isShutdown))
})
}))(c => if (c) _whilefun() else m.pure(()))
_whilefun()
})(
m.pure(connection.close())
)
})

24. async[F:AsyncMonad] {
val connection = await(receiveConnection)
try
while {
val command = readComand(connection)
val reply = await(handleOperation(command))
Command != Shutdown
} do ()
finally
connection.close()
}
m.flatMap(openConnection())(a => {
val connection: Connection[F] = a
m.withAction({
def _whilefun(): F[Unit] =
m.flatMap(
m.flatMap(readCommand(connection))((a: Command) => {
val command: Command = a
m.flatMap(handle(command))((a: Reply) => {
val reply: Reply = a
m.flatMap(
if (!reply.isMuted)
connection.send(reply.toBytes)
else
m.pure(())
)( _ => m.pure(!command.isShutdown))
})
}))(c => if (c) _whilefun() else m.pure(()))
_whilefun()
})(
m.pure(connection.close())
)
})

25. What we need from monad ?
def pure[T](t:T):F[T]
def map[A,B](fa:F[A])(f: A=>B):F[B]
def flatMap[A,B](fa:F[A])(f: A=>F[B]):F[B]
for control flow
def error[A](e: Throwable): F[A]
def restore[A](fa: F[A])(fx:Throwable => F[A]): F[A]
for try/catch
def adoptCallbackStyle[A](source: (Try[A]=>Unit) => Unit):F[A]
def spawn[A](op: =>F[A]): F[A]
def fulfill[T](t:F[T], timeout: Duration): Option[Try[T]]
for integration
with other
async monads
?

26. Different monads in one expression:
async[ ]{
…
val x = await[ ](something)
…
}
M1
M2
implicit def mc[A](x: [A]): [A]=
….
M2 M1
Possible, if we have implicit conversion:

27. Different monads in one expression:
async[M]{
…
val x = await[M](something)
…
}
LoggedM: M ~> Logged[M]
….
:
async[Logged[M]]{
…
val x = await[M](something)
…
} // now we can log context boundaries

28. Higher-order functions:
async {
val words = line.split(" ")
for(w <- words) {
await(channel.write(CountSignal.Data(w)))
}
await(channel.write(CountSignal.Finish))
}
ArrayOps[String].foreach[U](String => U)
(String => M[U])
summon[CpsShifted[ArrayOps[String],M].foreach(words,m)

29. class ArrayOpsAsyncShift[A] extends AsyncShift[ArrayOps[A]] {
def foreach[F[_],U](arrayOps: ArrayOps[A], monad: AsyncMonad[F])(
f: A => F[U]): F[Unit] = {
var r:F[Unit] = monad.pure(())
arrayOps.foreach{ a =>
val b = f(a)
r = monad.flatMap(r)(_ => monad.map(b)(_ =>()) )
}
r
}
…….
inline given shiftedArrayOps[A] as _ <: AsyncShift[ArrayOps[A]] =
new cps.runtime.ArrayOpsAsyncShift[A]()
Define implicit helper typeclass
Implement ‘shiftedVersions’ of async

30. Higher-order functions:
• Shifted version for standard library
• Provide guidelines for library authors.
• Future:
• In theory it’s possible to generate ‘shifter’ versions automatically
• Dotty store a compiled AST for each method in .class .
• Current limitations:
• it is impossible to access one via compile-time reflection.
• Third-order shifted lambda-functions
• Possible to transform in-place
• Hard to find real-wold example.
• on practice: higher-order ~~ second-order
• Application code:
• usually have ‘monadic’ version as business logic.

31. • Can we apply async/await on other monad ?
Let’s await on space, instead await on time.

32. • Knight tour problem:
case class State(
currentPoint: Point,
board: Chessboard,
trace: List[Point]
)
def nextMoves(state: State):List[State] = …..
def findPath(initPos:Point, state: State): List[State] =
async[List]{
val nextState = await(nextMoves(state))
if (board.isFull())
if (initPos == nextState.currentPoint)
nextState
else
await(Nil)
else
await(findPath(initPos,nextState))
}
// complex backtracking algorithm become simple.

33. Current implementation:
https://github.com/rssh/dotty-cps-async
How can you help(?)
- implement missing parts (it's still a lot of work)
- provide a small examples
- trying to use in own dotty experiments
Thanks for attentions.
// Ruslan Shevchenko <ruslan@shevchenko.kiev.ua>
RobotsMom @rssh1 - twitter
@rssh - github
ScalaUA - 2020