SlideShare a Scribd company logo
1 of 109
Download to read offline
Building
      Scalable,
Highly Concurrent &
   Fault-Tolerant
      Systems:
  Lessons Learned

     Jonas Bonér
       CTO Typesafe
      Twitter: @jboner
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                      Agony
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                      Agony
                                                                                                    and Pain
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                    lots of
                                                                                                      Pain
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
Agenda
• It’s All Trade-offs
• Go Concurrent
• Go Reactive
• Go Fault-Tolerant
• Go Distributed
• Go Big
Building Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons Learned
It’s all
Trade-offs
Performance
     vs
 Scalability
Latency
     vs
Throughput
Availability
    vs
Consistency
Go Concurrent
Shared mutable state
Shared mutable state
Together with threads...
Shared mutable state
              Together with threads...


...leads to
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
              ...and the root of all EVIL
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
              ...and the root of all EVIL


Please, avoid it at all cost
Shared mutable state
                     B L E
                 T
               U ! A
     Together with threads...

            M e!!
        IM at EVIL
              ...code that is totally INDETERMINISTIC

      e t
...leads to


     s s      ...and the root of all

    U avoid it at all cost
Please,
The problem with locks
• Locks do not compose
• Locks break encapsulation
• Taking too few locks
• Taking too many locks
• Taking the wrong locks
• Taking locks in the wrong order
• Error recovery is hard
You deserve better tools

• Dataflow Concurrency
• Actors
• Software Transactional Memory (STM)
• Agents
Dataflow Concurrency
• Deterministic
• Declarative
• Data-driven
  • Threads are suspended until data is available
  • Lazy & On-demand
• No difference between:
 • Concurrent code
 • Sequential code
• Examples: Akka & GPars
Actors
•Share NOTHING
•Isolated lightweight event-based processes
•Each actor has a mailbox (message queue)
•Communicates through asynchronous and
 non-blocking message passing
•Location transparent (distributable)
•Examples: Akka & Erlang
STM
• See the memory as a transactional dataset
• Similar to a DB: begin, commit, rollback (ACI)
• Transactions are retried upon collision
• Rolls back the memory on abort
• Transactions can nest and compose
• Use STM instead of abusing your database
  with temporary storage of “scratch” data
• Examples: Haskell, Clojure & Scala
Agents
• Reactive memory cells (STM Ref)
• Send a update function to the Agent, which
  1. adds it to an (ordered) queue, to be
  2. applied to the Agent asynchronously
• Reads are “free”, just dereferences the Ref
• Cooperates with STM
• Examples: Clojure & Akka
If we could start all over...
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
4. Finally - only if really needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
4. Finally - only if really needed
  • Add Monitors (Locks) and explicit Threads
Go Reactive
Never block
• ...unless you really have to
• Blocking kills scalability (and performance)
• Never sit on resources you don’t use
• Use non-blocking IO
• Be reactive
• How?
Go Async
  Design for reactive event-driven systems
1. Use asynchronous message passing
2. Use Iteratee-based IO
3. Use push not pull (or poll)
• Examples:
   • Akka or Erlang actors
   • Play’s reactive Iteratee IO
   • Node.js or JavaScript Promises
   • Server-Sent Events or WebSockets
   • Scala’s Futures library
Go Fault-Tolerant
Failure Recovery in Java/C/C# etc.
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
  • SCATTERED all over the code base
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control


                d o
• If this thread blows up you are screwed

               n !!
• So you need to do all explicit error handling

              a !
            c r
  WITHIN this single thread

           e te
• To make things worse - errors do not

          W et
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that

            b
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
  • SCATTERED all over the code base
Just

Let It Crash
Building Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons Learned
The right way
1. Isolated lightweight processes
2. Supervised processes
 • Each running process has a supervising process
 • Errors are sent to the supervisor (asynchronously)
 • Supervisor manages the failure
• Same semantics local as remote
• For example the Actor Model solves it nicely
Go Distributed
Performance
     vs
 Scalability
How do I know if I have a
 performance problem?
How do I know if I have a
 performance problem?

       If your system is
    slow for a single user
How do I know if I have a
  scalability problem?
How do I know if I have a
  scalability problem?

        If your system is
     fast for a single user
  but slow under heavy load
(Three) Misconceptions about
     Reliable Distributed Computing
                     - Werner Vogels

1. Transparency is the ultimate goal
2. Automatic object replication is desirable
3. All replicas are equal and deterministic

  Classic paper: A Note On Distributed Computing - Waldo et. al.
Fallacy 1
Transparent Distributed Computing
• Emulating Consistency and Shared
  Memory in a distributed environment
• Distributed Objects
 • “Sucks like an inverted hurricane” - Martin Fowler
• Distributed Transactions
 • ...don’t get me started...
Fallacy 2
                          RPC
• Emulating synchronous blocking method
  dispatch - across the network
• Ignores:
 • Latency
 • Partial failures
 • General scalability concerns, caching etc.
• “Convenience over Correctness” - Steve Vinoski
Instead
Instead
   Embrace the Network

    Use                                    it
                                    i th
                                   w
Asynchronous               don e

  Message             be
                  d
               an
   Passing
Guaranteed Delivery
  Delivery Semantics
   • No guarantees
   • At most once
   • At least once
   • Once and only once
It’s all lies.
It’s all lies.
The network is inherently unreliable
 and there is no such thing as 100%
        guaranteed delivery


       It’s all lies.
Guaranteed Delivery
Guaranteed Delivery
The question is what to guarantee
Guaranteed Delivery
  The question is what to guarantee
1. The message is - sent out on the network?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
5. The message is - starting to be processed by the receiver?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
5. The message is - starting to be processed by the receiver?
6. The message is - has completed processing by the receiver?
Ok, then what to do?
1. Start with 0 guarantees (0 additional cost)
2. Add the guarantees you need - one by one
Ok, then what to do?
1. Start with 0 guarantees (0 additional cost)
2. Add the guarantees you need - one by one

  Different USE-CASES
            Different GUARANTEES
                Different COSTS
Ok, then what to do?
    1. Start with 0 guarantees (0 additional cost)
    2. Add the guarantees you need - one by one

       Different USE-CASES
                  Different GUARANTEES
                       Different COSTS
 For each additional guarantee you add you will either:
• decrease performance, throughput or scalability
• increase latency
Just
Just


Use ACKing
Just


Use ACKing
and be done with it
Latency
     vs
Throughput
You should strive for
maximal throughput
          with
acceptable latency
Go Big
Go Big
 Data
Big Data
  Imperative OO programming doesn't cut it
• Object-Mathematics Impedance Mismatch
• We need functional processing, transformations etc.
• Examples: Spark, Crunch/Scrunch, Cascading, Cascalog,
  Scalding, Scala Parallel Collections
• Hadoop have been called the:
  • “Assembly language of MapReduce programming”
  • “EJB of our time”
Big Data
     Batch processing doesn't cut it
• Ala Hadoop
• We need real-time data processing
• Examples: Spark, Storm, S4 etc.
• Watch“Why Big Data Needs To Be Functional”
  by Dean Wampler
Go Big
 DB
When is
 a RDBMS
   not
good enough?
Scalingreads
  to a RDBMS
  is   hard
Scalingwrites
   to a RDBMS
is   impossible
Do we
really need
a RDBMS?
Do we
really need
a RDBMS?
Sometimes...
Do we
really need
a RDBMS?
Do we
      really need
      a RDBMS?
But many times we don’t
Atomic
Consistent
Isolated
Durable
Availability
    vs
Consistency
Brewer’s



CAP
theorem
You can only pick   2
     Consistency
     Availability
     Partition tolerance
At a given point in time
Centralized system
• In a centralized system (RDBMS etc.)
  we don’t have network partitions,
  e.g. P in CAP
• So you get both:

        Consistency
        Availability
Distributed system
• In a distributed (scalable) system
  we will have network partitions,
  e.g. P in CAP
• So you get to only pick one:

       Consistency
       Availability
Basically Available
Soft state
Eventually consistent
Think about your data
             Then think again
• When do you need ACID?
• When is Eventual Consistency a better fit?
• Different kinds of data has different needs
• You need full consistency less than you think
How fast is fast enough?
• Never guess: Measure, measure and measure
• Start by defining a baseline
 • Where are we now?
• Define what is “good enough” - i.e. SLAs
 • Where do we want to go?
 • When are we done?
• Beware of micro-benchmarks
...or, when can we go for a beer?
• Never guess: Measure, measure and measure
• Start by defining a baseline
 • Where are we now?
• Define what is “good enough” - i.e. SLAs
 • Where do we want to go?
 • When are we done?
• Beware of micro-benchmarks
To sum things up...
1. Maximizing a specific metric impacts others
  • Every strategic decision involves a trade-off
  • There's no "silver bullet"
2. Applying yesterday's best practices to the
   problems faced today will lead to:
  • Waste of resources
  • Performance and scalability bottlenecks
  • Unreliable systems
SO
GO
...now home and build yourself
            Scalable,
      Highly Concurrent &
         Fault-Tolerant
            Systems
Thank You
Email: jonas@typesafe.com
Web: typesafe.com
Twitter: @jboner

More Related Content

What's hot

Galera explained 3
Galera explained 3Galera explained 3
Galera explained 3Marco Tusa
 
Consumer offset management in Kafka
Consumer offset management in KafkaConsumer offset management in Kafka
Consumer offset management in KafkaJoel Koshy
 
Netty 세미나
Netty 세미나Netty 세미나
Netty 세미나Jang Hoon
 
Deep Dive into Building Streaming Applications with Apache Pulsar
Deep Dive into Building Streaming Applications with Apache Pulsar Deep Dive into Building Streaming Applications with Apache Pulsar
Deep Dive into Building Streaming Applications with Apache Pulsar Timothy Spann
 
Maxscale switchover, failover, and auto rejoin
Maxscale switchover, failover, and auto rejoinMaxscale switchover, failover, and auto rejoin
Maxscale switchover, failover, and auto rejoinWagner Bianchi
 
Prod-Like Integration Testing for Distributed Containerized Applications
Prod-Like Integration Testing for Distributed Containerized ApplicationsProd-Like Integration Testing for Distributed Containerized Applications
Prod-Like Integration Testing for Distributed Containerized ApplicationsVMware Tanzu
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
 
Overview of Zookeeper, Helix and Kafka (Oakjug)
Overview of Zookeeper, Helix and Kafka (Oakjug)Overview of Zookeeper, Helix and Kafka (Oakjug)
Overview of Zookeeper, Helix and Kafka (Oakjug)Chris Richardson
 
Big Data in Real-Time at Twitter
Big Data in Real-Time at TwitterBig Data in Real-Time at Twitter
Big Data in Real-Time at Twitternkallen
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaJeff Holoman
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafkaemreakis
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystemconfluent
 
Reactive Applications with Apache Pulsar and Spring Boot
Reactive Applications with Apache Pulsar and Spring BootReactive Applications with Apache Pulsar and Spring Boot
Reactive Applications with Apache Pulsar and Spring BootVMware Tanzu
 
Stability Patterns for Microservices
Stability Patterns for MicroservicesStability Patterns for Microservices
Stability Patterns for Microservicespflueras
 
Deploying Confluent Platform for Production
Deploying Confluent Platform for ProductionDeploying Confluent Platform for Production
Deploying Confluent Platform for Productionconfluent
 
A New Way of Thinking | NATS 2.0 & Connectivity
A New Way of Thinking | NATS 2.0 & ConnectivityA New Way of Thinking | NATS 2.0 & Connectivity
A New Way of Thinking | NATS 2.0 & ConnectivityNATS
 

What's hot (20)

Galera explained 3
Galera explained 3Galera explained 3
Galera explained 3
 
Kafka at scale facebook israel
Kafka at scale   facebook israelKafka at scale   facebook israel
Kafka at scale facebook israel
 
Consumer offset management in Kafka
Consumer offset management in KafkaConsumer offset management in Kafka
Consumer offset management in Kafka
 
Netty 세미나
Netty 세미나Netty 세미나
Netty 세미나
 
with NATS with Kubernetesの世界へ
with NATS with Kubernetesの世界へwith NATS with Kubernetesの世界へ
with NATS with Kubernetesの世界へ
 
Deep Dive into Building Streaming Applications with Apache Pulsar
Deep Dive into Building Streaming Applications with Apache Pulsar Deep Dive into Building Streaming Applications with Apache Pulsar
Deep Dive into Building Streaming Applications with Apache Pulsar
 
Maxscale switchover, failover, and auto rejoin
Maxscale switchover, failover, and auto rejoinMaxscale switchover, failover, and auto rejoin
Maxscale switchover, failover, and auto rejoin
 
Prod-Like Integration Testing for Distributed Containerized Applications
Prod-Like Integration Testing for Distributed Containerized ApplicationsProd-Like Integration Testing for Distributed Containerized Applications
Prod-Like Integration Testing for Distributed Containerized Applications
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
 
Overview of Zookeeper, Helix and Kafka (Oakjug)
Overview of Zookeeper, Helix and Kafka (Oakjug)Overview of Zookeeper, Helix and Kafka (Oakjug)
Overview of Zookeeper, Helix and Kafka (Oakjug)
 
Big Data in Real-Time at Twitter
Big Data in Real-Time at TwitterBig Data in Real-Time at Twitter
Big Data in Real-Time at Twitter
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystem
 
Reactive Applications with Apache Pulsar and Spring Boot
Reactive Applications with Apache Pulsar and Spring BootReactive Applications with Apache Pulsar and Spring Boot
Reactive Applications with Apache Pulsar and Spring Boot
 
Stability Patterns for Microservices
Stability Patterns for MicroservicesStability Patterns for Microservices
Stability Patterns for Microservices
 
Deploying Confluent Platform for Production
Deploying Confluent Platform for ProductionDeploying Confluent Platform for Production
Deploying Confluent Platform for Production
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
A New Way of Thinking | NATS 2.0 & Connectivity
A New Way of Thinking | NATS 2.0 & ConnectivityA New Way of Thinking | NATS 2.0 & Connectivity
A New Way of Thinking | NATS 2.0 & Connectivity
 

Viewers also liked

Akka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based ApplicationsAkka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based ApplicationsNLJUG
 
Akka cluster overview at 010dev
Akka cluster overview at 010devAkka cluster overview at 010dev
Akka cluster overview at 010devRoland Kuhn
 
Slides - Intro to Akka.Cluster
Slides - Intro to Akka.ClusterSlides - Intro to Akka.Cluster
Slides - Intro to Akka.Clusterpetabridge
 
Introduction to Actor Model and Akka
Introduction to Actor Model and AkkaIntroduction to Actor Model and Akka
Introduction to Actor Model and AkkaYung-Lin Ho
 
Introduction to Akka
Introduction to AkkaIntroduction to Akka
Introduction to AkkaJohan Andrén
 
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...Jonas Bonér
 
Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)Jonas Bonér
 

Viewers also liked (7)

Akka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based ApplicationsAkka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based Applications
 
Akka cluster overview at 010dev
Akka cluster overview at 010devAkka cluster overview at 010dev
Akka cluster overview at 010dev
 
Slides - Intro to Akka.Cluster
Slides - Intro to Akka.ClusterSlides - Intro to Akka.Cluster
Slides - Intro to Akka.Cluster
 
Introduction to Actor Model and Akka
Introduction to Actor Model and AkkaIntroduction to Actor Model and Akka
Introduction to Actor Model and Akka
 
Introduction to Akka
Introduction to AkkaIntroduction to Akka
Introduction to Akka
 
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
 
Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)
 

More from Jonas Bonér

The Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native ApplicationsThe Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native ApplicationsJonas Bonér
 
Cloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful ServerlessCloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful ServerlessJonas Bonér
 
Designing Events-first Microservices
Designing Events-first MicroservicesDesigning Events-first Microservices
Designing Events-first MicroservicesJonas Bonér
 
How Events Are Reshaping Modern Systems
How Events Are Reshaping Modern SystemsHow Events Are Reshaping Modern Systems
How Events Are Reshaping Modern SystemsJonas Bonér
 
Reactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at ScaleReactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at ScaleJonas Bonér
 
From Microliths To Microsystems
From Microliths To MicrosystemsFrom Microliths To Microsystems
From Microliths To MicrosystemsJonas Bonér
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersJonas Bonér
 
Life Beyond the Illusion of Present
Life Beyond the Illusion of PresentLife Beyond the Illusion of Present
Life Beyond the Illusion of PresentJonas Bonér
 
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsGo Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsJonas Bonér
 
Reactive Supply To Changing Demand
Reactive Supply To Changing DemandReactive Supply To Changing Demand
Reactive Supply To Changing DemandJonas Bonér
 
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsGo Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsJonas Bonér
 
Event Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveEvent Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveJonas Bonér
 
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVMState: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVMJonas Bonér
 
Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)Jonas Bonér
 

More from Jonas Bonér (15)

The Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native ApplicationsThe Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native Applications
 
Cloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful ServerlessCloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful Serverless
 
Designing Events-first Microservices
Designing Events-first MicroservicesDesigning Events-first Microservices
Designing Events-first Microservices
 
How Events Are Reshaping Modern Systems
How Events Are Reshaping Modern SystemsHow Events Are Reshaping Modern Systems
How Events Are Reshaping Modern Systems
 
Reactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at ScaleReactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at Scale
 
From Microliths To Microsystems
From Microliths To MicrosystemsFrom Microliths To Microsystems
From Microliths To Microsystems
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else Matters
 
Life Beyond the Illusion of Present
Life Beyond the Illusion of PresentLife Beyond the Illusion of Present
Life Beyond the Illusion of Present
 
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsGo Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
 
Reactive Supply To Changing Demand
Reactive Supply To Changing DemandReactive Supply To Changing Demand
Reactive Supply To Changing Demand
 
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsGo Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
 
Introducing Akka
Introducing AkkaIntroducing Akka
Introducing Akka
 
Event Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveEvent Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspective
 
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVMState: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
 
Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)
 

Recently uploaded

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 

Recently uploaded (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 

Building Scalable, Highly Concurrent & Fault Tolerant Systems - Lessons Learned

  • 1. Building Scalable, Highly Concurrent & Fault-Tolerant Systems: Lessons Learned Jonas Bonér CTO Typesafe Twitter: @jboner
  • 2. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 3. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 4. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Agony I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 5. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Agony and Pain I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again lots of Pain I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 6. Agenda • It’s All Trade-offs • Go Concurrent • Go Reactive • Go Fault-Tolerant • Go Distributed • Go Big
  • 9. Performance vs Scalability
  • 10. Latency vs Throughput
  • 11. Availability vs Consistency
  • 14. Shared mutable state Together with threads...
  • 15. Shared mutable state Together with threads... ...leads to
  • 16. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to
  • 17. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to ...and the root of all EVIL
  • 18. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to ...and the root of all EVIL Please, avoid it at all cost
  • 19. Shared mutable state B L E T U ! A Together with threads... M e!! IM at EVIL ...code that is totally INDETERMINISTIC e t ...leads to s s ...and the root of all U avoid it at all cost Please,
  • 20. The problem with locks • Locks do not compose • Locks break encapsulation • Taking too few locks • Taking too many locks • Taking the wrong locks • Taking locks in the wrong order • Error recovery is hard
  • 21. You deserve better tools • Dataflow Concurrency • Actors • Software Transactional Memory (STM) • Agents
  • 22. Dataflow Concurrency • Deterministic • Declarative • Data-driven • Threads are suspended until data is available • Lazy & On-demand • No difference between: • Concurrent code • Sequential code • Examples: Akka & GPars
  • 23. Actors •Share NOTHING •Isolated lightweight event-based processes •Each actor has a mailbox (message queue) •Communicates through asynchronous and non-blocking message passing •Location transparent (distributable) •Examples: Akka & Erlang
  • 24. STM • See the memory as a transactional dataset • Similar to a DB: begin, commit, rollback (ACI) • Transactions are retried upon collision • Rolls back the memory on abort • Transactions can nest and compose • Use STM instead of abusing your database with temporary storage of “scratch” data • Examples: Haskell, Clojure & Scala
  • 25. Agents • Reactive memory cells (STM Ref) • Send a update function to the Agent, which 1. adds it to an (ordered) queue, to be 2. applied to the Agent asynchronously • Reads are “free”, just dereferences the Ref • Cooperates with STM • Examples: Clojure & Akka
  • 26. If we could start all over...
  • 27. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core
  • 28. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming
  • 29. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow
  • 30. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed
  • 31. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming
  • 32. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed
  • 33. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM)
  • 34. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM) 4. Finally - only if really needed
  • 35. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM) 4. Finally - only if really needed • Add Monitors (Locks) and explicit Threads
  • 37. Never block • ...unless you really have to • Blocking kills scalability (and performance) • Never sit on resources you don’t use • Use non-blocking IO • Be reactive • How?
  • 38. Go Async Design for reactive event-driven systems 1. Use asynchronous message passing 2. Use Iteratee-based IO 3. Use push not pull (or poll) • Examples: • Akka or Erlang actors • Play’s reactive Iteratee IO • Node.js or JavaScript Promises • Server-Sent Events or WebSockets • Scala’s Futures library
  • 40. Failure Recovery in Java/C/C# etc.
  • 41. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control
  • 42. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed
  • 43. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread
  • 44. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed
  • 45. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with:
  • 46. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic
  • 47. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic • SCATTERED all over the code base
  • 48. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control d o • If this thread blows up you are screwed n !! • So you need to do all explicit error handling a ! c r WITHIN this single thread e te • To make things worse - errors do not W et propagate between threads so there is NO WAY OF EVEN FINDING OUT that b something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic • SCATTERED all over the code base
  • 51. The right way 1. Isolated lightweight processes 2. Supervised processes • Each running process has a supervising process • Errors are sent to the supervisor (asynchronously) • Supervisor manages the failure • Same semantics local as remote • For example the Actor Model solves it nicely
  • 53. Performance vs Scalability
  • 54. How do I know if I have a performance problem?
  • 55. How do I know if I have a performance problem? If your system is slow for a single user
  • 56. How do I know if I have a scalability problem?
  • 57. How do I know if I have a scalability problem? If your system is fast for a single user but slow under heavy load
  • 58. (Three) Misconceptions about Reliable Distributed Computing - Werner Vogels 1. Transparency is the ultimate goal 2. Automatic object replication is desirable 3. All replicas are equal and deterministic Classic paper: A Note On Distributed Computing - Waldo et. al.
  • 59. Fallacy 1 Transparent Distributed Computing • Emulating Consistency and Shared Memory in a distributed environment • Distributed Objects • “Sucks like an inverted hurricane” - Martin Fowler • Distributed Transactions • ...don’t get me started...
  • 60. Fallacy 2 RPC • Emulating synchronous blocking method dispatch - across the network • Ignores: • Latency • Partial failures • General scalability concerns, caching etc. • “Convenience over Correctness” - Steve Vinoski
  • 62. Instead Embrace the Network Use it i th w Asynchronous don e Message be d an Passing
  • 63. Guaranteed Delivery Delivery Semantics • No guarantees • At most once • At least once • Once and only once
  • 66. The network is inherently unreliable and there is no such thing as 100% guaranteed delivery It’s all lies.
  • 68. Guaranteed Delivery The question is what to guarantee
  • 69. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network?
  • 70. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC?
  • 71. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue?
  • 72. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver?
  • 73. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver? 5. The message is - starting to be processed by the receiver?
  • 74. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver? 5. The message is - starting to be processed by the receiver? 6. The message is - has completed processing by the receiver?
  • 75. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one
  • 76. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one Different USE-CASES Different GUARANTEES Different COSTS
  • 77. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one Different USE-CASES Different GUARANTEES Different COSTS For each additional guarantee you add you will either: • decrease performance, throughput or scalability • increase latency
  • 78. Just
  • 80. Just Use ACKing and be done with it
  • 81. Latency vs Throughput
  • 82. You should strive for maximal throughput with acceptable latency
  • 85. Big Data Imperative OO programming doesn't cut it • Object-Mathematics Impedance Mismatch • We need functional processing, transformations etc. • Examples: Spark, Crunch/Scrunch, Cascading, Cascalog, Scalding, Scala Parallel Collections • Hadoop have been called the: • “Assembly language of MapReduce programming” • “EJB of our time”
  • 86. Big Data Batch processing doesn't cut it • Ala Hadoop • We need real-time data processing • Examples: Spark, Storm, S4 etc. • Watch“Why Big Data Needs To Be Functional” by Dean Wampler
  • 88. When is a RDBMS not good enough?
  • 89. Scalingreads to a RDBMS is hard
  • 90. Scalingwrites to a RDBMS is impossible
  • 92. Do we really need a RDBMS? Sometimes...
  • 94. Do we really need a RDBMS? But many times we don’t
  • 96. Availability vs Consistency
  • 98. You can only pick 2 Consistency Availability Partition tolerance At a given point in time
  • 99. Centralized system • In a centralized system (RDBMS etc.) we don’t have network partitions, e.g. P in CAP • So you get both: Consistency Availability
  • 100. Distributed system • In a distributed (scalable) system we will have network partitions, e.g. P in CAP • So you get to only pick one: Consistency Availability
  • 102. Think about your data Then think again • When do you need ACID? • When is Eventual Consistency a better fit? • Different kinds of data has different needs • You need full consistency less than you think
  • 103. How fast is fast enough? • Never guess: Measure, measure and measure • Start by defining a baseline • Where are we now? • Define what is “good enough” - i.e. SLAs • Where do we want to go? • When are we done? • Beware of micro-benchmarks
  • 104. ...or, when can we go for a beer? • Never guess: Measure, measure and measure • Start by defining a baseline • Where are we now? • Define what is “good enough” - i.e. SLAs • Where do we want to go? • When are we done? • Beware of micro-benchmarks
  • 105. To sum things up... 1. Maximizing a specific metric impacts others • Every strategic decision involves a trade-off • There's no "silver bullet" 2. Applying yesterday's best practices to the problems faced today will lead to: • Waste of resources • Performance and scalability bottlenecks • Unreliable systems
  • 106. SO
  • 107. GO
  • 108. ...now home and build yourself Scalable, Highly Concurrent & Fault-Tolerant Systems
  • 109. Thank You Email: jonas@typesafe.com Web: typesafe.com Twitter: @jboner