SlideShare a Scribd company logo
1 of 109
Download to read offline
Building
      Scalable,
Highly Concurrent &
   Fault-Tolerant
      Systems:
  Lessons Learned

     Jonas Bonér
       CTO Typesafe
      Twitter: @jboner
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                      Agony
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                      Agony
                                                                                                    and Pain
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                    lots of
                                                                                                      Pain
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
Agenda
• It’s All Trade-offs
• Go Concurrent
• Go Reactive
• Go Fault-Tolerant
• Go Distributed
• Go Big
It’s all
Trade-offs
Performance
     vs
 Scalability
Latency
     vs
Throughput
Availability
    vs
Consistency
Go Concurrent
Shared mutable state
Shared mutable state
Together with threads...
Shared mutable state
              Together with threads...


...leads to
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
              ...and the root of all EVIL
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
              ...and the root of all EVIL


Please, avoid it at all cost
Shared mutable state
                     B L E
                 T
               U ! A
     Together with threads...

            M e!!
        IM at EVIL
              ...code that is totally INDETERMINISTIC

      e t
...leads to


     s s      ...and the root of all

    U avoid it at all cost
Please,
The problem with locks
• Locks do not compose
• Locks break encapsulation
• Taking too few locks
• Taking too many locks
• Taking the wrong locks
• Taking locks in the wrong order
• Error recovery is hard
You deserve better tools

• Dataflow Concurrency
• Actors
• Software Transactional Memory (STM)
• Agents
Dataflow Concurrency
• Deterministic
• Declarative
• Data-driven
  • Threads are suspended until data is available
  • Lazy & On-demand
• No difference between:
 • Concurrent code
 • Sequential code
• Examples: Akka & GPars
Actors
•Share NOTHING
•Isolated lightweight event-based processes
•Each actor has a mailbox (message queue)
•Communicates through asynchronous and
 non-blocking message passing
•Location transparent (distributable)
•Examples: Akka & Erlang
STM
• See the memory as a transactional dataset
• Similar to a DB: begin, commit, rollback (ACI)
• Transactions are retried upon collision
• Rolls back the memory on abort
• Transactions can nest and compose
• Use STM instead of abusing your database
  with temporary storage of “scratch” data
• Examples: Haskell, Clojure & Scala
Agents
• Reactive memory cells (STM Ref)
• Send a update function to the Agent, which
  1. adds it to an (ordered) queue, to be
  2. applied to the Agent asynchronously
• Reads are “free”, just dereferences the Ref
• Cooperates with STM
• Examples: Clojure & Akka
If we could start all over...
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
4. Finally - only if really needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
4. Finally - only if really needed
  • Add Monitors (Locks) and explicit Threads
Go Reactive
Never block
• ...unless you really have to
• Blocking kills scalability (and performance)
• Never sit on resources you don’t use
• Use non-blocking IO
• Be reactive
• How?
Go Async
  Design for reactive event-driven systems
1. Use asynchronous message passing
2. Use Iteratee-based IO
3. Use push not pull (or poll)
• Examples:
   • Akka or Erlang actors
   • Play’s reactive Iteratee IO
   • Node.js or JavaScript Promises
   • Server-Sent Events or WebSockets
   • Scala’s Futures library
Go Fault-Tolerant
Failure Recovery in Java/C/C# etc.
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
  • SCATTERED all over the code base
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control


                d o
• If this thread blows up you are screwed

               n !!
• So you need to do all explicit error handling

              a !
            c r
  WITHIN this single thread

           e te
• To make things worse - errors do not

          W et
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that

            b
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
  • SCATTERED all over the code base
Just

Let It Crash
The right way
1. Isolated lightweight processes
2. Supervised processes
 • Each running process has a supervising process
 • Errors are sent to the supervisor (asynchronously)
 • Supervisor manages the failure
• Same semantics local as remote
• For example the Actor Model solves it nicely
Go Distributed
Performance
     vs
 Scalability
How do I know if I have a
 performance problem?
How do I know if I have a
 performance problem?

       If your system is
    slow for a single user
How do I know if I have a
  scalability problem?
How do I know if I have a
  scalability problem?

        If your system is
     fast for a single user
  but slow under heavy load
(Three) Misconceptions about
     Reliable Distributed Computing
                     - Werner Vogels

1. Transparency is the ultimate goal
2. Automatic object replication is desirable
3. All replicas are equal and deterministic

  Classic paper: A Note On Distributed Computing - Waldo et. al.
Fallacy 1
Transparent Distributed Computing
• Emulating Consistency and Shared
  Memory in a distributed environment
• Distributed Objects
 • “Sucks like an inverted hurricane” - Martin Fowler
• Distributed Transactions
 • ...don’t get me started...
Fallacy 2
                          RPC
• Emulating synchronous blocking method
  dispatch - across the network
• Ignores:
 • Latency
 • Partial failures
 • General scalability concerns, caching etc.
• “Convenience over Correctness” - Steve Vinoski
Instead
Instead
   Embrace the Network

    Use                                    it
                                    i th
                                   w
Asynchronous               don e

  Message             be
                  d
               an
   Passing
Guaranteed Delivery
  Delivery Semantics
   • No guarantees
   • At most once
   • At least once
   • Once and only once
It’s all lies.
It’s all lies.
The network is inherently unreliable
 and there is no such thing as 100%
        guaranteed delivery


       It’s all lies.
Guaranteed Delivery
Guaranteed Delivery
The question is what to guarantee
Guaranteed Delivery
  The question is what to guarantee
1. The message is - sent out on the network?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
5. The message is - starting to be processed by the receiver?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
5. The message is - starting to be processed by the receiver?
6. The message is - has completed processing by the receiver?
Ok, then what to do?
1. Start with 0 guarantees (0 additional cost)
2. Add the guarantees you need - one by one
Ok, then what to do?
1. Start with 0 guarantees (0 additional cost)
2. Add the guarantees you need - one by one

  Different USE-CASES
            Different GUARANTEES
                Different COSTS
Ok, then what to do?
    1. Start with 0 guarantees (0 additional cost)
    2. Add the guarantees you need - one by one

       Different USE-CASES
                  Different GUARANTEES
                       Different COSTS
 For each additional guarantee you add you will either:
• decrease performance, throughput or scalability
• increase latency
Just
Just


Use ACKing
Just


Use ACKing
and be done with it
Latency
     vs
Throughput
You should strive for
maximal throughput
          with
acceptable latency
Go Big
Go Big
 Data
Big Data
  Imperative OO programming doesn't cut it
• Object-Mathematics Impedance Mismatch
• We need functional processing, transformations etc.
• Examples: Spark, Crunch/Scrunch, Cascading, Cascalog,
  Scalding, Scala Parallel Collections
• Hadoop have been called the:
  • “Assembly language of MapReduce programming”
  • “EJB of our time”
Big Data
     Batch processing doesn't cut it
• Ala Hadoop
• We need real-time data processing
• Examples: Spark, Storm, S4 etc.
• Watch“Why Big Data Needs To Be Functional”
  by Dean Wampler
Go Big
 DB
When is
 a RDBMS
   not
good enough?
Scalingreads
  to a RDBMS
  is   hard
Scalingwrites
   to a RDBMS
is   impossible
Do we
really need
a RDBMS?
Do we
really need
a RDBMS?
Sometimes...
Do we
really need
a RDBMS?
Do we
      really need
      a RDBMS?
But many times we don’t
Atomic
Consistent
Isolated
Durable
Availability
    vs
Consistency
Brewer’s



CAP
theorem
You can only pick   2
     Consistency
     Availability
     Partition tolerance
At a given point in time
Centralized system
• In a centralized system (RDBMS etc.)
  we don’t have network partitions,
  e.g. P in CAP
• So you get both:

        Consistency
        Availability
Distributed system
• In a distributed (scalable) system
  we will have network partitions,
  e.g. P in CAP
• So you get to only pick one:

       Consistency
       Availability
Basically Available
Soft state
Eventually consistent
Think about your data
             Then think again
• When do you need ACID?
• When is Eventual Consistency a better fit?
• Different kinds of data has different needs
• You need full consistency less than you think
How fast is fast enough?
• Never guess: Measure, measure and measure
• Start by defining a baseline
 • Where are we now?
• Define what is “good enough” - i.e. SLAs
 • Where do we want to go?
 • When are we done?
• Beware of micro-benchmarks
...or, when can we go for a beer?
• Never guess: Measure, measure and measure
• Start by defining a baseline
 • Where are we now?
• Define what is “good enough” - i.e. SLAs
 • Where do we want to go?
 • When are we done?
• Beware of micro-benchmarks
To sum things up...
1. Maximizing a specific metric impacts others
  • Every strategic decision involves a trade-off
  • There's no "silver bullet"
2. Applying yesterday's best practices to the
   problems faced today will lead to:
  • Waste of resources
  • Performance and scalability bottlenecks
  • Unreliable systems
SO
GO
...now home and build yourself
            Scalable,
      Highly Concurrent &
         Fault-Tolerant
            Systems
Thank You
Email: jonas@typesafe.com
Web: typesafe.com
Twitter: @jboner

More Related Content

What's hot

Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptxAlex Ivy
 
HA/DR options with SQL Server in Azure and hybrid
HA/DR options with SQL Server in Azure and hybridHA/DR options with SQL Server in Azure and hybrid
HA/DR options with SQL Server in Azure and hybridJames Serra
 
Exactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsExactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsGuozhang Wang
 
Introduction to Apache Cassandra
Introduction to Apache CassandraIntroduction to Apache Cassandra
Introduction to Apache CassandraRobert Stupp
 
Cassandra an overview
Cassandra an overviewCassandra an overview
Cassandra an overviewPritamKathar
 
2019 DevSecOps Reference Architectures
2019 DevSecOps Reference Architectures2019 DevSecOps Reference Architectures
2019 DevSecOps Reference ArchitecturesSonatype
 
Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016Markus Höfer
 
From Mainframe to Microservice: An Introduction to Distributed Systems
From Mainframe to Microservice: An Introduction to Distributed SystemsFrom Mainframe to Microservice: An Introduction to Distributed Systems
From Mainframe to Microservice: An Introduction to Distributed SystemsTyler Treat
 
Introduction to Amazon Relational Database Service (Amazon RDS)
Introduction to Amazon Relational Database Service (Amazon RDS)Introduction to Amazon Relational Database Service (Amazon RDS)
Introduction to Amazon Relational Database Service (Amazon RDS)Amazon Web Services
 
Open Source DataViz with Apache Superset
Open Source DataViz with Apache SupersetOpen Source DataViz with Apache Superset
Open Source DataViz with Apache SupersetCarl W. Handlin
 
Micro services Architecture
Micro services ArchitectureMicro services Architecture
Micro services ArchitectureAraf Karsh Hamid
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Databricks
 
Under the Hood of a Shard-per-Core Database Architecture
Under the Hood of a Shard-per-Core Database ArchitectureUnder the Hood of a Shard-per-Core Database Architecture
Under the Hood of a Shard-per-Core Database ArchitectureScyllaDB
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.
 
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...confluent
 

What's hot (20)

Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
 
HA/DR options with SQL Server in Azure and hybrid
HA/DR options with SQL Server in Azure and hybridHA/DR options with SQL Server in Azure and hybrid
HA/DR options with SQL Server in Azure and hybrid
 
Exactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsExactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka Streams
 
Lost with data consistency
Lost with data consistencyLost with data consistency
Lost with data consistency
 
Multi tenant architecture
Multi tenant architectureMulti tenant architecture
Multi tenant architecture
 
Introduction to Apache Cassandra
Introduction to Apache CassandraIntroduction to Apache Cassandra
Introduction to Apache Cassandra
 
Deep Dive on Amazon DynamoDB
Deep Dive on Amazon DynamoDB Deep Dive on Amazon DynamoDB
Deep Dive on Amazon DynamoDB
 
Cassandra an overview
Cassandra an overviewCassandra an overview
Cassandra an overview
 
2019 DevSecOps Reference Architectures
2019 DevSecOps Reference Architectures2019 DevSecOps Reference Architectures
2019 DevSecOps Reference Architectures
 
Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016
 
From Mainframe to Microservice: An Introduction to Distributed Systems
From Mainframe to Microservice: An Introduction to Distributed SystemsFrom Mainframe to Microservice: An Introduction to Distributed Systems
From Mainframe to Microservice: An Introduction to Distributed Systems
 
Introduction to Amazon Relational Database Service (Amazon RDS)
Introduction to Amazon Relational Database Service (Amazon RDS)Introduction to Amazon Relational Database Service (Amazon RDS)
Introduction to Amazon Relational Database Service (Amazon RDS)
 
Open Source DataViz with Apache Superset
Open Source DataViz with Apache SupersetOpen Source DataViz with Apache Superset
Open Source DataViz with Apache Superset
 
Micro services Architecture
Micro services ArchitectureMicro services Architecture
Micro services Architecture
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
 
Under the Hood of a Shard-per-Core Database Architecture
Under the Hood of a Shard-per-Core Database ArchitectureUnder the Hood of a Shard-per-Core Database Architecture
Under the Hood of a Shard-per-Core Database Architecture
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
 
Big Data Architectural Patterns
Big Data Architectural PatternsBig Data Architectural Patterns
Big Data Architectural Patterns
 
Cassandra Database
Cassandra DatabaseCassandra Database
Cassandra Database
 
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...
 

Viewers also liked

Akka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based ApplicationsAkka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based ApplicationsNLJUG
 
Akka cluster overview at 010dev
Akka cluster overview at 010devAkka cluster overview at 010dev
Akka cluster overview at 010devRoland Kuhn
 
Slides - Intro to Akka.Cluster
Slides - Intro to Akka.ClusterSlides - Intro to Akka.Cluster
Slides - Intro to Akka.Clusterpetabridge
 
Introduction to Actor Model and Akka
Introduction to Actor Model and AkkaIntroduction to Actor Model and Akka
Introduction to Actor Model and AkkaYung-Lin Ho
 
Introduction to Akka
Introduction to AkkaIntroduction to Akka
Introduction to AkkaJohan Andrén
 
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...Jonas Bonér
 
Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)Jonas Bonér
 

Viewers also liked (7)

Akka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based ApplicationsAkka in Practice: Designing Actor-based Applications
Akka in Practice: Designing Actor-based Applications
 
Akka cluster overview at 010dev
Akka cluster overview at 010devAkka cluster overview at 010dev
Akka cluster overview at 010dev
 
Slides - Intro to Akka.Cluster
Slides - Intro to Akka.ClusterSlides - Intro to Akka.Cluster
Slides - Intro to Akka.Cluster
 
Introduction to Actor Model and Akka
Introduction to Actor Model and AkkaIntroduction to Actor Model and Akka
Introduction to Actor Model and Akka
 
Introduction to Akka
Introduction to AkkaIntroduction to Akka
Introduction to Akka
 
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
 
Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)Building Reactive Systems with Akka (in Java 8 or Scala)
Building Reactive Systems with Akka (in Java 8 or Scala)
 

More from Jonas Bonér

We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?Jonas Bonér
 
Kalix: Tackling the The Cloud to Edge Continuum
Kalix: Tackling the The Cloud to Edge ContinuumKalix: Tackling the The Cloud to Edge Continuum
Kalix: Tackling the The Cloud to Edge ContinuumJonas Bonér
 
The Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native ApplicationsThe Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native ApplicationsJonas Bonér
 
Cloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful ServerlessCloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful ServerlessJonas Bonér
 
Designing Events-first Microservices
Designing Events-first MicroservicesDesigning Events-first Microservices
Designing Events-first MicroservicesJonas Bonér
 
How Events Are Reshaping Modern Systems
How Events Are Reshaping Modern SystemsHow Events Are Reshaping Modern Systems
How Events Are Reshaping Modern SystemsJonas Bonér
 
Reactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at ScaleReactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at ScaleJonas Bonér
 
From Microliths To Microsystems
From Microliths To MicrosystemsFrom Microliths To Microsystems
From Microliths To MicrosystemsJonas Bonér
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersJonas Bonér
 
Life Beyond the Illusion of Present
Life Beyond the Illusion of PresentLife Beyond the Illusion of Present
Life Beyond the Illusion of PresentJonas Bonér
 
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsGo Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsJonas Bonér
 
Reactive Supply To Changing Demand
Reactive Supply To Changing DemandReactive Supply To Changing Demand
Reactive Supply To Changing DemandJonas Bonér
 
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsGo Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsJonas Bonér
 
Event Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveEvent Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveJonas Bonér
 
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVMState: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVMJonas Bonér
 
Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)Jonas Bonér
 

More from Jonas Bonér (17)

We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?
 
Kalix: Tackling the The Cloud to Edge Continuum
Kalix: Tackling the The Cloud to Edge ContinuumKalix: Tackling the The Cloud to Edge Continuum
Kalix: Tackling the The Cloud to Edge Continuum
 
The Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native ApplicationsThe Reactive Principles: Design Principles For Cloud Native Applications
The Reactive Principles: Design Principles For Cloud Native Applications
 
Cloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful ServerlessCloudstate—Towards Stateful Serverless
Cloudstate—Towards Stateful Serverless
 
Designing Events-first Microservices
Designing Events-first MicroservicesDesigning Events-first Microservices
Designing Events-first Microservices
 
How Events Are Reshaping Modern Systems
How Events Are Reshaping Modern SystemsHow Events Are Reshaping Modern Systems
How Events Are Reshaping Modern Systems
 
Reactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at ScaleReactive Microsystems: The Evolution of Microservices at Scale
Reactive Microsystems: The Evolution of Microservices at Scale
 
From Microliths To Microsystems
From Microliths To MicrosystemsFrom Microliths To Microsystems
From Microliths To Microsystems
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else Matters
 
Life Beyond the Illusion of Present
Life Beyond the Illusion of PresentLife Beyond the Illusion of Present
Life Beyond the Illusion of Present
 
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsGo Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
 
Reactive Supply To Changing Demand
Reactive Supply To Changing DemandReactive Supply To Changing Demand
Reactive Supply To Changing Demand
 
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsGo Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
 
Introducing Akka
Introducing AkkaIntroducing Akka
Introducing Akka
 
Event Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveEvent Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspective
 
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVMState: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
 
Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)Pragmatic Real-World Scala (short version)
Pragmatic Real-World Scala (short version)
 

Recently uploaded

Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDropbox
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsNanddeep Nachan
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024The Digital Insurer
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...apidays
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 

Recently uploaded (20)

Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 

Building Scalable, Highly Concurrent & Fault Tolerant Systems - Lessons Learned

  • 1. Building Scalable, Highly Concurrent & Fault-Tolerant Systems: Lessons Learned Jonas Bonér CTO Typesafe Twitter: @jboner
  • 2. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 3. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 4. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Agony I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 5. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Agony and Pain I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again lots of Pain I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 6. Agenda • It’s All Trade-offs • Go Concurrent • Go Reactive • Go Fault-Tolerant • Go Distributed • Go Big
  • 7.
  • 9. Performance vs Scalability
  • 10. Latency vs Throughput
  • 11. Availability vs Consistency
  • 14. Shared mutable state Together with threads...
  • 15. Shared mutable state Together with threads... ...leads to
  • 16. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to
  • 17. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to ...and the root of all EVIL
  • 18. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to ...and the root of all EVIL Please, avoid it at all cost
  • 19. Shared mutable state B L E T U ! A Together with threads... M e!! IM at EVIL ...code that is totally INDETERMINISTIC e t ...leads to s s ...and the root of all U avoid it at all cost Please,
  • 20. The problem with locks • Locks do not compose • Locks break encapsulation • Taking too few locks • Taking too many locks • Taking the wrong locks • Taking locks in the wrong order • Error recovery is hard
  • 21. You deserve better tools • Dataflow Concurrency • Actors • Software Transactional Memory (STM) • Agents
  • 22. Dataflow Concurrency • Deterministic • Declarative • Data-driven • Threads are suspended until data is available • Lazy & On-demand • No difference between: • Concurrent code • Sequential code • Examples: Akka & GPars
  • 23. Actors •Share NOTHING •Isolated lightweight event-based processes •Each actor has a mailbox (message queue) •Communicates through asynchronous and non-blocking message passing •Location transparent (distributable) •Examples: Akka & Erlang
  • 24. STM • See the memory as a transactional dataset • Similar to a DB: begin, commit, rollback (ACI) • Transactions are retried upon collision • Rolls back the memory on abort • Transactions can nest and compose • Use STM instead of abusing your database with temporary storage of “scratch” data • Examples: Haskell, Clojure & Scala
  • 25. Agents • Reactive memory cells (STM Ref) • Send a update function to the Agent, which 1. adds it to an (ordered) queue, to be 2. applied to the Agent asynchronously • Reads are “free”, just dereferences the Ref • Cooperates with STM • Examples: Clojure & Akka
  • 26. If we could start all over...
  • 27. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core
  • 28. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming
  • 29. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow
  • 30. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed
  • 31. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming
  • 32. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed
  • 33. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM)
  • 34. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM) 4. Finally - only if really needed
  • 35. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM) 4. Finally - only if really needed • Add Monitors (Locks) and explicit Threads
  • 37. Never block • ...unless you really have to • Blocking kills scalability (and performance) • Never sit on resources you don’t use • Use non-blocking IO • Be reactive • How?
  • 38. Go Async Design for reactive event-driven systems 1. Use asynchronous message passing 2. Use Iteratee-based IO 3. Use push not pull (or poll) • Examples: • Akka or Erlang actors • Play’s reactive Iteratee IO • Node.js or JavaScript Promises • Server-Sent Events or WebSockets • Scala’s Futures library
  • 40. Failure Recovery in Java/C/C# etc.
  • 41. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control
  • 42. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed
  • 43. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread
  • 44. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed
  • 45. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with:
  • 46. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic
  • 47. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic • SCATTERED all over the code base
  • 48. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control d o • If this thread blows up you are screwed n !! • So you need to do all explicit error handling a ! c r WITHIN this single thread e te • To make things worse - errors do not W et propagate between threads so there is NO WAY OF EVEN FINDING OUT that b something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic • SCATTERED all over the code base
  • 50.
  • 51. The right way 1. Isolated lightweight processes 2. Supervised processes • Each running process has a supervising process • Errors are sent to the supervisor (asynchronously) • Supervisor manages the failure • Same semantics local as remote • For example the Actor Model solves it nicely
  • 53. Performance vs Scalability
  • 54. How do I know if I have a performance problem?
  • 55. How do I know if I have a performance problem? If your system is slow for a single user
  • 56. How do I know if I have a scalability problem?
  • 57. How do I know if I have a scalability problem? If your system is fast for a single user but slow under heavy load
  • 58. (Three) Misconceptions about Reliable Distributed Computing - Werner Vogels 1. Transparency is the ultimate goal 2. Automatic object replication is desirable 3. All replicas are equal and deterministic Classic paper: A Note On Distributed Computing - Waldo et. al.
  • 59. Fallacy 1 Transparent Distributed Computing • Emulating Consistency and Shared Memory in a distributed environment • Distributed Objects • “Sucks like an inverted hurricane” - Martin Fowler • Distributed Transactions • ...don’t get me started...
  • 60. Fallacy 2 RPC • Emulating synchronous blocking method dispatch - across the network • Ignores: • Latency • Partial failures • General scalability concerns, caching etc. • “Convenience over Correctness” - Steve Vinoski
  • 62. Instead Embrace the Network Use it i th w Asynchronous don e Message be d an Passing
  • 63. Guaranteed Delivery Delivery Semantics • No guarantees • At most once • At least once • Once and only once
  • 66. The network is inherently unreliable and there is no such thing as 100% guaranteed delivery It’s all lies.
  • 68. Guaranteed Delivery The question is what to guarantee
  • 69. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network?
  • 70. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC?
  • 71. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue?
  • 72. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver?
  • 73. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver? 5. The message is - starting to be processed by the receiver?
  • 74. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver? 5. The message is - starting to be processed by the receiver? 6. The message is - has completed processing by the receiver?
  • 75. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one
  • 76. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one Different USE-CASES Different GUARANTEES Different COSTS
  • 77. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one Different USE-CASES Different GUARANTEES Different COSTS For each additional guarantee you add you will either: • decrease performance, throughput or scalability • increase latency
  • 78. Just
  • 80. Just Use ACKing and be done with it
  • 81. Latency vs Throughput
  • 82. You should strive for maximal throughput with acceptable latency
  • 85. Big Data Imperative OO programming doesn't cut it • Object-Mathematics Impedance Mismatch • We need functional processing, transformations etc. • Examples: Spark, Crunch/Scrunch, Cascading, Cascalog, Scalding, Scala Parallel Collections • Hadoop have been called the: • “Assembly language of MapReduce programming” • “EJB of our time”
  • 86. Big Data Batch processing doesn't cut it • Ala Hadoop • We need real-time data processing • Examples: Spark, Storm, S4 etc. • Watch“Why Big Data Needs To Be Functional” by Dean Wampler
  • 88. When is a RDBMS not good enough?
  • 89. Scalingreads to a RDBMS is hard
  • 90. Scalingwrites to a RDBMS is impossible
  • 92. Do we really need a RDBMS? Sometimes...
  • 94. Do we really need a RDBMS? But many times we don’t
  • 96. Availability vs Consistency
  • 98. You can only pick 2 Consistency Availability Partition tolerance At a given point in time
  • 99. Centralized system • In a centralized system (RDBMS etc.) we don’t have network partitions, e.g. P in CAP • So you get both: Consistency Availability
  • 100. Distributed system • In a distributed (scalable) system we will have network partitions, e.g. P in CAP • So you get to only pick one: Consistency Availability
  • 102. Think about your data Then think again • When do you need ACID? • When is Eventual Consistency a better fit? • Different kinds of data has different needs • You need full consistency less than you think
  • 103. How fast is fast enough? • Never guess: Measure, measure and measure • Start by defining a baseline • Where are we now? • Define what is “good enough” - i.e. SLAs • Where do we want to go? • When are we done? • Beware of micro-benchmarks
  • 104. ...or, when can we go for a beer? • Never guess: Measure, measure and measure • Start by defining a baseline • Where are we now? • Define what is “good enough” - i.e. SLAs • Where do we want to go? • When are we done? • Beware of micro-benchmarks
  • 105. To sum things up... 1. Maximizing a specific metric impacts others • Every strategic decision involves a trade-off • There's no "silver bullet" 2. Applying yesterday's best practices to the problems faced today will lead to: • Waste of resources • Performance and scalability bottlenecks • Unreliable systems
  • 106. SO
  • 107. GO
  • 108. ...now home and build yourself Scalable, Highly Concurrent & Fault-Tolerant Systems
  • 109. Thank You Email: jonas@typesafe.com Web: typesafe.com Twitter: @jboner