Fight Complexity with Functional Programming

•

0 likes•50 views

Presented by: Gopal S. Akshintala, Microsoft Presented at All Things Open 2020 Abstract: A Metric-driven approach to reduce Cognitive Complexity in a code base, using Functional Programming, demoed **hands-on**, by solving a complex real-world ubiquitous design challenge - REST API Bulk Request Validation, with an extensible Framework that separates what-to-do (Validations) from how-to-do (Validation Orchestration). Let's do a case study of a successful implementation done by our team in the world's largest SaaS org, _Salesforce_, through our in-house baked FOSS library **Vader**.

Technology

FP is cool 😎, coz
it’s complex enough to
make devs feel like
they are doing
something smart!
@GopalAkshintala
- Some Manager (Not Mine!)

IDE Theme
Light
Vs
Dark
@GopalAkshintala

@GopalAkshintala
Immutability
Referential Transparency
Side-effects @ Boundaries
FP Principles
…………………………………………………
…………………………………………………
OOP Principles

Imperative
Vs
Declarative
The Jotlin
Dev
@GopalAkshintala

Imperative
Code
-
-
-
Imperative
@GopalAkshintala

Let’s Concatenate
last names in a
team with a
delimiter 🤝
lastName1 🤝 lastName2 🤝 lastName2
@GopalAkshintala
Team Building!

@GopalAkshintala
What about
Corner cases?

@GopalAkshintala
lastName1 🤝 lastName2 🤝 lastName2 🤝

Declarative
Imperative
Code
-
-
-
Imperative
Vs
Declarative
@GopalAkshintala

How to Concatenate
Last names in
Parallel?
@GopalAkshintala

@GopalAkshintala
How to make this Parallel?

How-to-dos What-to-dos
Looping through
the list
Aggregating the
results
Validating the
names
Extracting last
names
@GopalAkshintala

The
Context:Core
Philosophy
Do
Things differently
without
doing
Different things!
@GopalAkshintala

@GopalAkshintala
Reading about
Functors, Monads, Applicatives!

Either Monad
EggValidation Failure
Context
@GopalAkshintala
Left Right

Source @GopalAkshintala
Left State
Stays Left

Are Monads used in
Enterprise
Software?
@GopalAkshintala

The
Batch
Validation
Framework
@GopalAkshintala

@GopalAkshintala
Validation layer
Service layer
DB layer
Aggregated Response

Framework
Meta
Requirements
@GopalAkshintala

One Egg
One Validation
Many Eggs
One Validation
@GopalAkshintala

Many Eggs
Many Validations
!
@GopalAkshintala

Chain of Responsibility
@GopalAkshintala
Is FP the Best Fit?

@GopalAkshintala
2D ProblemEggs
Validations

Making of a
Extensible Machine
@GopalAkshintala
Validations Configuration Orchestration
(What-to-do) (How-to-do) (How-to-do)

Heterogeneous
DataTypes in pipeline
@GopalAkshintala

Uniform DataTypes
with Monad Boxes
@GopalAkshintala

Standardize
Validations
@GopalAkshintala

$@GopalAkshintala val validate1Simple: Validator<Egg, ValidationFailure> = { if (simpleRule(it)) true.right() else NO_EGG_TO_VALIDATE_1.left() } validate1Simple fun validate1( badEggFailureBucketMap: MutableMap<Int, ValidationFailure>, eggIndex: Int, iterator: MutableIterator<Egg?>, eggToBeValidated: Egg? ): Boolean { if (!simpleRule(eggToBeValidated)) { iterator.remove() badEggFailureBucketMap[eggIndex] = ValidationFailures.NO_EGG_TO_VALIDATE_1 return false } return true }$

@GopalAkshintala
Validator
typealias Validator<ValidatableT, FailureT>
= suspend (ValidatableT) -> Either<FailureT, Any?>

Transformation
map
flatMap
filter
take…
Aggregation
reduce
groupBy
zip,unzip
fold,foldRight
scan,runningReduce
@GopalAkshintala
Dozen FP Operations for
Daily Programming

FP is NOT Complex
It fights
Cognitive-Complexity
@GopalAkshintala

Min.
Cognitive-Complexity
Same
Time-Complexity
@GopalAkshintala

Railway-Oriented
Programming
@GopalAkshintala

@GopalAkshintala
bit.ly/imp-vs-dec
bit.ly/ro-validation-kt
bit.ly/fcwfp-kt-slides
{ }

@GopalAkshintala
gopal.akshintala@gmail.com

Similar to Fight Complexity with Functional Programming

With the new Apache Arrow integration in PySpark 2.3, it is now starting become reasonable to look to the Python world and ask “what else do we want to steal besides tensorflow”, or as a Python developer look and say “how can I get my code into production without it being rewritten into a mess of Java?” Regardless of your specific side(s) in the JVM/Python divide, collaboration is getting a lot faster, so lets learn how to share! In this brief talk we will examine sharing some of the wonders of Spacy with the Java world, which still has a somewhat lackluster set of options for NLP.

Sharing (or stealing) the jewels of python with big data & the jvm (1)

Holden Karau

Powering tensorflow with big data (apache spark, flink, and beam) dataworks...

Holden Karau

Prototype4Production Presented at FOSSASIA2015 at Singapore

Dhruv Gohil

05092019 The Battle of the IDEs by Ko Turk at the AlmereJUG / Conspect

Ko Turk

Apache Spark has been a great driver of not only Scala adoption, but introducing a new generation of developers to functional programming concepts. As Spark places more emphasis on its newer DataFrame & Dataset APIs, it’s important to ask ourselves how we can benefit from this while still keeping our fun functional roots. We will explore the cases where the Dataset APIs empower us to do cool things we couldn’t before, what the different approaches to serialization mean, and how to figure out when the shiny new API is actually just trying to steal your lunch money (aka CPU cycles).

Keeping the fun in functional w/ Apache Spark @ Scala Days NYC

Holden Karau

Contributing to Apache Spark 3

Holden Karau

My talk on Piter Py 2016

Alex Chistyakov

More Serious Business Kitty Description: While some deep learning systems have promised to not require any kind of data preparation or cleaning, in practice many folks find that effectively training their models requires some amount of data preparation and often we spend more time on our data preparation than anything else. This talk will examine tools for data preparation that can be used at scale on "big-data" and then how to use their results on-line at serving time (where we hopefully no longer require a cluster to predict every new user). Less Serious Business Kitty Description: Deep Learning, in addition to being a world class tool for detecting the presence of cats, requires large amounts of data for training. As much vendors may say "no data prep required", they are all lying*. This talk will look tools to build a deep learning pipeline with feature prep on top of existing big data technologies without rewriting your code for serving. Traditionally feature prep done in a big data system, like Spark, Flink, or Beam, would have to be rewritting for the on-line serving component. This is about as much fun as when we have to rewrite our sample Python code into Java, as for some reason that's what a lot companies associate with "production." Come for the deep learning buzz-words, stay for the how to perform on-line serving without writing Java code. *All vendors are optimists when it comes to their own products, including the vendors who pay Holden and Gris but they pay us so its ok.

Simplifying training deep and serving learning models with big data in python...

Holden Karau

What does OOP stand for?

Colin Riley

TensorFlow is all kinds of fancy, from helping startups raising their series A in Silicon Valley to detecting if something is a cat. However, when things start to get “real,” you may find yourself no longer just dealing with mnist.csv but instead needing do large-scale data prep as well as training. Holden Karau details how to use TensorFlow in conjunction with Apache Spark, Flink, and Beam to create a full machine learning pipeline—including the annoying feature engineering and data prep components that we like to pretend don’t exist. Holden also explains why these feature prep stages need to be integrated into the serving layer. She concludes by examining changing industry trends, like Apache Arrow, and how they impact cross-language development for things like deep learning. Even if you’re not trying to raise a round of funding in Silicon Valley, this talk will give you tools to do interesting machine learning problems at scale.

Powering Tensorflow with big data using Apache Beam, Flink, and Spark - OSCON...

Holden Karau

With the community working on preparing the next versions of Apache Spark you may be asking yourself ‘how do I get involved in contributing to this?’ With such a large volume of contributions, it can be hard to know how to begin contributing yourself. Holden Karau offers a developer-focused head start, walking you through how to find good issues, formatting code, finding reviewers, and what to expect in the code review process. In addition to looking at how to contribute code we explore some of the other ways you can contribute to to Apache Spark from helping test release candidates, to doing the all important code reviews, bug triage, and many more (like answering questions).

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Databricks

Making the big data ecosystem work together with python apache arrow, spark,...

Holden Karau

Making the big data ecosystem work together with Python & Apache Arrow, Apach...

Holden Karau

Jonathan Coveney: Why Pig?

mortardata

Every-time there is a new piece of big data technology we often see many different specific implementations of the concepts, which often eventually consolidate down to a few viable options, and then frequently end up getting rolled into part of another larger project. This talk will examine this trend in big data ecosystem, look at the exceptions to the "rule", and look at how better interchange formats like Apache Arrow have the potential to change this going forward. In addition to general vague happy feelings (or sad depending on your ideas about how software should be made), this talk will look at some specific examples with deep learning, so if anyone is looking for a little bit of pixie dust to sprinkle on a failing business plan to take to silicon valley to raise a series A, you'll get something out this as well. Video - https://www.youtube.com/watch?v=P_YKrLFZQJo

Are general purpose big data systems eating the world?

Holden Karau

Apache Toree

Asim Jalis

Do Languages Matter?

Bruce Eckel

Powering tensor flow with big data using apache beam, flink, and spark cern...

Holden Karau

Everything You Were Taught About Java Is Wrong

Tim Boudreau

Comefrom

Siddharth Shukla

Similar to Fight Complexity with Functional Programming (20)

Sharing (or stealing) the jewels of python with big data & the jvm (1)

Powering tensorflow with big data (apache spark, flink, and beam) dataworks...

Prototype4Production Presented at FOSSASIA2015 at Singapore

05092019 The Battle of the IDEs by Ko Turk at the AlmereJUG / Conspect

Keeping the fun in functional w/ Apache Spark @ Scala Days NYC

Contributing to Apache Spark 3

My talk on Piter Py 2016

Simplifying training deep and serving learning models with big data in python...

What does OOP stand for?

Powering Tensorflow with big data using Apache Beam, Flink, and Spark - OSCON...

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Making the big data ecosystem work together with python apache arrow, spark,...

Making the big data ecosystem work together with Python & Apache Arrow, Apach...

Jonathan Coveney: Why Pig?

Are general purpose big data systems eating the world?

Apache Toree

Do Languages Matter?

Powering tensor flow with big data using apache beam, flink, and spark cern...

Everything You Were Taught About Java Is Wrong

Comefrom

More from All Things Open

Presented at the ATO RTP Meetup Presented by Jeremy Proffit, Director of DevSecOps & SRE for Customer Care and Communications, Ally Title: Building Reliability - The Realities of Observability Abstract: Join me as we discuss true observability, learn what works and what doesn't. We'll not only discuss dashboards, monitoring and alerting, but how these can be built by automation or included in your IAC modules. We'll talk about how to properly alert staff based on priority to keep your staff and yourself sane. And even discuss architecture and how it impacts reliably and why serverless isn't always the best at being reliable.

Building Reliability - The Realities of Observability

Fight Complexity with Functional Programming

Recommended

Recommended

More Related Content

Similar to Fight Complexity with Functional Programming

Similar to Fight Complexity with Functional Programming (20)

More from All Things Open

More from All Things Open (20)

Recently uploaded

Recently uploaded (20)

Fight Complexity with Functional Programming