How DYNO apply Big Data and Machine Learning to resolve real world problem?

1. Big Data at DYNO
Tu Pham - IO extended 2017

2. CTO @ DYNO
Data As A Service
Technologies: Java, Python, all kind of databases and
Cloud platform from Google, Aws, Azure.
Interests: Cloud computing, machine learning,
system architecture, technology evolution,
distributed systems.
Husband, Father, GDE, Open source contributor
Tu Pham
foto: Lars Kruse, Aarhus Universitet
2

3. Current system
> 10 000 users
- 4 countries: US, UK, VN,
Singapore
- 700M user profiles
- 800M new raw data daily (90
GB)
- Hundred of jobs daily
Images by ConnieZhou

4. Let’s compare: 10 TB
- 45,813,058.125 book (200
pages of 240,000 characters)
- 2,621,440 MP3 files (with
4MB average file size)
- 3,495,250 MP3 files (with
3MB average file size)
Images by ConnieZhou

5. From 2014, we bring success to
t h o u s a n d o n l i n e m a r k e t i n g
campaigns based on our big data
system.

6. DYNO
DYNO pocesses ability to build,
organize, and operate a Big Data
system with an efficiency and speed
then apply Machine Learning
algorithms to make magic happen.
These are our advantages: unique technology,
talents, company culture, growth hack,
extreme product focus.

7. Yes, We Can Power that
User Identify Social User Modeling Social CRM
Advertising Network Big Data System Data Mining

8. Our Clients

9. Organize the world’s
information and make it
universally accessible and
useful.
Our mission is aligned with Google’s mission
With Sundar Pichai - CEO of Google
2

10. Source: Boston Consulting Group:
The Mobile Revolution: How Mobile Technologies Drive a Trillion-DollarImpact
IDC,2015
By 2020, there will be 8 Billion connected smart phones
2X more than today.
And 32 Billion connected “IOT”devices
6X more than today.

12. Google’s Data Services for everyone

13. A common configuration: draw conclusions
CloudDatalab
Events, metrics,
etc.
Stream
Visualization and BI
Raw logs, files,
assets, Google
Analytics data etc. Co-workers
Batch
Batch
B C Applications and
A Reports
Confidential +Proprietary
A serverless big data stack that
scales automatically

14. 10+ Years of Tackling Big Data Problems
Google CloudPlatform 13
Google
Papers
20082002 2004 2006 2010 2012 2014 2015
GFS
Map
Reduce
Flume
Java
Millwheel
Open
Source
2005
Google
Cloud
Products BigQuery Pub/Sub Dataflow Bigtable
BigTable Dremel PubSub
Apache
Beam
Tensorflow

17. Cloud Dataflow
Cloud Platform
Dataflow is a unified programming model and a managed service for
developing and executing a wide range of data processing patterns
including ETL, batch computation, and continuous computation.

18. Cloud Dataprep
Cloud Platform
Dataprep is an intelligent data service for visually exploring, cleaning,
and preparing structured and unstructured data for analysis. Cloud
Dataprep is serverless and works at any scale.

19. Cloud Dataproc
Cloud Platform
Use Google Cloud Dataproc, an Apache Hadoop, Apache Spark, Apache
Pig, and Apache Hive service, to easily process big datasets at low cost.
Control your costs by quickly creating managed clusters of any size and
turning them off when you're done.

27. Building what’s next 33
Scales automatically
No setup or administration
Stream up to 100,000 rows/sec
Easily integrates with third-partysoftware
Google BigQuery
makes complex data analysis simple

29. Confidential +
Proprietary
Google BigQuery Performance Example ?
Running an inefficient regular expression over 100 billion rowsin
less than 60 seconds
Source: https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-bigquery-
query

30. Google BigQuery
The Power of Google Dremel for everyone
Storage Compute
Fast Ingest
Query
Terabit Network

31. Flow step 1: Stream hundred of million event to Cloud Storage

32. Flow step 2: Aggregate 2B user engagement action with Data Proc

33. > val df = spark.read.parquet("/log/2017/07/user_engagement/1.snappy.parquet")
df: org.apache.spark.sql.DataFrame = [id: string, categoryId: string ... 5 more fields]
Flow step 2: Aggregate 2B user engagement action with Data Proc

34. Flow step 2: Aggregate 2B user engagement action with Data Proc
> df.show()
+----------------------------+---------------------------+------------------------------+-----------------------------+---------+---------+------------------+
| id | categoryId | topicId | userId | action | value | created |
+----------------------------+---------------------------+------------------------------+-----------------------------+---------+---------+------------------+
|"100011125479181_..|"253397751448382"|"253397751448382_...| "100011125479181 "|"view"| "" |1490621079|
|"100004354358107_..|"253397751448382"|"253397751448382_...| "100004354358107"|"view"| "" |1490491531|
|"100014752680147_..|"253397751448382"|"253397751448382_...| "100014752680147"|"view"| "" |1490457109|
|"5c0cba20-654b-11...|"253397751448382"|"253397751448382_...|"1265404853580995...| "love"| "" |1490457109|
|"5c0cba21-654b-11...|"253397751448382"|"253397751448382_...|"1265404853580995...| "love"| "" |1490457109|
|"5c3c7cb0-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb1-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb2-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb3-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb4-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb5-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb6-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb7-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb8-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cb9-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cba-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cbb-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cbc-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cbd-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|
|"5c3c7cbe-654b-11...|"253397751448382"|"253397751448382_...|"1264382007016613...| "love"| "" |1490457109|

35. Flow step 2: Aggregate 2B user engagement action with Data Proc
> val df_group_count = df.groupBy("userId","categoryId", "action").count().show()
+--------------------------+---------------------------+----------+----------+
| userId | categoryId | action | count |
+--------------------------+---------------------------+----------+----------+
|"100011896037126"|"253397751448382"|"love" | 2 |
|"100010391178709"|"253397751448382"|"love" | 1 |
|"100011186707422"|"253397751448382"|"love" | 1 |
|"100012202096674"|"253397751448382"|"love" | 1 |
|"100006411671757"|"253397751448382"|"love" | 1 |
|"100013620943281"|"253397751448382"|"love" | 1 |
|"100009214460050"|"253397751448382"|"love" | 1 |
|"100005333179465"|"253397751448382"|"love" | 1 |
|"100005035988496"|"253397751448382"|"love" | 1 |
|"100013770611991"|"253397751448382"|"love" | 2 |
|"100005162145247"|"253397751448382"|"love" | 1 |
|"100011338607899"|"253397751448382"|"love" | 1 |
|"100014561310840"|"253397751448382"|"love" | 1 |
|"100013095813707"|"253397751448382"|"love" | 1 |
|"100005734717764"|"253397751448382"|"love" | 1 |
|"100007839645885"|"253397751448382"|"love" | 1 |
|"100006016514971"|"253397751448382"|"love" | 1 |
|"100004108243097"|"253397751448382"|"love" | 2 |
|"100008185535342"|"253397751448382"|"love" | 1 |

36. Flow step 2: Aggregate 2B user engagement action with Data Proc
> df_group_count.coalesce(1).write.parquet("/log/2017/07/user_engagement/stat/
avg_engagement_per_user_by_category_id")
BOOM! We have the result!

37. Flow step 3: Data enrich flow

38. Flow step 4: Data visualization – Number of unique user per topic

39. Flow step 4: Data visualization – AVG User engagement per topic

40. Challenges at DYNO - Advertising Network
- Thing we have to do
+ Deliver right ad at right time for right people
+ Mining user information to help brands understand
their audience
+ Build flexible ad targeting infrastructure
+ Analyze user behavior to improve ad relevancy at real
time

41. Challenges at DYNO - User Profiling
- The truth:
+ 65 social network all around the world
+ 2B monthly active user from Facebook (300 PB data
warehouse)
+ Unlimited data still offline
- The problem:
+ How you know N account from Facebook, Google,
Twitter, Linkedin, StackOverFlow, Github, … belong to
one person or not ?

42. Challenges at DYNO - Image Processing
- The problem
+ Detect board sets of objects (House, Car, Motorbike)
+ Find topical entities (Logo, Celebrity, New Event)
+ Face detection

43. We are hiring - Data Mining
Responsibilities
- Designing, mining, testing machine learning algorithms for
delivering valued information from DYNO data warehouse.
Requirements
- BS/MS degree in Computer Science, Engineering or a
related subject
- Knowledge of Relational Databases, SQL or NOSQL
- Familiar with basic machine learning algorithms
- (Plus) Familiar with big data system (Volume, variety and
velocity)

44. We are hiring - Back End
Responsibilities
- Designing and developing high-volume, low-latency applications
for mission-critical systems and delivering high-availability and
performance
- Contributing in all phases of the development lifecycle
- Writing well designed, testable, efficient code
Requirements
- BS/MS degree in Computer Science, Engineering or a related
subject
- Language: Java, Python
- Knowledge of Relational Databases, SQL or NOSQL
- (Plus) Familiar with big data system (Volume, variety and velocity)

45. Big Data Challenges At DYNO - Image Extraction
- Thing we have to do
+ Deliver right ad at right time for right people
+ Mining user information to help brands understand
their audience
+ Build flexible ad targeting infrastructure
+ Analyze user behavior to improve ad relevancy at real
time

46. JOIN THE FLIGHT
IO Extended 2017
Facebook: fb/pham.phuong.tu
Twitter: @phamptu
Email: tu@dyno.vn