แชร์ประสบกาณ์สิ่งที่ทีม Dev ของ Wongnai เคยผ่านมา ตั้งแต่เริ่มต้นเมื่อปี 2010 จนถึงปัจจุบัน และสิ่งที่จะทำในอนาคต

1. Wongnai
Engineering Story
Pattrawoot Suesatayasilp
@pattrawoots
CTO & Co-founder @ Wongnai

2. Wongnai
• A startup company
• Our mission is to help Thai people discover and
connect to great local businesses.
• We believe in trials and errors.
• Time is our most valuable resource.

3. What It Means To
Be Wongnai

4. Wongnai
User-Generated
Content Website
CMS, SEO
Search
Location-Based
Service
Social Network
Feed, Follow, Like,
Comment
Commerce
Deals, Payment

5. Web & App

6. Web & App

7. Statistics
300k sessions/day
53M page views/month
20k requests/min (3x during holidays)

8. Statistics
80%
20%
Desktop
Mobile (App/Web)
34%
66%
Mobile App
Mobile Web

9. • During meal — Lunch, Dinner
• Push Notifications
• Social Post / Dramas
High Traffic
Dinner
Lunch
Push

10. • Weekends especially the long ones
• Special Occasion (Valentine’s)
• Holidays
High Traffic
Chinese
New
Year Valentine’s
Long
WeekendSat - Sun

11. Our 1st dev server
Apple iMac Core 2 Duo 2.0GHz + DynDns

12. Our 1st prod server
Dell PowerEdge R210
฿30,xxx

13. Move To The Cloud

14. 2010-2013 User Growth
0
250000
500000
750000
1000000
Oct '10 Mar '11 Aug '11 Jan '12 Jun '12 Nov '12 Apr '13 Sep '13
Sep 2012
200k subscribers

15. Email Blast
• MailChimp until it became too
expensive. (almost 200k
subscribers)
• Set Up our own mail server
• OpenEMM
• No additional cost but many
problems

16. Email Blast
• Slow send rate — Takes too long to send all
• Consume too much bandwidth
• Poor open rate — Some went to Junk

17. Amazon SES
• Send Rate (emails/second)
• Up to 90 emails/second
• Whitelist IP Addresses
• Migrating to SES wasn’t quite smooth.
Amazon SES

18. SEO & Scalability
• Large number of web pages (> 100k pages)
expecting Google bot to crawl
• Poor latency and not quite reliable (e.g.
power outage)
• 2013’s goal is to reach 1M users. Be ready to
scale.
• Cloud computing is the way to go.

19. Migrate to AWS
• Million of photos to
move to the cloud
• Near-zero downtime
• Installed mobile apps
must continue
seamlessly.

20. Migrate to AWS
Amazon Web ServicesHost in TH
www.wongnai.com
photo.wongnai.com
Static Content
Servers
Web Server
Database

21. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Upload photos gradually
Database
Update
‘synced’
flag
Static Content
Servers

22. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Database
Static Content
Servers
w
w
w.w
ongnai.com
1

23. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Database
Static Content
Servers
photo.w
ongnai.com
Identify photo’s
location
2
3

24. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Database
Static Content
Servers
Redirect to
the
old
host
4

25. Migrate to AWS
Amazon Web ServicesHost in TH
pull
not-synced-yet
images
Web Server
Database
Static Content
Servers
5

26. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Database
Static Content
Servers
w
w
w.w
ongnai.com
1

27. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Database
Static Content
Servers
photo.w
ongnai.com
2
3
Identify photo’s
location

28. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Database
Static Content
Servers
Redirect to
S3
4

29. Migrate to AWS
Amazon Web ServicesHost in TH
Web Server
Database
pull already-synced
images
Static Content
Servers
5

30. Database & ORM

31. When a list of reviews
are displayed
a review
• review’s details
• user (reviewer)
• location
• photos
• likes
• preview comments
• and many more …

32. Select N+1 Problem
Reviewer
Review
Details
Review
Photo
Photo
Photo
Photo
PhotoPreview
Comments
Location
Many tables needs
to be joined to get
enough info for
displaying a
review.

33. Large Table Joining
Review
500,000 rows
User
2,000,000 rows
Location
200,000 rows
Photo
3,000,000 rows

34. Large Table Joining
• As a result of experiments, selecting rows
separately without table joining performs much
better. (No bottleneck at the database) — Feel
the NoSQL?
Single SQL
command
Multiple SQL
command
VS

35. Schema Changes
• Mid 2015 — We decided to combine Food and
Beauty into a single app
• Tables need to be merged. Columns need to be
added. Millions of rows are affected.

36. What we expected
large table

37. What we expected
large table

38. What actually happens
create new tablelarge table

39. What actually happens
create new tablelarge table
copy
• A very large number of I/O operations need
to perform to update the schema

40. I/O Speed
• Storage for Amazon RDS (Amazon EBS)
• IOPS — The number of I/O operations completed per second)
• Base Perf: 300 IOPS (for 100GB storage)
• Burst 3,000 IOPS Maximum — Non-provisioned IOPS Storage
• Burst duration is limited.
• Error 404 — almost 24hrs
• Liquibase and many other schema migration tools
• Only one column can be added at a time!

41. Why still SQL?
• ACID Transaction
• NoSQL is not good at searching. (Compared to
RDBMS)
• NoSQL was not popular yet, and also still lacked
many key features that SQL has. (MongoDB was
just one-year old. Cassandra is still in beta.)

42. Search

43. Types of Search
Any good
restaurants
near me?
Show me all
branches of
Sushi Hiro
Show me all
restaurants in
Central World
Search for
“Bonchon”

44. Indexing
• Apache Solr
• Fast read but slow write
• Our indexes get updated very often, and they
affects ranking — e.g. new review/photo/check-
in, create new place, edit place’s info, etc.
• Tried updating indexes real-time and Solr’s soft-
commit but the performance was not satisfied.
• So, we add another layer to do batch processing.

45. Indexing
• Why not Amazon Elasticsearch Service?
• Too difficult for testing
• We don’t want to stick with AWS.
• Hybrid cloud computing?

46. Cache
• EHCache
• Most of the content are location-based.
• What should be cached? article, guide
• Content keeps updating
• e.g. Your review may get liked right after posted.
• Most of our caches have very short life. (< 1 min) and
never get invalidated. Shared cache is not needed.

47. Scheduled Task
• Quartz (by Terracotta) — Job Scheduler
• Score Updater for calculating ranking
• Business, User, Review, Photo, Activity, etc.
• Leaderboard
• Statistical Data Updater
• Number of Photos/Reviews/Check-ins/…
• etc.

48. Web Stack
เร็วส์Java
@roofimon
“ ”

49. Web Stack
Java Spring SolrMySQL Hibernate Cassandra
RequireJSjQuery Backbone Less Bootstrap

50. Infra/DevOps
Tomcat
Git
Jenkins
DockerMaven
ZookeeperHTTP Server

51. Architecture
Solr
Cloud
HTTP/HTTPS
Load Balancer
Amazon
RDS
ZooKeeper
Cluster
Cassandra
Cluster
Static Content
Servers
Monolithic
Application
Servers
Amazon
SES

52. • 6 year ago, we were new to web development.
• Default Templating System — JSP
• No pre-processor was there back then, so we
wrote our own pre-processors in Maven Lifecycle
(Phase: Prepare package).
• Minify JS, CSS files — 1 file per page.
• YUI Compressor
Web Stack (Frontend)

53. • Asynchronous Module Definition
• RequireJS
• Why not Angular?
• SEO is one of the most important thing for a
content website like Wongnai.com
• JavaScript SEO (prerender.io / …)
• More code to maintain
• Code refactoring takes time. Not worth for
now
Web Stack (Frontend)

54. • Sonar (now SonarQube)
• No longer use it as we found it doesn’t help
us that much. Our dev team is relative small.
Manual code review is far better.
• Jenkins
• No distributed build yet.
• Only used on dev environment
DevOps

55. What’s Next
• Personalization
• Machine Learning / Scala / Spark
• NoSQL for data that doesn’t require searchability
• e.g. Feed, Notifications
• Advanced Message Queuing Protocol (AMQP)
• RabbitMQ
• Microservices
• Image Server

56. Process & Tools
• Asana, Slack, Google Docs
• Agile
• Manual First
• TDD
• Daily Scrum
• 2-week Sprint (1-month Sprint before)
• Retrospect

57. Dev Team
6Full-Stack
Web
Developers
3 iOS
Developers
+ 1 Part-timer
2 Android
Developers
1 QA Engineer
1 UI/UX
Designer

58. Dev Team
We are
Hiring!
6Full-Stack
Web
Developers
3 iOS
Developers
+ 1 Part-timer
2 Android
Developers
1 QA Engineer
1 UI/UX
Designer

59. Q&A
Q&A
Join Us!
wongnai.com/careers