http://herokujp.doorkeeper.jp/events/3405

1. Treasure Data
Treasure Data and Heroku
Masahiro Nakagawa
Heroku Meetup #8 TreasureData + Waza Report!!
Thu, 04 Apr 2013
Friday, April 5, 13

2. Who are you?
 Masahiro Nakagawa
• @repeatedly / masa@treasure-data.com
 Treasure Data, Inc.
• Senior Software Engineer, since 2012/11
 Open Source projects
• D Programming Language
• MessagePack: D, Python, etc...
• Fluentd: Core, mongo, etc...
• etc...
2
Friday, April 5, 13

3. Introduction to
Treasure Data
Friday, April 5, 13

4. Company Overview
 Silicon Valley-based Company
• All Founders are Japanese
• Hironobu Yoshikawa
• Kazuki Ohta
• Sadayuki Furuhashi
 OSS Enthusiasts
• MessagePack, Fluentd, etc.
4
Friday, April 5, 13

5. Investors
 Bill Tai
 Naren Gupta - Nexus Ventures, Director of Redhat, TIBCO
 Othman Laraki - Former VP Growth at Twitter
 James Lindenbaum, Adam Wiggins, Orion Henry - Heroku
Founders
 Anand Babu Periasamy, Hitesh Chellani - Gluster Founders
 Yukihiro “Matz” Matsumoto - Creator of Ruby
 Dan Scheinman - Director of Arista Networks
 Jerry Yang - Founder of Yahoo!
 + 10 more people
• and....
5
Friday, April 5, 13

6. Treasure Data = Cloud + Big Data
Cloud Big Data-as-a-Service
Database-as-a-service
Enterprise
Lightweight RDBMS Traditional
RDBMS Data Warehouse
DB2
On-Premise
$34B $10B
market market
1Bil entry Data Volume
Or 10TB
© 2012 Forrester Research, Inc. Reproduction Prohibited 6
Friday, April 5, 13

7. Why Cloud? ‘Time’ is Money
Ideal
Customer Expectation
Value
Maintain
Obsolete
RedShift over time
EMR AWS
(or hosted Hadoops) Reality
EC2
Step-by-step manual
integrations (On-Premise)
S3 Upgrade
HW/SW Selection, PoC, Deploy...
Time
Sign-up or PO
7
Friday, April 5, 13

8. Full Stack Support for Big Data Reporting
Our best-in-class architecture Data from almost any source
and operations team ensure the can be securely and reliably
integrity and availability of your uploaded using td-agent in
data. streaming or batch mode.
Our SQL, REST, JDBC, ODBC You can store gigabytes to
and command-line interfaces petabytes of data efficiently and
support all major query tools securely in our cloud-based
and approaches. columnar datastore.
8
Friday, April 5, 13

9. 9
Product
Data Collection Data Warehouse Data Analysis
Web Log
BI Tools
App Log Streaming Upload REST Tableau, QlickView
Open-Source JDBC / ODBC
Log Collector >60billion / month Excel, etc.
Sensor Columnar Storage SQL
2,000+ companies (HiveQL)
(incl. LinkedIn, etc) + or
Hadoop Pig
Bulk Loader MapReduce
RDBMS Dashboard
CSV / TSV Bulk Upload
CRM MySQL, Parallel Upload Result push Custom App,
250bil+ records
Postgres RDBMS, FTP, etc.
2mil+ jobs
ERP Oracle, etc.
Value Proposition:
“Time-to-Answer” 20bil+, 2 weeks, 2 weeks, 3 weeks,
3bil+, 3 weeks 2 weeks,
UK/Austria US Japan
Singapore US
Multi-Tenant: Single Code for Everyone - no code modification, Improving the Platform Faster.
Friday, April 5, 13

10. Customer Use Cases
Friday, April 5, 13

11. 11
Our Customers – Fortune Global 500 leaders and
start-ups including:
Friday, April 5, 13

12. Example in AdTech: MobFox
1. Europe’s largest independent mobile ad exchange.
2. 20 billion imps/month (circa Jan. 2013)
3. Serving ads for 15,000+ mobile apps (circa Jan. 2013)
4. Needed Big Data Analytics infrastructure ASAP.
12
Friday, April 5, 13

13. Two Weeks From Start to Finish!
13
Friday, April 5, 13

14. Viki.com: “Global Hulu”
Friday, April 5, 13

15. Viki.com Before
 Hard to manage Hadoop
 Complicated data collection
Friday, April 5, 13

16. Viki.com After
 No more Hadoop maintenance
 Versatile data collector, td-agent
Friday, April 5, 13

17. Our Usage
Friday, April 5, 13

18. https://console.treasure-data.com/
18
Friday, April 5, 13

19. http://fluentd.org/
19
Friday, April 5, 13

20. Other usage
 Staging environment
 Internal testing application
 Proxy server for our used services
20
Friday, April 5, 13

21. Heroku integration
Friday, April 5, 13

22. http://blog.treasure-data.com/post/44003014921/treasure-
data-is-sponsoring-heroku-waza-2013
22
Friday, April 5, 13

23. Matz
http://www.wired.com/business/2013/03/heroku-waza/ http://instagram.com/p/WTIEwpA_9-/#
23
Friday, April 5, 13

24. Heroku addons
https://addons.heroku.com/provider/resources/technical/how/overview
24
Friday, April 5, 13

25. 25
Friday, April 5, 13

26. https://addons.heroku.com/treasure-data
26
Friday, April 5, 13

27. Using Heroku addon
 Setup “td” command
• Install via td-toolbelt or rubygems
 Setup “td” heroku plugin
• heroku plugins:install https://github.com/treasure-data/
heroku-td.git
 Add ‘td’ gem to your Gemfile
• or STDOUT log collecting
 “heroku td” is now available for Treasure Data
• “heroku td xxx”: xxx is the same as “td” command
https://devcenter.heroku.com/articles/treasure-data
27
Friday, April 5, 13

28. Just STDOUT
 Use STDOUT to collect event logs
• No need libraries
• log forward via Heroku syslog drain
 Format
• @[db_name.table_name] json_in_one_line
• Ruby:
puts '@[service.users] {"name":"D", "via":"Phobos"}'
http://blog.treasure-data.com/post/41886298790/just-
stdout-the-simplest-most-flexible-way-to-collect
28
Friday, April 5, 13

29. 29
Friday, April 5, 13

30. Conclusion
 Treasure Data
• Cloud based Big-data analytics platform
• Provide Machete for Big data reporting
 Heroku and Treasure Data
• Treasure Data addon
• easy to integrate with your Heroku app
• STDOUT log collecting with Heroku syslog drain
30
Friday, April 5, 13

31. Big Data for the Rest of Us
www.treasure-data.com | @TreasureData
Friday, April 5, 13