Managing big data and running supercomputing jobs used to be for only well-funded research organizations and large corporations, but not any longer. AWS has democratized supercomputing and big data for the masses! AWS can provide you with the 64th fastest supercomputer in the world, on-demand and pay as you go. Hear from Ben Butler, Head of AWS Big Data Marketing, to learn how our customers are using big data and high performance computing to change the world. Not only is AWS technology available to everyone, but it is self-service and cheaper than ever before, featuring innovative technology and flexible pricing models – our AWS cloud computing platform has disrupted big data and HPC. Learn from customer successes, as Ben shares real-world case studies describing the specific big data and high performance computing challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the tools needed to get you started quickly.
7. GB TB
PB
95% of the 1.2 zettabytes of
data in the digital universe is
unstructured
70% of of this is user-
generated content
Unstructured data growth
explosive, with estimates of
compound annual growth
(CAGR) at 62% from 2008 –
2012. Source: IDC
ZB
EB
Big Data: Unconstrained data growth
15. Generated data
Available for analysis
Data volume
Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011
IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares
16. Elastic and highly scalable
No upfront capital expense
Only pay for what you use
+
+
Available on-demand
+
=
Remove
constraints
19. Big data and AWS cloud computing
Big data Cloud computing
Variety, volume, and velocity
requiring new tools
Variety of compute, storage,
and networking options
20. Big data and AWS cloud computing
Big data Cloud computing
Potentially massive datasets Massive, virtually unlimited capacity
21. Big data and AWS cloud computing
Big data Cloud computing
Iterative, experimental style of
data manipulation and analysis
Iterative, experimental style of
infrastructure deployment/usage
22. Big data and AWS cloud computing
Big data Cloud computing
Frequently not a steady-state
workload; peaks and valleys
At its most efficient with highly
variable workloads
23. Big data and AWS cloud computing
Big data Cloud computing
Absolute performance not as
critical as “time to results”; shared
resources are a bottleneck
Parallel compute projects allow each
workgroup to have more autonomy,
get faster results
35. Bizo: Digital Ad. Tech Metering with Amazon Kinesis
Continuous Ad
Metrics Extraction
Incremental Ad
Statistics
Computation
Metering Record Archive
Ad Analytics Dashboard
36. Free steak campaign
Facebook page
Mars exploration ops
Consumer social app
Ticket pricing optimizationSAP & SharePoint Securities Trading Data Archiving
Marketing web site Interactive TV apps Financial markets analytics
Consumer social app Big data analytics
Web site & media sharing
Disaster recovery
Media streaming Web and mobile apps
Streaming webcasts
Facebook app Consumer social app
Business line of sight Mobile analytics
IT operations Digital media
Core IT and media
Ground campaign
53. Why AWS for HPC?
Low cost with flexible pricing Efficient clusters
Unlimited infrastructure
Faster time to results
Concurrent Clusters on-demand
Increased collaboration
54. Cluster compute instances
Implement HVM process execution
Intel® Xeon® processors
10 Gigabit Ethernet –C3 has Enhanced Networking, SR-IOV
cc2.8xlarge
32 vCPUs
2.6 GHz Intel Xeon
E5-2670 Sandy Bridge
60.5 GB RAM
4 x 840 GB
Local HDD
c3.8xlarge
32 vCPUs
2.8 GHz Intel Xeon
E5-2680v2 Ivy Bridge
60GB RAM
2 x 320 GB
Local SSD
AWS High Performance Computing
55. c3.8xlarge
32 vCPUs
2.8 GHz Intel Xeon
E5-2680v2 Ivy Bridge
60GB RAM
2 x 320 GB
Local SSD
Top 500 Super Computer using Amazon EC2
64th fastest supercomputer, Nov 2013
26,496 Intel® Xeon® cores
Linpack Performance (Rmax) 484.2 TFlop/s
Theoretical (Rpeak) 593.5 Tflops/s
c3.8xlarge
32 vCPUs
2.8 GHz Intel Xeon
E5-2680v2 Ivy Bridge
60GB RAM
2 x 320 GB
Local SSD
c3.8xlarge
32 vCPUs
2.8 GHz Intel Xeon
E5-2680v2 Ivy Bridge
60GB RAM
2 x 320 GB
Local SSD
c3.8xlarge
32 vCPUs
2.8 GHz Intel Xeon
E5-2680v2 Ivy Bridge
60GB RAM
2 x 320 GB
Local SSD
56. Network placement groups
Cluster instances deployed in a Placement
Group enjoy low latency, full bisection
10 Gbps bandwidth
10Gbps
AWS High Performance Computing
57. GPU compute instances
cg1.4xlarge
Intel® Xeon® X5570
33.5 vCPUs
22.5GB RAM
2x NVIDIA GPU
448 Cores
3GB Mem
g2.2xlarge
Intel® Xeon E5-2670
8vCPUs
15GB RAM
1x NVIDIA GPU
1536 Cores
4GB Mem
G2 instances
1 NVIDIA Kepler GK104 GPU
I/O Performance: Very High (10 Gigabit Ethernet)
CG1 instances
2 x NVIDIA Tesla “Fermi” M2050 GPUs
I/O Performance: Very High (10 Gigabit Ethernet)
AWS High Performance Computing
59. Making Production Cloud HPC easy from 64 cores to
…
Pharma
Johnson &
Johnson
Manufacturing
HGST, a Western
Digital Company
Financial Services
Pacific Life Insurance
Genomics
Life Technologies
Research
The Aerospace
Corporation
… 156,314 cores for better solar panel materials for $33k, not $68M
Amazon EC2
16,788 Spot
Instances
Amazon S3
4TB
Processed
Spot Instances
on all 8 Regions
1.21 PetaFLOPS
Intel SandyBridge
on CC2
62. Founded in 2003 through the combination of the hard drive
businesses of IBM, the inventor of the hard drive, and
Hitachi, Ltd (“Hitachi”)
Acquired by Western Digital in 2012
More than 4,200 active worldwide patents
Headquartered in San Jose, California
Approximately 41,000 employees worldwide
Develops innovative, advanced hard disk drives, enterprise-class
solid state drives, external storage solutions and services
Delivers intelligent storage devices that tightly integrate hardware
and software to maximize solution performance
6
Capacity Enterprise
Performance Enterprise
Cloud & Datacenter
Enterprise SSD
(+3 acquisitions in 2013)
7200 RPM &
CoolSpin
HDDs
Ultrastar®
Ultrastar® &
MegaScale DC™
10K & 15K
HDDs
PCIe
SAS
63. 6
April 2013
Zero to Cloud in less than 12 Month
By 31 Dec 2013:
Cloud eMail – Microsoft Office365
Cloud eMail archiving/eDiscovery
External SingleSignOn (off VPN)
Cloud File/Collaboration – BOX
USe– Salesforce.com
Integrated to save files in BOX
Cloud–High Performance Computing
(HPC) on AWS
Cloud – Big Data Platform on AWS
Cloud - data mart and provisioning service,
with AWS Red Shift
64. Evolution of Data Centers and HPC @ HGST
SJC
Servo
Team
Japan
Servo
Team
US
Head
Team
US
HAMR
Team
MN
PCB
Team
Old
Mail
System
HGST
Datacenters
On Premise
Off Premise
HPC
Clusters
An Agile Enterprise Datacenter Integrating
On-Premise and Cloud Solutions
Servo
Team in
SJC
PCB
Team in
MN
HAMR
Team in
US
Head
Team in
US
Servo
Team in
Jpn
Evaluate New Storage
Technologies and
Solutions “In House”
(HDD, SSD, etc.)HGST On Site
Business,
Production and
Enterprise
Computing
Siloes of
Clusters
On Premise
Internal
Wiki’s,
etc.
Cloud
65. HPC: Molecular Dynamics Simulation
• HGST uses Molecular Dynamics Simulation for
RnD of materials and lubricants needed for HDD’s
• Research to achieve higher memory densities,
faster read/write capabilities, smaller form factors
and lower power consumption
“Model Job Size”
used at HGST
Complexity
[atoms]
Number of
Time Steps Job Type “Frequency”
Small 300,000 100 200 per day, 2 days per week once or twice a month
Medium 300,000 1000 20 Medium jobs during the day, 4 days per month
Large 300,000 30000 3 large jobs per day, 6 days per month
Very Large 300,000 3000000 1 large job per month
66. Time
Before: Shared 512 Core
Super Computer
512 core 512core 512core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
Today: AWS EC2 CC2
(Max Total 512 core)
512corewaiting
256 core 256 core
128 core 128 core
2W
waiting
waiting
All Jobs Run In Parallel on AWS 1.67x Throughput Improvement
Shape Compute To Match Work To Be Done
67. HPC: Micro Magnetic Simulation
• Model new technologies for future
HGST HDD products
• Finite-difference time-domain (FDTD)
numerical analysis solver
– Accurately simulation of large, complex
models of many variable parameters and
materials
– Scale across large clusters
• AWS C3 Instances provide significant
improvement for both scalability and
simulation throughput
AWS C3 Instances Provided 1.5x or Better Simulation Performance
68. “Cloud HPC”: What’s Next…..
Deploy Graphics User Interfaces
HPC Applications
• Pre and Post Processing in Cloud vs.
data migration back to local systems
AWS C3 Performance Validated Across
Many Applications
• Improve overall performance and reduce
monthly AWS compute bill
69. • “Reduce Data Search Parties”
– Stop playing “Where’s Waldo with your Data”
• “ I know I have that data….. somewhere?”
– Data Aggregation to a Common Platform with common access tools
• Improve Yields by Accessing More Data in a More Timely Manner
– By having end to end visibility to:
• Every test, every diagnostic and all info from all components of a product (internal and external)
– Speed up yield improvement ramp up on new products
– Improve steady state yield on existing products.
“Big Data” in Manufacturing
70. • Metrics:
– Collecting >2M manufacturing/testing binary files daily
– Collecting from ~500 tables across 6 databases tens of millions of records daily
– Over 140 users to date in early piloting
– Over 150 attendees participated in BDP training
• Highlights: although early in the overall journey, HGST’s BDP is already demonstrating early benefits:
HGST: BDP Key Metrics and Highlights
Development Engineer: demonstrated the joining of data sets for detailed
logistics tracking—analyses that is very difficult to conduct with current
systems
Ops Engineer: a recent production issue required detailed historical data. Current systems
did not have the required retention for this data. However, the team was able to pull the data
from the BDP in minutes, as opposed to 3+ weeks to pull the data from tape archive
Development Engineer: obtained technical data from the BDP in
hours as opposed to 3+ weeks to pull from tape archive
DATA SEARCH
PARTIES
YIELD
71. 3. Tailor Data for Consumers
• With the base Big Data platform established, the focus shifts to enabling specific business use cases.
The typical pattern involves:
HGST’s BDP Journey
1. Collect Data
Core Data
Processing
4. Develop Consumers
Derived DB
Enriched
Hive
tables
Hadoop
Analytics Libraries
Dimension
Reduction
Hive
Batch Analytics
Python
R
...
Sampling
Custom
Websites
Specific reports /
visualizations
Specific
analytics
Coredata
Hive / API
Core Data
Processing
The next phase will help to build the
specific website/reports/visualizations
that are tailored to the specific
business use case
The core effort to date has focused on building
the platform, ingesting core data sets, providing
base visualization/data mining tools, and
beginning to prepare the data for specific use
cases
2. Update Core API
Early
Successes
From Here
72. Commercial HPC Applications: Cloud Ready?
• HPC environments desire Cloud Computing with in-
house machines
– “Hybrid” Data Centers
• On-premise workstations + clusters
(some legacy, some new) with
burst/over-flow/connection to Cloud
– EULAs
• Should Comprehend Cloud
• Should allow License server placement
in cloud and accessible on-premise
• Make it easy to add cloud computing to
current licenses
• Consumption Based Pricing
– No consistency across vendors
– Not aligned with time based consumption pricing
73. “We’ve Only Just Begun….”
• Current Results in less than 12 months
• Re-aligning Business Group Leadership, Development Teams,
Research and Development Teams on New Capabilities Model
• Demands and Uses Expected To Grow And Accelerate Market
Success
73
2013 “Heavy Lifting” Provides Foundation
for 2014 Acceleration
77. AWS Online Software Store
http://aws.amazon.com/marketplace
Big Data Case Studies
Learn from other AWS customers
https://aws.amazon.com/solutions/case-
studies/big-data
78. AWS Online Software Store
https://aws.amazon.com/marketplace
AWS Marketplace
79. AWS Online Software Store
http://aws.amazon.com/marketplace
AWS Public Data Sets
Free access to big data sets
https://aws.amazon.com/publicdatasets