AWS essentials for use in scientific computing

1. AWS for scientific computing
jss 2011-05-12

2. TOC
• Overview: Cloud Eco System
• Amazon Web Services for scientific computing
• EC2 Deployment
• S3, EBS and local Storage
• Queueing / Messaging

3. Cloud Layers
Software as a Service - SaaS
running applications
Platform as a Service - PaaS
software environments and services
Infrastructure as a Service - IaaS
virtual machines and network

4. Elastic Compute Cloud (EC2)
• Virtual Data Center
• Instance Types
• Regions
• Availability Zones
• Security Groups (Firewalling)
Can go down just like anyone else
• Amazon Machine Images (AMI)

5. EC2 Setup
• Management:
• AWS web console
• CLI: ec2-tools
• Python: boto (check dev/src/tba)
• Use the setup script
• Get SSH fingerprint from instance’s boot console
• Use some form of parallel SSH for additional setup (Twisted conch
works great, fabric for serial deployment)

6. The DNS Problem
• Instances get new IP on boot
• Solutions:
• use Virtual Private Cloud (Amazon VPC)
• auto-update hosts file
• set up DNS server

7. Storage
• Instance
fast, cleared on boot, no charges
• Elastic Block Storage (EBS)
fast, networked, instant snapshot-able, N attachments per instance
• Simple Storage Service (S3)
HTTP access, CDN (CloudFront)

8. Messaging
• Simple Queueing Service (Amazon SQS)
• AMQP / RabbitMQ (c.f. knk’s recent rocketcircus)
• Rolling your own (e.g. XML RPC) is fine, too

9. TLDR
AWS for the occasional number crunch:
• 1/2 day to get basic setup running
(longer for robust setups)
• profit