Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deliver Best-in-Class HPC Cloud Solutions Without Losing Your Mind

While cloud computing offers virtually unlimited capacity, harnessing that capacity in an efficient, cost effective fashion can be cumbersome and difficult at the workload level. At the organizational level, it can quickly become chaos.

You must make choices around cloud deployment, and these choices could have a long-lasting impact on your organization. It is important to understand your options and avoid incomplete, complicated, locked-in scenarios. Data management and placement challenges make having the ability to automate workflows and processes across multiple clouds a requirement.

In this webinar, you will:

• Learn how to leverage cloud services as part of an overall computation approach
• Understand data management in a cloud-based world
• Hear what options you have to orchestrate HPC in the cloud
• Learn how cloud orchestration works to automate and align computing with specific goals and objectives
• See an example of an orchestrated HPC workload using on-premises data

From computational research to financial back testing, and research simulations to IoT processing frameworks, decisions made now will not only impact future manageability, but also your sanity.

  • Login to see the comments

  • Be the first to like this

Deliver Best-in-Class HPC Cloud Solutions Without Losing Your Mind

  1. 1. Deliver Best-in-Class HPC Cloud Solutions Without Losing Your Mind WEBINAR April 13, 2016, 11:00 AM ET
  2. 2. Housekeeping • Audio help • Attachments • Questions • Rating
  3. 3. Today’s Speakers Rick Friedman Vice President, Solution Development Cycle Computing Scott Jeschonek Director of Product Management, Cloud Avere Systems
  4. 4. Agenda • Discuss the current state of HPC • Clouds and their impact on your HPC world • Reasons why you aren’t 100% cloud-based already • The Hybrid Cloud and HPC • Possible implementations • Delivering File Systems using Avere Systems • Orchestration using Cycle Computing
  5. 5. HPC Today (and Yesterday, and Tomorrow)
  6. 6. What Drives Today’s Needs • Data – Who, what, when, how much, where? • Datacenter limitations – Can I defy physics? • User expectations – Can we even do that? • Technology shifts – What is the “best practice”?
  7. 7. Big Compute Workloads: How are they handled? Compute Demand vs. Cluster Size Cluster Size Compute Demand Missed Opportunity Wasted Resources • Internal infrastructure has huge value and some limitations • Access, not capacity, is the barrier to continued growth • Perception limits scale of problem solving • Public cloud = cost-effective, readily available resources to users with problems & deadlines. • Financial services, manufacturing and life sciences are leading the way.
  8. 8. Basic HPC Environment Requirements Resource Manager Jobs Manager / Scheduler Workload NAS Storage Lots of compute resources (“Grid”)
  9. 9. Advantages of Clouds Significantly reduce infrastructure management costs both in money and time Maintain operational flexibility during scale-out jobs…let the provider deal with scale challenges
  10. 10. Why the Cloud for Big Compute? • Scientist / Engineer User perspective – Zero queue times, capacity in minutes – Scale compute to problems size, not vice versa – Try / support new computational approaches and software quickly • SysArchitect perspective – Dynamically adjust workloads to “lowest cost/impact” provider – Focus on computational excellence, not hardware management – Support a wide range of user types efficiently • Organizational perspective – Match spending to actual consumption – Increase responsiveness to business dynamics – Grow user base without hardware limitations
  11. 11. Clouds Have Awesome New Capabilities • Big Data – Analytics Tools – Massively scalable NoSQL – Data warehousing • Machine Learning – Voice/Vision/Speech – Early days
  12. 12. So…why isn’t everything in the cloud? • Current infrastructure investment (capex) • Cloud costs not yet completely in line • Software infrastructure in place – Costs to refactor, dependencies to consider • Data environment in one or more data centers • Orchestration and management of cloud clusters is hard • Network bandwidth / latency concerns • Business Continuity
  13. 13. Other Reasons You’re Not 100% in the Cloud • Corporate budgets • Corporate policies • Corporate politics • Education / awareness • Government regulations • Interest groups • Vendor relationships
  14. 14. Near Future, Hybrid Cloud Tokyo office London office Analysts Analysts NYC office Analysts AnalystsAnalysts Analysts AnalystsAnalysts AnalystsAnalysts Hong Kong office • Adoption of one or more cloud providers • > 1 hedge on price and SLA • Mix of on-prem and cloud resources • Regulatory, proprietary and/or security characteristics will likely keep data in the DC NAS Primary DC Cloud Provider 1 Cloud Provider 2 NAS Secondary DC
  15. 15. Cloud Compute Environment Data HPC in the Cloud Cloud Compute API Scheduler NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs On-Premises Data Center
  16. 16. Cloud Compute Environment HPC in the Cloud, “Grids on Demand” Cloud Compute API Data NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs On-Premises Data Center Scheduler1 Scheduler2 Scheduler3 Scheduler4
  17. 17. Challenges with HPC in the Cloud • How do you get the data close to your compute nodes? • How do you orchestrate on-demand clusters/grids of compute nodes? • How does this all come together??
  18. 18. Cloud Compute Environment Data Access Layer Cloud Compute API Scheduler1 Data NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs On-Premises Data Center Data Access Layer Scheduler2 Scheduler3 Scheduler4 • File System • Caching Layer • Only load necessary blocks of files • Opaque to compute nodes
  19. 19. Advantages of Data Access / Cache Layer • Keep your data on prem! – Data in cloud is only there while the compute nodes work the jobs. – Reduce the security objections, simplify the move to cloud • Increase cloud compute performance – using file system caching, most of the data will be in RAM, close to the nodes – Avoids ingest latencies and slashes transit latency after first read • Scale out – Using solution that facilitates 10s of 1000s of core file system connections
  20. 20. Typical File Access in Hadoop Cluster Caching files will work for certain types of jobs Where typical file is accessed By multiple clients source: http://blog.cloudera.com/blog/2012/09/what-do-real-life-hadoop-workloads-look-like/
  21. 21. Hybrid Cloud using Avere FXT and vFXT Edge Filers Cloud Compute On-Prem Compute Cloud Storage On-Prem Storage NAS Object Bucket 1 Bucket 2 Bucket n Virtual Compute Farm Virtual FXT File Storage for Private Object NAS Optimization Cloud NAS Physical FXT The “Edge” = locating your data Close to your compute Without truly moving it from your NAS environment
  22. 22. Avere Building Blocks “Avere is uniquely positioned to offer scale across tens of thousands of cloud compute cores while leaving the data where it originates, on premises, with it’s global file system and caching capabilities.” - Unnamed CTO Cloud Compute Virtual FXT NAS Object Physical FXT Cloud On-Premises File Acceleration
  23. 23. Cloud Compute Environment Orchestration and Management Layer Cloud Compute API Data On-Premises Data Center Scheduler1 Scheduler2 Scheduler3 Scheduler4 NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs
  24. 24. Optimization • Benchmark instances • Make Workflow UI • Human workflow Provisioning • Workload placement Optimal scale • Cost optimization • Data scheduling Cluster Configuration • Multi-cloud, without changes • Pre-set or User-defined “types” • Abstraction for all cluster data, attributes (roles, OS, etc) Monitoring • Auto-scaling • Usage tracking • Error Handling • Reporting Internal File: Declarative Cluster Definition Packages, Installers Containers, Data Admin Scope Configure Run on Cloud Optimize User Complete Multi-Cloud Workflow Control
  25. 25. User Web UI API CMD Line Job & Data Workflow Automated Job Placement, Cost optimization Auto-scaling, Benchmarking, Compliance, Reporting tools Multi-cloud Without Changes Internal Cluster How Cycle Makes Cloud Productive • Scientist / Engineer productivity: – Simple workflows – Zero queue time – Auto-scaling • SysAdmin productivity: – Instant access to additional resources – Workflows linking internal and multiple clouds – Simple reliable tools to enable apps with special requirements • Organizational productivity: – Secure, consistent cloud access – Usage tracking – Ability to leverage multiple providers
  26. 26. Big Data w/o Disrupting Production • Challenge – Estimate the carbon stored in Saharan biomass – Rapidly establish a baseline for later research using large amounts of high-resolution remote sensing data – Existing internal compute resources fully committed – Limited window to complete processing • Cycle solution – Full workflow including data management between internal data capture and cloud processing – Leverage spot pricing to minimize cost while maximizing computation • Results – Linearly scalable, predictable enabling plan for next steps – Science being done that could not be done otherwise – 1 month start to initial runs 26
  27. 27. Overall Architecture – Data In-House Cloud Compute Scheduler Avere FXT Edge Filer Avere FXT Workload Cloud API NAS Storage Scheduler Cloud Storage
  28. 28. What We Covered… • The Current State of HPC • Clouds and Their Impact on Your HPC World • Reasons Why You aren’t 100% Cloud-based Already • The Hybrid Cloud and HPC • Possible Implementations • Delivering File Systems Using Avere Systems • Orchestration Using Cycle Computing
  29. 29. Thank you! Cycle Computing Contact Info: More about Avere Systems: askavere@averesystems.com www.averesystems.com 1-888.88.AVERE https://twitter.com/averesystems https://www.youtube.com/user/AvereSystems https://www.linkedin.com/company/589037 info@cyclecomputing.com www.cyclecomputing.com 888.292.5320 https://twitter.com/cyclecomputing https://www.youtube.com/user/CycleComputing https://www.linkedin.com/company/692068

×