Presentation on Cloud, Cloud federation and the EGI federated cloud. 5/6/15

1. 1
Mobile Cloud
David Wallom
University of Oxford

2. The NIST Cloud Definition Framework
Community
Cloud
Private
Cloud
Public Cloud
Hybrid Clouds
Deployment
Models
Service
Models
Essential
Characteristics
Common
Characteristics
Software as a
Service (SaaS)
Platform as a
Service (PaaS)
Infrastructure as a
Service (IaaS)
Resource Pooling
High Perf Network Access Rapid Elasticity
Measured Service
On Demand Self-Service
Low Cost Software
Virtualization Service Orientation
Advanced Security
Homogeneity
Massive Scale Resilient Computing
Geographic Distribution
Based upon original chart created by Alex Dowbor - http://ornot.wordpress.com

3. What I think mobile cloud is?
• User
– Utilise any cloud [group, institutional, Community, Public] for any useful applications
– Utilise any suitable tool to make usage easier [Slipstream, Starcluster, a.n.other]
• Resource Provider
– Free choice which cloud system to deploy, migrate when necessary
– Support utilisation of public cloud
• Technology Provider
– Open marketplace where product quality decide uptake

4. • Globally distributed;
• different resources/costs;
• different applications;
• non standardised: different AAA and UI.
Private/Public Multiple Clouds
Users
CLIMB cloudAmazon cloud
Embassy cloud
EGI cloud partner
Azure cloud

5. Mediated Private/Public Multiple Clouds
Management
Interface
CLIMB cloud
Amazon cloud
Embassy cloud
EGI cloud partner
Users
• Automation;
• load balancing;
• costs reduction;
• usability.

6. • Federation of Local and Global resources
• Elasticity managed by local cloud not user
• different resources/cost;
• different applications;
• non standardised: different AAA but single UI through private provider
Hybrid Multiple Clouds
Users
CLIMB cloud
Amazon cloud
Embassy cloud
EGI cloud partner

7. Rationale for mobility
Growth of Providers
• High Throughput/Performance Computing
–Academic resource providers
• Federated Cloud
–Diversity of resource providers
7
Tens of 1000’s Millions
Few related use cases
Few application models
Many diverse use cases
& application models
Growth of Research Communities

8. 86/5/2015
Value proposition
The EGI Federated Cloud, a federation of institutional private Clouds,
offering Cloud Services to researchers in Europe and worldwide
A single cloud system able to
• Scale to user needs
• Integrate multiple different providers to give resilience
• Prevent vendor lock-in
• Enable resource provision targeted towards the research
community
Standards based federation of IaaS cloud:
• Exposes a set of independent cloud services accessible to users
utilising a common standards profile
• Allows deployment of services across multiple providers and
capacity bursting

9. 96/5/2015
EGI Cloud Infrastructure
EGI Core Platform
Federated
AAI
Service
Registry
Monitoring Accounting
EGI Cloud Infrastructure Platform
VM
Management
Information
Discovery
Storage
Management
Help and
Support
Security Co-
ordination
Training and
Outreach
EGICollaborationTools
EGIApplication
DB
Image
Repository
EGICloudServiceMarketplace
Sustainable
Business
Models
User Communities
Monitoring and control of utilisation
Technical Consultancy and Support
Uniform interfaces to Cloud
Compute and Storage
Cloud Management Stacks
(OpenStack, OpenNebula, Synnefo, …)
Cloud Management Stacks
(OpenStack, OpenNebula, Synnefo, …)Resource Provider InfrastructureResource Provider Infrastructure
SecureendorsedApplicationand
ServiceDeployment

10. 106/5/2015
On demand compute to run any kind of workloads on
virtual machines
• OCCI as standard interface
– Contextualization extension
– Profiling to ensure interoperability
• Clients
– Commandline
– ruby and java clients
– OCCI connectors in brokers
• Servers
– rOCCI-server
• Open Nebula
• StratusLab*
• Apache Cloudstack
• AWS
– OCCI-OS
– snf-occi
VM Management
VM VM
Site A
VM VM
Site B

11. 116/5/2015
Automatic and secure distribution of endorsed VM
images for Virtual Organisations
• Web based front-end in
AppDB
• OVF for packaging images
• HEPiX lists for distribution
– Endorsed by VO managers
– Signed metadata
– RP subscribe and download
• EGI endorses basic OS
images
VM Image Management
Site B
VM
Image
Site A
VM
Image Site C
VM
Image
EGI AppDB
VM
Image

12. 126/5/2015
Block Storage
• Persistent Block Level
Storage to attach to VMs
• Manage through VM
OCCI
Object Storage
• Data storage accessible
fro anywhere at any
time
• Sharing/Serve data
• CDMI as standard API
Storage Management
VM

13. 136/5/2015
Integration with EGI core services:
Accounting
• Collects VM usage
metrics in a central
accounting repository

14. 146/5/2015
Integration with EGI core services:
Monitoring
• Ensure service
availability and
reliability

15. 156/5/2015
FedCloud Infrastructure
• One year of production
• Resources
– 21 providers from 14 NGIs
• 55% Openstack, 42% Open Nebula, 3%
Syneffo
– 17 interested in joining from 7 new
NGIs
• Usage
~700K VMs
~9M CPU hours wall time
EGI Conference 2015

16. 166/5/2015
Usage Models
Supporting multiple diverse usage models:
• Service Hosting
– Long-running services (e.g. web, database or application
servers)
• Compute and data intensive workloads
– Batch and interactive (e.g. IPython, R, matlab) with scalable
and customized environments not limited to the traditional
job model
• Datasets repository
– Store and manage large datasets for your applications
• Disposable and testing environments
– Host training events, test new developments and
applications without overhead

17. 176/5/2015
High Level Tools (PaaS & SaaS)
• Extend the IaaS capabilities of the EGI cloud
• ‘Alternatives’ of the OCCI client and API
• More than OCCI
• External contributions ( support many other clouds too)
• SaaS
• Identity Federation
Catania
Science
Gateway
• PaaS for automating
deployments
• Helix Nebula
Slipstream
• Programming framework
for auto-parallelisation
COMPSs
• Abstraction on top of
various HPC/HTC/cloud
VMDIRAC
• Workflow development
and enactmentWS-PGRADE
• VM lifecycle managerVcycle

18. 186/5/2015
Federated Cloud user engagement and
support
Dedicated technical consultancy for each community
(support@egi.eu)
• Step by step guides
• Tutorials
• Examples
Doc
• Main OS available
• Secure endorsed
• Contextualisation
Basic VM
Images
• Resource providers
commitment
• Support to create a
production VO
Migration
into
production
• Initial requirements
• Allocation of technical
experts
• Milestones
F2F/Web
Meetings
• Technical integration
support from EGI/NGI
team
• Periodic meetings
Continuous
tracking and
support
• Resources for application
prototyping
• Enabled in all sites
• Up to 6 months (renew)
Incubator
VO

19. 196/5/2015
Use Case Discipline Classification
Usecases
- 12 @ Launch
- 76 to date
- 11 production

20. 206/5/2015
ESA HN Flagship on the EGI
Federated Cloud
The European Space Agency deployed part of the ESA Helix Nebula
Flagship into the EGI Federated Cloud:
1. Use SlipStream Infrastructure Broker to deploy a custom computing
cluster on multiple Federated Cloud sites (CESNET, GRNET)
2. Connect the computing cluster to the ESA G-POD system and run
SAR processing for volcano and earthquake monitoring
Automatic
Cluster
Deployment
EGI Federated Cloud
Computing Cluster
Cluster
Head
Working
Nodes
OCCI
Globus
Job submission
BLUE BOX

21. 216/5/2015
Peachnote
Peachnote is a music score search engine and analysis platform.
Hundreds of thousands of music scores are
being digitized by libraries all over the world.
In contrast to books, they generally remain
inaccessible for content-based retrieval and
algorithmic analysis.
There is no analogue to Google Books for
music scores, and no large corpora exists that
can empower advanced analysis on music
scores.
Peachnote want to help change that providing
visitors and researchers access to a massive
amount of symbolic music data.
EGI Federated Cloud
OMR
Worker
PDF
Splitter
OMR
Feeder
OMR
Worker
…

22. 226/5/2015
OpenModeller on the Biovel
Portal
Evalua
mod
BENELUX confe
Ghent, 02. April
The Ecological Niche Modeling (ENM) Workflow takes as input a
file containing species occurrence points to create a model with
the openModeller Web Service.
• The EUBrazilOpenBio ENM service is
exposed through an extended
openModeller Web Service interface
• Multi-staging and multi-parametric oM
experiments are implemented through
COMPSs that dynamically creates the
virtual resources to execute the
operations.
• An OCCI connector is used for the VMs
management while data management
supports CDMI endpoints.
ENM Service (OMWS2)
VENUS-C Cloud Middleware
COMPSs Workflow
Orchestrator
OCCI CDMI
EGI Federated Cloud
Service available at
https://portal.biovel.eu/

23. 236/5/2015
OpenRefine on the BioVel Portal
It is used within BioVeL for
taxonomic data to ensure
data quality and integrity.
The service integrates
existing know-how of
taxonomic data with existing
functionality in Google
Refine to develop BioVeL
specific data cleaning tools.
Evaluat
mode
Service available at
https://portal.biovel.eu/
OpenRefine (originally developed as Google Refine) is a
framework for viewing and cleaning large amounts of messy
data.

24. 246/5/2015
BioSTIF - Biodiversity Spatio-temportal
interface on the BioVel Portal
BioSTIF provides an easy GIS interface to vizualize and quality
control biodiversity data in time and space
Evaluat
mode
The interface contains;
• a map widget (displaying
geographical data)
• a timeline widget
(displaying data on a time
scale)
• a table widget (displaying
data in table format).
Service available at
https://portal.biovel.eu/

25. EGI Conference 2015 25

27. Conclusions
• There are a number of different models of mobile
cloud available, each with different pros + cons
• EGI Federated Cloud is one way of providing
mobile cloud [that demonstrably works]
• Most EU countries ahead of UK after years of
being behind
• If you want production services operations takes
time and effort