Deck from Galaxy 2017 conference in Australia on scaling Galaxy bioinformatics tools via GCP or Google Cloud Platform

1. 1
Scaling Galaxy on GCP
LynnLangit
Cloud and Data Architect
Google Developer Cloud Expert, AWS Community Hero, Microsoft Data Platform MVP

2. 2
Agenda
• Scaling Up
• Virtual Machines
• Hello Galaxy
• Adding Tools to Galaxy
• Genomic Data on GCP
• Scaling Out
• Docker Containers
• Google Persistent Disks
• Pipelines
• Google Genomics APIs
• Big Query
Galaxy on Google Cloud Platform

3. 3
Google Cloud in Australia
Data center here in 2017

4. 4
Galaxy on GCP – Scale Up

5. 5Google Cloud Platform 5
Demo 1- Hello Galaxy on Google Cloud

6. 6Google Cloud Platform 6
Demo 2 - Adding Galaxy Tools

7. 77
• Cloud Storage (file) buckets
• Source data
• Compute Engine Virtual Machines
• Virtual Machine Image files
• External VM persistent hard disks with your source data
GCP Virtual Machine Services
Key Concepts:
-- VM configuration as code
-- Fast, cheap scalable VMs

8. 88
• Re-size Virtual Machines
• Attach more persistent disks
• Update base image
• Monitor with Stackdriver
Scale Up Patterns

9. 99

10. 1010
Genomic Data
• Files at GCS
• gs://genomics-public-data
• Query via BigQuery
• https://bigquery.cloud.google.com/queries/genomics-public-
data
• Code via Genomics API
• Implements Global Alliance for Genomics and Health
APIs
• Genome browser - https://gabrowse.appspot.com
• Google Genomics example code on GitHub

11. 1111

12. 12
Galaxy on GCP – Scale Out

13. 1313
GCP Docker Container Services
• Cloud Storage
• Container Engine / Docker
Key Concepts:
-- Container configuration as code
-- Fast, cheap scalable Docker Containers

14. 1414

15. 1515
• Docker Container Cluster
• Kubernetes manager
• Container orchestration
Scale Out Patterns

16. 1616
GCP Serverless Services
• Cloud Functions
• Microservices
Key Concepts:
-- Function configuration as code
-- Fast, cheap scalable Microservices

17. 17
Galaxy on GCP – Advanced Pipelines

18. 18Google Cloud Platform 18
Demo 3 – Using the Google Genomics API &
BigQuery

19. 1919
BigQuery
• ANSI SQL Queries
• Query-as-a-service
Key Concepts:
-- SQL query configuration as code
-- Fast, cheap scalable SQL Queries

20. 20
Private Datasets Public Datasets
Variant Analysis
MSSNG Autism
Cloud Storage
Scientist
High
Throughput
Genome
Sequencers
1000 Genomes
Cloud Storage
Patient Data
Cloud Storage
Illumina Platform
Cloud Storage
Ref Genomes
Cloud Storage
TCGA
Cloud Storage
Analytics
Online Analytics
BigQuery
Batch Analytics
Cloud Dataflow
Lab Notebooks
Cloud Datalab
Data Ingest
Genomics
BAM
FASTQ

21. 21
Ingest
Elastic Cluster
Storage
Analytics
Carrier
Interconnect
High
Throughput
Genome
Sequencer
s
Scientist
Raw Datafiles
Cloud Storage
Processed Data
Cloud Storage
Metadata
Cloud SQL
Lab notebooks
Cloud Datalab
HPC Cluster
Compute
Engine
10 Nodes
Ingest Server
Compute
Engine
Online Analytics
BigQuery
Cloud Load
Balancing
Cloud
Network
Genomics, Secondary
Analysis

22. 2222
• Cloud Storage / Public datasets on
GCP
• Big Query
• Cloud Dataflow
• Genomics API
Advanced GCP Pipelines Core
Products
Key Concepts:
-- Pipeline configuration as code
-- Fast, cheap scalable cloud services

23. 23
Resources

24. 2424
• Cloud Storage (files) -- here
• Compute Engine (VMs) -- here
• Container Engine (Docker) -- here
• Big Query (SQL) -- here
• Cloud Dataflow (pipelines) -- here
• Genomics API-- here
• Genomics Cookbook– here
• Public datasets on GCP-- here
• Google’s Genomic code samples – here
• Lynn’s GitHub code samples -- here
Resources

25. 25
More about Google Cloud Services

26. 26
Compute
Compute
Engine
App
Engine
Container
Engine
Container
Registry
Cloud
Functions
Networking
Cloud Virtual
Network
Cloud Load
Balancing
Cloud
CDN
Cloud
Interconnect
Cloud
DNS
Big Data
BigQuery
Cloud
Dataflow
Cloud
Dataproc
Cloud
Datalab
Cloud
Pub/Sub
Genomics
Identity & Security
Cloud IAM
Cloud Resource
Manager
Cloud Security
Scanner
Cloud Platform
Security
Storage and Databases
Cloud
Storage
Cloud
Bigtable
Cloud
Datastore
Cloud SQL
Persistent
Disk
Machine Learning
Cloud Machine
Learning
Vision API
Speech
API
Natural
Language API
Translation
API
Google Cloud Platform Services Part One
Jobs API

27. 27
Management Tools
Stackdriver Monitoring Logging
Error
Reporting
Trace Debugger
Deployment
Manager
Cloud
Endpoints
Cloud
Console
Developer Tools
Cloud
SDK
Deployment
Manager
Cloud Source
Repositories
Cloud Tools for
Android Studio
Cloud Tools
for IntelliJ
Cloud Tools for
PowerShell
Cloud Tools for
Visual Studio
Google Plug-in
for Eclipse
Cloud Test
Lab
Google Cloud Platform Services Part Two
Cloud
Shell
Cloud Mobile
App
Billing
App
Cloud
APIs

28. 28

29. 29

30. 30
GCE Persistence Options – Disks,
etc.… Created From Notes
Image GCS File or Disk File path <bucket>/<folder>/<file>
Disk must detached from VM
Snapshot Disk or Instance (boot) Can create an Instance FROM a Snapshot
Persistent
Disk
Image –or-
Snapshot –or-
Blank
Blank disk must be formatted
Can create an Instance or Snapshot FROM a Disk
Bucket GCS console for file Access via path gs://<bucketName>/<fileName>
VM Instance
Boot Disk
Image –or-
Snapshot –or-
Disk
Images -> OS, Application or Custom Image
N/A
From Saved Disk
VM Instance
Additional
Disk
Local Scratch –or-
Standard persistent –or-
SSD persistent
Max 8 at 375 GB each.
500 GB  64 TB
Read/Write or Read Only
Attach up to 16 Disks* per VM

31. 31