3. 3
GCP and Patterns
Developer-first
• Fast, flexible and cheap
• Virtual Machines / GCE
• Storage / GCS
Servers ➡ Containers ➡ Functions
• Data Warehouse
• Internet of Things (IoT)
• Bioinformatics
1. Modern Cloud by Example 2. GCP Data Pipeline Patterns
**And also, something New…
7. 7
Virtual Machines /
GCE
• Fast
• Spin up in seconds
• Tools - SSH, gcloud console
• Flexible
• Custom sizing – slider
• OS variety – Linux or Windows
• Cheap and Simple
• Auto discount for use
• Pre-emptible
Storage / GCS
• Fast
• Very fast within region
• Tools included
• Flexible
• 4 storage options
• Simple to use / understand
• Cheap
• Pricing by type
11. 11
Big Data > Data Warehouse
Reference table
Query / Compute
BigQuery
Customer Lists / Reference
Data
Export Ad
Data
Cloud Storage
Id matching
Cloud Dataflow
Marketing List
DoubleClick
Campaign Manager
Google Analytics
Relevant Users
Cloud Storage
Analysts
DataStudio
360
Dashboards
15. 15
Big Data > Time Series Analysis
Batch Storage
BigQuery
Storage
Cloud Storage
Time Series Processing
Cloud Dataflow
Analysis
Cloud Datalab
Storage
Cloud
Bigtable*
Processing
Cloud Dataproc
Time Series Files
Cloud Storage
ML
Cloud ML
Streaming
Time Series Streaming
Cloud Pub/Sub
*Note: Use Bigtable with
NoSQL workloads of 1 TB or more
16. 16
Streaming
Big Data > Complex Event Processing
Cloud Apps
Compute Engine
Streamin
g
Batch
Push to Devices
App Engine
Rules Engine
Cloud Dataflow Data Analysis
Cloud Datalab
Mobile Devices
Push Notifications
Report &
Share
Business Analysis
Cloud Apps
Compute
Engine
On-Premises
Databases
On-Premises
Applications
Processed Events
Cloud Bigtable
Events Time Series
Data
Warehouse
BigQuery
Execution Results
Streaming
Cloud Pub/Sub
Transactions
Processing
Cloud Dataflow
Transaction Streams
Messaging
Cloud Pub/Sub
Rules Actions
ETL
Cloud Dataflow
Transform Data
Cloud Data
Cloud Storage
Rules Engine
Cloud Dataproc
17. 1717
Files
• Cloud Storage
Compute
• Big Query
• Cloud Dataflow
Other
• 3rd party ETL
• 3rd party dashboards
Core Products for Data
Warehousing
More on Big Query…
• Interactive or Batch query
• ANSI SQL compliant
• Cost control - Purchase ‘slots’
• NoOps Data Warehouse
28. 28
Patient
Analytics
Life Sciences > Patient Monitoring
Analytics
Process Data
Prediction API
Ingest
Cloud Pub/Sub
Storage
Cloud Bigtable
Alerts
Notifications
Cloud Pub/Sub
Health Care
Professional
Patient Monitors
(pulse, blood
sugar, exercise)
29. 29
Private Datasets Public Datasets
Life Sciences > Variant Analysis
MSSNG Autism
Cloud Storage
Scientist
High
Throughput
Genome
Sequencers
1000 Genomes
Cloud Storage
Patient Data
Cloud Storage
Illumina Platform
Cloud Storage
Ref Genomes
Cloud Storage
TCGA
Cloud Storage
Analytics
Online Analytics
BigQuery
Batch Analytics
Cloud Dataflow
Lab Notebooks
Cloud Datalab
Data Ingest
Genomics
BAM
FAST
Q
30. 30
Ingest
Elastic Cluster
Storage
Analytics
Life Sciences > Genomics, Secondary Analysis
Carrier
Interconnect
High
Throughput
Genome
Sequencer
s
Scientist
Raw Datafiles
Cloud Storage
Processed Data
Cloud Storage
Metadata
Cloud SQL
Lab notebooks
Cloud Datalab
HPC Cluster
Compute
Engine
10 Nodes
Ingest Server
Compute
Engine
Online Analytics
BigQuery
Cloud Load
Balancing
Cloud
Network
31. 3131
• Cloud Storage
• Big Query
• Compute Engine
• Cloud Dataflow
• Public datasets on GCP
Core Products for
Bioinformatics