Snowflake Data Science and AI/ML at Scale

© 2020 Snowflake Inc. All Rights Reserved
Data Science and
AI/ML at Scale
Scott Hruby | Snowflake Solutions Engineer

© 2020 Snowflake Inc. All Rights Reserved© 2020 Snowflake Inc. All Rights Reserved
Snowflake 101

“The rapid rise of gathered/analyzed digital data
is often core to the holistic success of the fastest growing &
most successful companies of our time around the world.”
– Mary Meeker, Bond Capital
DATA:
THE WORLD’S MOST VALUABLE RESOURCE

DATA:
THE NEW SUPERPOWER

NEW TECHNOLOGY TRENDS
CHANGE HOW WE USE DATA
5
Analytics is growing in
importance, everywhere,
and for everyone
IoT, mobile, and
social open up new
opportunities for insight
Cloud gives us the
ability to scale and
centralize data
Rise of the Cloud Explosion of Data Diversification of Analytics

On Premises
EDW
1st Gen Cloud
EDW
Data Lake,
Hadoop
Cloud Data
Platform
All Data
All Users
Fast Answers
SQL Database
JOURNEY TO A CLOUD DATA PLATFORM

TRADITIONAL DATA ARCHITECTURE
Complex, Costly & Constrained
OLTP
Databases
Enterprise
Applications
Third-Party
Web/Log
Data
IoT
Data
Integration
Data
Transformation
Data
Analytics
Normalization
& Aggregation
Ad Hoc
Analysis
Real-time
Analytics
Operational
Reporting
DATA
SOURCES
DATA
CONSUMERS
ELT
Streaming
ELT
Data Marts
Data Warehouses
Backups
File Sharing
CubesData Lake
CDC Data Science
7

SNOWFLAKE CLOUD DATA PLATFORM
ONE PLATFORM, ONE COPY OF DATA,
MANY WORKLOADS
8
DATA
SOURCES
OLTP DATABASES
ENTERPRISE
APPLICATIONS
THIRD-PARTY
WEB/LOG DATA
IoT
DATA
CONSUMERS
DATA MONETIZATION
OPERATIONAL
REPORTING
AD HOC ANALYSIS
REAL-TIME ANALYTICS

SNOWFLAKE ARCHITECTURE
9

Data Science & AI

DATA SCIENCE IS PREDICTIVE ANALYTICS
Descriptive
Analytics
Diagnostic
Analytics
What
happened?
Why did it
happen?
reports self driving car
Predictive
Analytics
Prescriptive
Analytics
What will
happen?
How can we
make it happen?

85%of #bigdata
projects fail to move
past preliminary
stages
Gartner - Nov. 2017
80% of analytics
insights will not
deliver business
outcomes through
2022
Gartner - Jan 2019
87% of data
science projects
never make it into
production
VentureBeat AI - July 2019
80% of AI projects
will “remain alchemy,
run by wizards”
through 2020
Gartner - Jan 2019

EFFICIENT
DATA
PREPARATION
EXTENSIVE
PARTNER
ECOSYSTEM
CONSOLIDATED
SOURCE FOR
ALL DATA
KEY REQUIREMENTS FOR DATA SCIENCE
Structured, Semi structured, and
Unstructured data
3rd Party Data Sharing
Streaming & Batch
Dedicated compute clusters
for each team
No resource contention
Integration with the latest
ML tools and libraries
Consistent experience for
BI and ML users

ValueofData DATA SCIENCE MATURITY
● Defining business use cases
● Focused on BI
● Exploring tools
● Experimenting with languages
● Training and hiring
● Testing data pipelines
DATA CHALLENGES
● 15% data utilization
● Understanding data
● Preparing data
OVERALL CHALLENGES
● Acquiring talent
● Designing & building consistent
processes & pipelines
● Defining business value
● Working on creating efficiencies
● Mapping ML to business value
● Planning growth & new use cases
● Testing various languages & tools
● Training and Hiring
● Optimizing data pipelines
DATA CHALLENGES
● 70% data utilization + external
● Data wrangling, ETL + data
validation
OVERALL CHALLENGES
● Training and Hiring
● Scalability of tools & team
● Cost containment
● Creating ROI consistently
● Large data science team
● ML built into product/service
● Standardized on tools/languages
● Advanced ML for video, audio,
and images
● Automated data pipelines
DATA CHALLENGES
● 130% data utilization
● Obtaining more data
● Streaming data
OVERALL CHALLENGES
● Retaining talent
● Optimize, simulate, & test model
efficacy
● Staying ahead of competitors
1. IDENTIFYING USE CASES 2. LIMITED PRODUCTION 3. COMPETITIVE ADVANTAGE

© 2020 Snowflake Inc. All Rights Reserved 15
DATA
SOURCES
OLTP DATABASES
ENTERPRISE
APPLICATIONS
THIRD-PARTY
WEB/LOG DATA
IoT
DATA
CONSUMERS
DATA MONETIZATION
OPERATIONAL
REPORTING
AD HOC ANALYSIS
REAL-TIME ANALYTICS
SNOWFLAKE CLOUD DATA PLATFORM

DATA SCIENCE WITH SNOWFLAKE

ML FRAMEWORKS & LIBRARIES
STATISTICAL ALGORITHMS NEURAL NETWORK / DEEP LEARNING
● Suitable for most Data Science problems involving
structured and semi-structured data.
● Good performance with small amounts of data
● Mature - have been around for a while
● Mostly used with unstructured data like images, audio,
and video
● Performance increases with the size of the training data

NOTEBOOK-BASEDAUTOMATION-BASED (AutoML)
ML PARTNER ECOSYSTEM
Powerful but ComplexFast and Easy but Less Customizable
MLlib

© 2020 Snowflake Inc. All Rights Reserved 25
PROVEN BY OVER 5,400 CUSTOMERS

EVER EXPANDING ECOSYSTEM

© 2020 Snowflake Computing Inc. All Rights Reserved
SNOWFLAKE SECURITY AT A GLANCE
Snowflake
Operational Controls
• NIST 800-53
• SOC2 Type 2
• HIPAA
• PCI
• FedRAMP
Access
• All communication
secured & encrypted
• TLS 1.2 encryption
in both trusted and
untrusted networks
• IP Whitelisting
Authentication
• Password Policy
enforcement
• Multifactor Authentication
• SAML 2.0 support for
Federated Authentication
Application
• Flexible user
management
• Role-based
access control for
granular control
• RBAC for data
and actions
Data
• Encrypted at rest
• Hierarchical key model
rooted in Cloud HSM
• Automatic key rotation
• Time Travel 1-90 days
• Tri-Secret Secure
• Query statement
encryption
Infrastructure
• AWS, Azure Physical Security
• AWS, Azure Redundancy
• Regional Data Centers
▪ US
▪ EU
▪ AP
28

© 2020 Snowflake Computing Inc. All Rights Reserved
COMPREHENSIVE DATA PROTECTION
Protection against infrastructure failures
All data transparently & synchronously replicated
3+ ways across independent infrastructure
Protection against corruption & user errors
“Time travel” feature enables instant roll-back to
any point in time during chosen retention window
Long-term data protection
Zero-copy clones + optional export to cloud
object storage enable user-managed data copies
SELECT* FROM T0…
T0 T1 T2
New data Modified data
Daily
Weekly
29

© 2020 Snowflake Computing Inc. All Rights Reserved© 2020 Snowflake Inc. All Rights Reserved
Dynamically Mask Protected (PII, PHI)
Column Data at Query Time
• No change to the stored data
• Mask or partial mask using constant
value, hash, and custom functions
• Unmask for authorized users only
Policy Based Control
• Table/View owners and privileged users
(such as accountadmin) unauthorized
by default
• Centralized policy mgt
Ease of Management
• Apply single policy to multiple columns
• Prevent secure view explosion
Alice
(Unauthorized)
Bob
(Authorized)
ID Phone SSN
101 ***-***-5534 *********
102 ***-***-3564 *********
103 ***-***-9787 *********
ID Phone SSN
101 408-123-5534 *********
102 510-335-3564 *********
103 214-553-9787 *********
POLICIES
INGEST RAW DATA
GOVERNANCE AND SECURITY
Dynamic data masking
PRIVATE PUBLIC GA

DB 1
Table 1
Column 1
DB 1
View 1
Column 1
DB n
Table n
Column n
<policy condition>
<masking function>
Masking Policy
Resource(s)
Policy
Admin
Apply
CASE
WHEN invoker_role() IN (‘pii_role’) THEN val
WHEN invoker_role() IN (‘support’) THEN
regexp_replace(val,'.+@','*****@')
ELSE ‘********’
END;
Masking Policy Example
Unmask
Partial mask
Mask
Masking Policy
• Policy contains condition(s) and masking
function to apply under those conditions
• Policy is applied to one or more table,
view, or external table columns in an
account
• Nested policy execution for views - policy
on table executed before policy on view(s)
Supports
• All data types
• Data sharing
• Streams
• Clone carries over policy associations
Dynamic data masking policies
PRIVATE PUBLIC GA

ID Phone SSN
101 408-123-5534 387-78-3456
102 510-334-3564 226-44-8908
103 214-553-9787 359-9987-0098
Ingest tokenized data
Ingest Protected (PII/PHI) Data
as Externally Tokenized
• Using Protegrity agents on ETL
tools.
De-tokenize for Authorized
Users at Query Time
• Protegrity DSG called using external
functions to de-tokenize data.
• For unauthorized users, Protegrity
DSG is not called.
Policy Based Control
• Table/View owners and privileged
users (such as accountadmin)
unauthorized by default
• Centralized policy mgt
Customer VPC / VNet
Data Security
Gateway
(DSG)
POLICIES
Tokenized
De-tokenized
De-tokenized
Tokenized
ID Phone SSN
101 111-222-3333 000-78-9999
102 002-778-9904 779-66-8908
103 100-887-8888 111-00-8888
REST API
Alice
(Unauthorized)
Bob
(Authorized)
EXTERNAL
FUNCTION
External tokenization using third party
PRIVATE PUBLIC GA

“I can only show you the door. You’re the one
that has to walk through it”

THANK YOU

Snowflake Data Science and AI/ML at Scale

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Snowflake Data Science and AI/ML at Scale

Similar to Snowflake Data Science and AI/ML at Scale (20)

More from Adam Doyle

More from Adam Doyle (20)

Recently uploaded

Recently uploaded (20)

Snowflake Data Science and AI/ML at Scale