Active Governance Across the Delta Lake with Alation

1 | Proprietary & Confidential
Alation
Data Governance on the
Delta Lake
Raja Perumal
November 2020

Agenda
• The Alation Data Catalog
• Alation Active Data Governance
- Our Approach
- Capabilities Overview
• Alation and Databricks

3 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Data Culture Is Quickly Becoming an Imperative
The Benefits are
Compelling
• Insights-driven
businesses on track to
earn $1.8T by 2021
• 7x more likely to
increase revenue as
result of big data
• 2.8x more likely to have
double-digit growth
• 1.6x more likely to
increase revenue
The Stakes are High
and Mistakes Costly
• Compliance with new
laws and regulations
• Building analyses and
models on bad data
• The best defense is
a good offense
Activities Critical to Data &
Analytics Team Success
*Insights-Driven Businesses Set the Pace for Global Growth, October 2018

The Pandemic Is Accelerating Change
In May, we did a full project review and it was
determined that the enterprise data catalog initiative
was of vital strategic importance to our business.”
– LEADING AUTOMOBILE MANUFACTURER
COVID-19 is Accelerating
Digital Transformation
Data & Analytics
Are Key Enablers
Data and analytics are the key accelerant of an
organization’s digitization and transformation efforts.”
–
The future got accelerated because of #covid19! Are you ready?
#PostPandemic
”
”

The Alation Vision
To empower a curious and rational world
Data Culture
Data
Literacy
Enable proper
interpretation & analysis
Data
Governance
Take responsibility
& authority
Data Search
& Discovery
Find &
understand

Data Is Supposed to Be an Asset, Not a Liability
Too much data,
not too little
Capture tribal knowledge,
enable collaboration
Balance compliance
with access
Data Explosion
• 63 ZB in 2021 (volume)
• 300+ DBMSs (variety)
• IoT and Edge (velocity)
Evolving Laws
• HIPAA / PII / PHI
• Europe / GDPR
• Calif / CCPA
Changing Workforce
• Turnover, restructuring
• New SMEs, lost stewards
• Remote work (WFH)

What is a Data Catalog?
• A repository of metadata on information
sources across an organization
- Search & discovery
- Collaboration & analysis
- Curation & data governance
• Catalogs a broad range
of information assets
- Data sets, tables, articles, reports,
queries, visualizations, conversations
Answers these core questions:
How to find information? Can it be used? Should it be used? How should it be used?

Five Things To Look For in a Data Catalog
Intelligence
Behavioral, linguistic and inferential intelligence drive accelerated time to insight.
Collaboration
Crowdsourcing, conversations and expert steward identification drive usage and trust.
Guided Navigation
Guide users to the right data to accelerate analyst onboarding.
Active Governance
Enforce policies at the point of consumption to balance business value and compliance.
Broad, Deep Connectivity
Maximize access to trusted data to speed discovery and insight.

Not All Data Catalogs Are Created Equal
Tool Adjunct Governance Suite Catalog as Platform
Purpose
Find assets associated
with a given tool or
infrastructure
One-stop shopping for
loosely integrated
metadata suite
Find, understand, and
govern assets across
the enterprise
Origins The tool or
infrastructure
Governance and
compliance
Data search and
discovery
Orientation Tool / infrastructure Data / control People / access
Architecture Add-on Cobbled together Integrated platform
Data strategy Offense Defense Both
There are three distinct types of catalog on the market today

The Catalog is a Platform (For Data Management)
Solutions
APIs/SDKs
Search Data Domains SQL Editor Collaboration Glossaries Analytics
Active Metadata Catalog
Behavior Analysis Engine
Universal Connector Framework
Platform
HDFSRelational BI …Snowflake AWS Azure GCP
Data Search &
Discovery
Governance &
Stewardship
Data Privacy Cloud Data Migration Other…
Databricks
The Catalog is a Platform (for Data Management)

Data Explosion
Data Growth (IDC) and
increasing regulations.
The sum of all data
created, captured, or
replicated at core, edge,
and endpoint locations -
will expand from 33 ZB in
2018 to 175 ZB by 2025.”
Data Breaches
The number of reported
data breaches increased
33% in 2019, resulting in
more than 7.9 billion
personal records exposed.”
Top Concern
Data governance has
surpassed cybersecurity
preparedness as the top
concern of Chief Audit
Executives.”
Data Governance is on Everyone’s Mind
”
” ”

Traditional Approach to Governance is Failing
Challenges with traditional approach
• Focus on governing data rather than people’s behavior
• Prolonged, Expensive
• Defensive – Focused primarily on risk mitigation
• Lacks engagement with business stakeholders
• Not exercised at the point of data use

Active Data Governance - a Different, Value-
Driven Approach
People-First Operationalized Collaborative Intelligent
Operationalize policy
into action at point of use;
measure via analytics
Guide behavior;
discover and formalize
relationship with data
Automated and guided
via AI and ML
Crowdsourcing and
community-driven
Adaptive
Adjust to changes in the business
environment with agility

Active Data Governance Process
• Pragmatic, not theoretical
• Business, not IT-centric
• Incremental, not 'big-bang'
implementation
• Community-driven, not
committee-controlled
• Guided, not gated
participation
• Employee value, not steward
& IT centric value
• Data usage, not
documentation dead ends

Data Analyst Data Scientist IT PersonaData Steward Business User
Alation empowers a diverse set of personas along the value chain

Scale and Automate Governance with Alation
Data Policies
• Singular view of enterprise data policies and agile approvals
• Policies surfaced at the point of data consumption
Data Stewardship
Data Lineage
• Automatic lineage extraction from data sources and BI tools
• Cross-system, column-level lineage
• Impact analysis
Business Glossary
• Agile, intelligent stewardship with analytics and dashboards
to measure and address curation gaps
• Business-approved definitions
• Automatic suggestion/association of business terms,
auto-titling
Crowdsourcing & Collaboration
• Out-of-the box collaboration capabilities and gamification
Metadata Security
• Role-based security
• Integration with identity management systems
Data Inventorying, Insights
• Deep, broad connectivity including QLI; automated
metadata extraction
• Behavior Insights-top users, assets
Data Quality
• Data quality alerts, scores, trust flags, warnings cataloged
adjacent to data
• Quality information surfaced at the point of data consumption
PII Discovery & Classification
• Sensitive data discovery
• Automatic classification & tagging

Alation & Databricks

Challenges in Machine Learning on a Cloud Platform
Moving to a cloud
environment is a
complex project
requiring significant
time and resources
Data scientists waste
time and energy
finding data to use for
experiments
Data scientists have
difficulty finding
expertise about a data
set, and analysis
already done

Establish a Cloud Environment
Popularity identifies essential data assets
to migrate, lineage identifies how data is
used, warnings communicate new location
Unified data service ingests data into data
lake, manages ETL and security, and
enables access for ML and BI
Lower the risk of moving to the cloud,
build a dynamic cloud environment and
accelerate use of data lakes and cloud
processing for faster ML
Joint Value
Proposition
Moving to a cloud
environment is a complex
project requiring significant
time and resources
Solution
Establish, maintain,
and benefit from a cloud
data environment

Select the Best Data for Machine Learning
Discover data sources across the
enterprise, understand usage patterns
and quickly select data for analysis
Simplifies the data pipeline across
batch and streaming data to create a
fast and reliable cloud platform
Data scientists can efficiently discover,
understand, transform and analyze data
at enterprise scale on the full dataset
Joint Value
Proposition
Data scientists waste time
and energy finding data to
use for experiments
Solution
Find and use the best data
for fast and accurate ML

Collaborate to Drive Data Culture
Context and conversations enable data
scientists to collaborate, and the
catalog identifies data users and experts
Track experiments and models, publish
dashboards, and facilitate hand-offs
among 1000s of users in real-time
Collaboration results in better predictive
models and business insights are shared
with more users across the enterprise
Joint Value
Proposition
Data scientists have
difficulty finding expertise
about a data set, and
analysis already done
Solution
Collaborate to share
knowledge and increase
the value of data

DATABRICKS CONFIDENTIAL
Alation & SQL Analytics Service
DELTA LakeODBC/
JDBC
Drivers
BI & SQL
Client
Connectors
Routing
Service
Query
Planning
Query
Execution
Databricks
SQL Analytics

SQL Analytics Service Endpoints
SQL Optimized Compute
SQL Endpoints give a quick way to setup SQL /
BI optimized compute. You pick a tshirt size.
Databricks will ensure configuration that
provides the highest price/performance.
Concurrency Scaling Built-in
Virtual clusters can load balance queries across
multiple clusters behind the scenes, providing
unlimited concurrency.

Alation and Databricks
Natural language
search across entire
enterprise, and find
data in data lakes
Find data in all
enterprise sources,
including Databricks
and Delta Lake
Identify data experts
and users for
collaboration and
connect to build data
science models in
Databricks
People who know
about the data can
add context for
more users to
understand data
Conversations to
collaborate with data
experts and users
Migrate to a cloud
environment with
confidence by
identifying the most
popular data to
prioritize

Who to Contact to Learn More
Raja Perumal
Business Development (ISV)
Raja.Perumal@alation.com

Active Governance Across the Delta Lake with Alation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Active Governance Across the Delta Lake with Alation

Similar to Active Governance Across the Delta Lake with Alation (20)

More from Databricks

More from Databricks (20)

Recently uploaded

Recently uploaded (20)

Active Governance Across the Delta Lake with Alation