Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Active Governance Across the Delta Lake with Alation Slide 1 Active Governance Across the Delta Lake with Alation Slide 2 Active Governance Across the Delta Lake with Alation Slide 3 Active Governance Across the Delta Lake with Alation Slide 4 Active Governance Across the Delta Lake with Alation Slide 5 Active Governance Across the Delta Lake with Alation Slide 6 Active Governance Across the Delta Lake with Alation Slide 7 Active Governance Across the Delta Lake with Alation Slide 8 Active Governance Across the Delta Lake with Alation Slide 9 Active Governance Across the Delta Lake with Alation Slide 10 Active Governance Across the Delta Lake with Alation Slide 11 Active Governance Across the Delta Lake with Alation Slide 12 Active Governance Across the Delta Lake with Alation Slide 13 Active Governance Across the Delta Lake with Alation Slide 14 Active Governance Across the Delta Lake with Alation Slide 15 Active Governance Across the Delta Lake with Alation Slide 16 Active Governance Across the Delta Lake with Alation Slide 17 Active Governance Across the Delta Lake with Alation Slide 18 Active Governance Across the Delta Lake with Alation Slide 19 Active Governance Across the Delta Lake with Alation Slide 20 Active Governance Across the Delta Lake with Alation Slide 21 Active Governance Across the Delta Lake with Alation Slide 22 Active Governance Across the Delta Lake with Alation Slide 23 Active Governance Across the Delta Lake with Alation Slide 24 Active Governance Across the Delta Lake with Alation Slide 25
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

Active Governance Across the Delta Lake with Alation

Download to read offline

Alation provides a single interface to provide users and stewards to provide active and agile data governance across Databricks Delta Lake and Databricks SQL Analytics Service. Understand how Alation can expand adoption in the data lake while providing safe and responsible data consumption.

  • Be the first to like this

Active Governance Across the Delta Lake with Alation

  1. 1. 1 | Proprietary & Confidential Alation Data Governance on the Delta Lake Raja Perumal November 2020
  2. 2. 2 | Proprietary & Confidential Agenda • The Alation Data Catalog • Alation Active Data Governance - Our Approach - Capabilities Overview • Alation and Databricks
  3. 3. 3 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Data Culture Is Quickly Becoming an Imperative The Benefits are Compelling • Insights-driven businesses on track to earn $1.8T by 2021 • 7x more likely to increase revenue as result of big data • 2.8x more likely to have double-digit growth • 1.6x more likely to increase revenue The Stakes are High and Mistakes Costly • Compliance with new laws and regulations • Building analyses and models on bad data • The best defense is a good offense Activities Critical to Data & Analytics Team Success *Insights-Driven Businesses Set the Pace for Global Growth, October 2018
  4. 4. 4 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ The Pandemic Is Accelerating Change In May, we did a full project review and it was determined that the enterprise data catalog initiative was of vital strategic importance to our business.” – LEADING AUTOMOBILE MANUFACTURER COVID-19 is Accelerating Digital Transformation Data & Analytics Are Key Enablers Data and analytics are the key accelerant of an organization’s digitization and transformation efforts.” – The future got accelerated because of #covid19! Are you ready? #PostPandemic ” ”
  5. 5. 5 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ The Alation Vision To empower a curious and rational world Data Culture Data Literacy Enable proper interpretation & analysis Data Governance Take responsibility & authority Data Search & Discovery Find & understand
  6. 6. 6 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Data Is Supposed to Be an Asset, Not a Liability Too much data, not too little Capture tribal knowledge, enable collaboration Balance compliance with access Data Explosion • 63 ZB in 2021 (volume) • 300+ DBMSs (variety) • IoT and Edge (velocity) Evolving Laws • HIPAA / PII / PHI • Europe / GDPR • Calif / CCPA Changing Workforce • Turnover, restructuring • New SMEs, lost stewards • Remote work (WFH)
  7. 7. 7 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ What is a Data Catalog? • A repository of metadata on information sources across an organization - Search & discovery - Collaboration & analysis - Curation & data governance • Catalogs a broad range of information assets - Data sets, tables, articles, reports, queries, visualizations, conversations Answers these core questions: How to find information? Can it be used? Should it be used? How should it be used?
  8. 8. 8 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Five Things To Look For in a Data Catalog Intelligence Behavioral, linguistic and inferential intelligence drive accelerated time to insight. Collaboration Crowdsourcing, conversations and expert steward identification drive usage and trust. Guided Navigation Guide users to the right data to accelerate analyst onboarding. Active Governance Enforce policies at the point of consumption to balance business value and compliance. Broad, Deep Connectivity Maximize access to trusted data to speed discovery and insight.
  9. 9. 9 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Not All Data Catalogs Are Created Equal Tool Adjunct Governance Suite Catalog as Platform Purpose Find assets associated with a given tool or infrastructure One-stop shopping for loosely integrated metadata suite Find, understand, and govern assets across the enterprise Origins The tool or infrastructure Governance and compliance Data search and discovery Orientation Tool / infrastructure Data / control People / access Architecture Add-on Cobbled together Integrated platform Data strategy Offense Defense Both There are three distinct types of catalog on the market today
  10. 10. 10 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ 7 | Proprietary & Confidential The Catalog is a Platform (For Data Management) Solutions APIs/SDKs Search Data Domains SQL Editor Collaboration Glossaries Analytics Active Metadata Catalog Behavior Analysis Engine Universal Connector Framework Platform HDFSRelational BI …Snowflake AWS Azure GCP Data Search & Discovery Governance & Stewardship Data Privacy Cloud Data Migration Other… Databricks The Catalog is a Platform (for Data Management)
  11. 11. 11 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Data Explosion Data Growth (IDC) and increasing regulations. The sum of all data created, captured, or replicated at core, edge, and endpoint locations - will expand from 33 ZB in 2018 to 175 ZB by 2025.” Data Breaches The number of reported data breaches increased 33% in 2019, resulting in more than 7.9 billion personal records exposed.” Top Concern Data governance has surpassed cybersecurity preparedness as the top concern of Chief Audit Executives.” Data Governance is on Everyone’s Mind ” ” ”
  12. 12. 12 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Traditional Approach to Governance is Failing Challenges with traditional approach • Focus on governing data rather than people’s behavior • Prolonged, Expensive • Defensive – Focused primarily on risk mitigation • Lacks engagement with business stakeholders • Not exercised at the point of data use
  13. 13. 13 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Active Data Governance - a Different, Value- Driven Approach People-First Operationalized Collaborative Intelligent Operationalize policy into action at point of use; measure via analytics Guide behavior; discover and formalize relationship with data Automated and guided via AI and ML Crowdsourcing and community-driven Adaptive Adjust to changes in the business environment with agility
  14. 14. 14 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Active Data Governance Process • Pragmatic, not theoretical • Business, not IT-centric • Incremental, not 'big-bang' implementation • Community-driven, not committee-controlled • Guided, not gated participation • Employee value, not steward & IT centric value • Data usage, not documentation dead ends
  15. 15. 15 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Data Analyst Data Scientist IT PersonaData Steward Business User Alation empowers a diverse set of personas along the value chain
  16. 16. 16 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Scale and Automate Governance with Alation Data Policies • Singular view of enterprise data policies and agile approvals • Policies surfaced at the point of data consumption Data Stewardship Data Lineage • Automatic lineage extraction from data sources and BI tools • Cross-system, column-level lineage • Impact analysis Business Glossary • Agile, intelligent stewardship with analytics and dashboards to measure and address curation gaps • Business-approved definitions • Automatic suggestion/association of business terms, auto-titling Crowdsourcing & Collaboration • Out-of-the box collaboration capabilities and gamification Metadata Security • Role-based security • Integration with identity management systems Data Inventorying, Insights • Deep, broad connectivity including QLI; automated metadata extraction • Behavior Insights-top users, assets Data Quality • Data quality alerts, scores, trust flags, warnings cataloged adjacent to data • Quality information surfaced at the point of data consumption PII Discovery & Classification • Sensitive data discovery • Automatic classification & tagging
  17. 17. 17 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Alation & Databricks
  18. 18. 18 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Challenges in Machine Learning on a Cloud Platform Moving to a cloud environment is a complex project requiring significant time and resources Data scientists waste time and energy finding data to use for experiments Data scientists have difficulty finding expertise about a data set, and analysis already done
  19. 19. 19 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Establish a Cloud Environment Popularity identifies essential data assets to migrate, lineage identifies how data is used, warnings communicate new location Unified data service ingests data into data lake, manages ETL and security, and enables access for ML and BI Lower the risk of moving to the cloud, build a dynamic cloud environment and accelerate use of data lakes and cloud processing for faster ML Joint Value Proposition Moving to a cloud environment is a complex project requiring significant time and resources Solution Establish, maintain, and benefit from a cloud data environment
  20. 20. 20 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Select the Best Data for Machine Learning Discover data sources across the enterprise, understand usage patterns and quickly select data for analysis Simplifies the data pipeline across batch and streaming data to create a fast and reliable cloud platform Data scientists can efficiently discover, understand, transform and analyze data at enterprise scale on the full dataset Joint Value Proposition Data scientists waste time and energy finding data to use for experiments Solution Find and use the best data for fast and accurate ML
  21. 21. 21 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Collaborate to Drive Data Culture Context and conversations enable data scientists to collaborate, and the catalog identifies data users and experts Track experiments and models, publish dashboards, and facilitate hand-offs among 1000s of users in real-time Collaboration results in better predictive models and business insights are shared with more users across the enterprise Joint Value Proposition Data scientists have difficulty finding expertise about a data set, and analysis already done Solution Collaborate to share knowledge and increase the value of data
  22. 22. 22 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ DATABRICKS CONFIDENTIAL Alation & SQL Analytics Service DELTA LakeODBC/ JDBC Drivers BI & SQL Client Connectors Routing Service Query Planning Query Execution Databricks SQL Analytics
  23. 23. 23 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ SQL Analytics Service Endpoints SQL Optimized Compute SQL Endpoints give a quick way to setup SQL / BI optimized compute. You pick a tshirt size. Databricks will ensure configuration that provides the highest price/performance. Concurrency Scaling Built-in Virtual clusters can load balance queries across multiple clusters behind the scenes, providing unlimited concurrency.
  24. 24. 24 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™ Alation and Databricks Natural language search across entire enterprise, and find data in data lakes Find data in all enterprise sources, including Databricks and Delta Lake Identify data experts and users for collaboration and connect to build data science models in Databricks People who know about the data can add context for more users to understand data Conversations to collaborate with data experts and users Migrate to a cloud environment with confidence by identifying the most popular data to prioritize
  25. 25. 25 | Proprietary & Confidential Who to Contact to Learn More Raja Perumal Business Development (ISV) Raja.Perumal@alation.com

Alation provides a single interface to provide users and stewards to provide active and agile data governance across Databricks Delta Lake and Databricks SQL Analytics Service. Understand how Alation can expand adoption in the data lake while providing safe and responsible data consumption.

Views

Total views

419

On Slideshare

0

From embeds

0

Number of embeds

2

Actions

Downloads

22

Shares

0

Comments

0

Likes

0

×