IT Professional Shares Expertise on Microsoft Data Platforms
•Download as PPTX, PDF•
41 likes•19,656 views
This document provides an overview and summary of the author's background and expertise. It states that the author has over 30 years of experience in IT working on many BI and data warehouse projects. It also lists that the author has experience as a developer, DBA, architect, and consultant. It provides certifications held and publications authored as well as noting previous recognition as an SQL Server MVP.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to IT Professional Shares Expertise on Microsoft Data Platforms
Similar to IT Professional Shares Expertise on Microsoft Data Platforms (20)
More from James Serra
More from James Serra (19)
Recently uploaded
Recently uploaded (20)
IT Professional Shares Expertise on Microsoft Data Platforms
- 2. About Me Microsoft, Big Data Evangelist In IT for 30 years, worked on many BI and DW projects Worked as desktop/web/database developer, DBA, BI and DW architect and developer, MDM architect, PDW/APS developer Been perm employee, contractor, consultant, business owner Presenter at PASS Business Analytics Conference, PASS Summit, Enterprise Data World conference Certifications: MCSE: Data Platform, Business Intelligence; MS: Architecting Microsoft Azure Solutions, Design and Implement Big Data Analytics Solutions, Design and Implement Cloud Data Platform Solutions Blog at JamesSerra.com Former SQL Server MVP Author of book “Reporting with Microsoft SQL Server 2012”
- 3. I tried to understand the Microsoft data platform on my own… And felt like I was body slammed by Randy Savage: Let’s prevent that from happening…
- 4. Data platform continuum Hybrid Cloud On premises Shared Lower cost Dedicated Higher cost Higher administration Lower administration Off premises
- 5. On-Premises The “evolution” of data platforms
- 6. IaaSOn-Premises The “evolution” of data platforms
- 7. PaaSIaaSOn-Premises The “evolution” of data platforms
- 8. PaaSIaaSOn-Premises Pay per query The “evolution” of data platforms
- 9. Microsoft Big Data Portfolio SQL Server Stretch Business intelligence Machine learning analytics Insights SQL Server 2017 SQL Server 2016 Fast Track Azure SQL DW Databricks Cosmos DB HDInsight Hadoop Analytics Platform System Sequential Scale Out + AcrossScale Up Key Relational Non-relational On-premisesCloud Microsoft has solutions covering and connecting all four quadrants – that’s why SQL Server is one of the most utilized databases in the world Azure SQL Database SQL Server in Azure VM
- 10. VM hosted on Microsoft Azure Infrastructure (“IaaS”) • From Microsoft images (gallery) or your own images (custom) SQL 2008R2 / 2012 / 2014 / 2016 / 2017 Web / Standard / Enterprise Images refreshed with latest version, SP, CU • Windows Server 2008 R2 / 2012 R2 / 2016, Linux RHEL / Ubuntu • Fast provisioning (~10 minutes). • Accessible via RDP and Powershell • Full compatibility with SQL Server “Box” software Pay per use • Per minute (only when running) • Cost depends on size and licensing • EA customers can use existing SQL licenses (BYOL) • Network: only outgoing (not incoming) • Storage: only used (not allocated) Elasticity • 1 core / 2 GB mem / 1 TB 128 cores / 3.5 TB mem / 256 TB
- 11. Azure SQL Database A relational database-as-a-service (“PaaS”), fully managed by Microsoft. For cloud-designed apps when near-zero administration and enterprise-grade capabilities are key. Perfect for organizations looking to dramatically increase the DB:IT ratio.
- 12. Azure SQL Database Managed Instance Managed Instance Instance scoped programming model with high compatibility to on-premises databases Single Standalone managed database best for predictable and stable workloads Elastic pool Shared resource model best for greater efficiency through multi-tenancy Best for modernization at scale with low cost and effort
- 13. Supports compatibility modes (SQL Server 2005+), Instance sizes up to 8TB Security • TDE • SQL Audit • Row level security • Always Encrypted
- 14. Scalable High performanceReliable and available Adapts on-demand to your workload's needs, auto-scaling up to 100TB per database. 100 TB
- 15. Programming Model General Purpose Business Critical Hyperscale Elastic Pools Instance (MI) GA, 8TB GA, 4TB Private Preview, 100TB April private preview Database (Single) GA, 4TB GA, 4TB Public Preview, 100TB GA
- 16. More choices and full integration into Azure’s ecosystem and services Managed community MySQL, PostgreSQL, and MariaDB Scale in seconds with built-in high availability Secure and compliantLanguages and frameworks of your choice Industry-leading global reach AZURE DATABASE SERVICES FOR MYSQL, POSTGRESQL, AND MARIADB Easy Lift and Shift Enterprise Ready My
- 17. SMP vs MPP • Uses many separate CPUs running in parallel to execute a single program • Shared Nothing: Each CPU has its own memory and disk (scale-out) • Segments communicate using high-speed network between nodes MPP - Massively Parallel Processing • Multiple CPUs used to complete individual processes simultaneously • All CPUs share the same memory, disks, and network controllers (scale-up) • SQL Server implementations traditionally have been SMP • Mostly, the solution is housed on a shared storage SMP - Symmetric Multiprocessing
- 18. SMP vs MPP
- 19. Azure SQL Data Warehouse A relational data warehouse-as-a-service, fully managed by Microsoft. Industries first elastic cloud data warehouse with enterprise-grade capabilities. Support your smallest to your largest data storage needs while handling queries up to 100x faster.
- 20. AZURE RELATIONAL DATABASE PLATFORM PowerBI,AppServices,DataFactory, Analytics,ML,Cognitive,Bot… Global Azure with 54 Regions Azure Compute SQL Data Warehouse Azure Storage SQL Database MariaDBPostgreSQL Flexible: On-demand scaling, Resource governance Trusted: HA/DR, Backup/Restore, Security, Audit, Isolation Intelligent: Advisors, Tuning, Monitoring Database Services Platform MySQL
- 21. Azure Database Migration Service (DMS) A seamless, end-to-end solution for moving on-premises SQL Server, Oracle, and other relational databases to the cloud. Azure Database Migration Guide https://datamigration.microsoft.com/
- 22. Relational and non-relational defined Relational databases (RDBMS, SQL Databases) • Example: Microsoft SQL Server, Oracle Database, IBM DB2 • Mostly used in large enterprise scenarios • Analytical RDBMS (OLAP, MPP) solutions are SQL DW, Redshift, Teradata, Netezza Non-relational databases (NoSQL databases) • Example: Azure Cosmos DB, MongoDB, Cassandra • Four categories: Key-value stores, Wide-column stores, Document stores and Graph stores Hadoop: Made up of Hadoop Distributed File System (HDFS), YARN and MapReduce (Ideal for data lake) OLTP vs OLAP/DW SMP vs MPP
- 23. A globally distributed, massively scalable, multi-model database service Column-family Document Graph Turnkey global distribution Elastic scale out of storage & throughput Guaranteed low latency at the 99th percentile Comprehensive SLAs Five well-defined consistency models Table API Key-value Azure Cosmos DB MongoDB API Cassandra API
- 24. Advanced Analytics Social LOB Graph IoT Image CRM INGEST STORE PREP MODEL & SERVE (& store) Data orchestration and monitoring Big data store Transform & Clean Data warehouse AI BI + Reporting Azure Data Factory SSIS Azure Data Lake Storage Gen2 Blob Storage Azure Data Lake Storage Gen1 SQL Server 2019 Big Data Cluster Azure Databricks Azure HDInsight PolyBase & Stored Procedures Power BI Dataflow Azure Data Lake Analytics Azure SQL Data Warehouse Azure Analysis Services SQL Database (Single, MI, HyperScale) SQL Server in a VM Cosmos DB Power BI Aggregations
- 25. Blob Storage Data Lake Store Azure Data Lake Storage Gen2 Large partner ecosystem Global scale – All 50 regions Durability options Tiered - Hot/Cool/Archive Cost Efficient Built for Hadoop Hierarchical namespace ACLs, AAD and RBAC Performance tuned for big data Very high scale capacity and throughput Large partner ecosystem Global scale – All 50 regions Durability options Tiered - Hot/Cool/Archive Cost Efficient Built for Hadoop Hierarchical namespace ACLs, AAD and RBAC Performance tuned for big data Very high scale capacity and throughput
- 26. LRS Multiple replicas across a datacenter Protect against disk, node, rack failures Write is ack’d when all replicas are committed Superior to dual-parity RAID 11 9s of durability SLA: 99.9% GRS Multiple replicas across each of 2 regions Protects against major regional disasters Asynchronous to secondary 16 9s of durability SLA: 99.9% RA-GRS GRS + Read access to secondary Separate secondary endpoint RPO delay to secondary can be queried SLA: 99.99% (read), 99.9% (write) Zone 1 ZRS Replicas across 3 Zones Protect against disk, node, rack and zone failures Synchronous writes to all 3 zones 12 9s of durability Available in 8 regions SLA: 99.9% Zone 2 Zone 3
- 27. Caching Layer (Avere tech) Extra Hot Tier - Premium (SSD + NVME) Hot Tier (HDD) Cool Tier (HDD) Cooler Tier Archive Tier Deep Storage Tier (Glass, DNA, etc.) Analytics Engines (Hadoop, Spark, SCOPE …) High Performance Compute AI / ML Current Future Edge Azure File Sync Azure Backup Data Box Data Box Edge Azure Stack REST HDFS NFS SMB … Automatic Lifecycle Management Avere FXT
- 28. File Sync • Windows Srv <-> Azure • Local caching • With offline (Databox) can 'sync' remainder Fuse • Mount blobs as local FS • Commit on write • Linux Site Replication • On premise & cloud • Windows, Linux • Physical, virtual • Hyper-V, VMWare Network Acceleration • Aspera • Signiant AZCopy • Throughput +30% • S3 to Azure Blobs • Sync to cloud • Hi Latency 10-100% NetApp • CloudSync • SnapMirror • SnapVault Data Factory • On premise & cloud sources • Structured & unstructured • Over 60 connectors • UI design data flow Partners • Peer Global File Service • Talon FAST • Zerto • … Offline • Data Box • Data Box Heavy • Data Box Disk • Disk Import / Export Fast Data Transfer microsoft.com/en-us/garage/profiles/fast-data-transfer/
- 29. Data Box Heavy PREVIEW • Capacity: 1 PB • Weight 500+ lbs • Secure, ruggedized appliance • Preview September 2018 • Same service as Data Box, but targeted to petabyte- sized datasets. Data Box Gateway PREVIEW • Virtual device provisioned in your hypervisor • Supports storage gateway, SMB, NFS, Azure blob, files • Preview: September 2018 • Virtual network transfer appliance (VM), runs on your choice of hardware. Data Box Edge PREVIEW • Local Cache Capacity: ~12 TB • Includes Data Box Gateway and Azure IoT Edge. • Preview: September 2018 • Data Box Edge manages uploads to Azure and can pre-process data prior to upload. Data Box • Capacity: 100 TB • Weight: ~50 lbs • Secure, ruggedized appliance • GA September 2018 • Data Box enables bulk migration to Azure when network isn’t an option. Data Box DiskPREVIEW • Capacity: 8TB ea.; 40TB/order • Secure, ruggedized USB drives orderable in packs of 5 (up to 40TB). • Currently in Preview • Perfect for projects that require a smaller form factor, e.g., autonomous vehicles. Order Fill UploadSend Return Cloud to Edge Edge to Cloud Pre-processing ML Inferencing Network Data Transfer Edge Compute Offline Data Transfer Online Data Transfer
- 30. Exactly what is a data lake? A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until it is needed. • Inexpensively store unlimited data • Collect all data “just in case” • Store data with no modeling – “Schema on read” • Complements EDW • Frees up expensive EDW resources • Quick user access to data • ETL Hadoop tools • Easily scalable • Place to move older data (archive) • Place to backup data to
- 31. Needs data governance so your data lake does not turn into a data swamp!
- 32. Objectives Plan the structure based on optimal data retrieval Avoid a chaotic, unorganized data swamp Data Retention Policy Temporary data Permanent data Applicable period (ex: project lifetime) etc… Business Impact / Criticality High (HBI) Medium (MBI) Low (LBI) etc… Confidential Classification Public information Internal use only Supplier/partner confidential Personally identifiable information (PII) Sensitive – financial Sensitive – intellectual property etc… Probability of Data Access Recent/current data Historical data etc… Owner / Steward / SME Subject Area Security Boundaries Department Business unit etc… Time Partitioning Year/Month/Day/Hour/Minute Downstream App/Purpose Common ways to organize the data:
- 33. Data Warehouse Serving, Security & Compliance • Business people • Low latency • Complex joins • Interactive ad-hoc query • High number of users • Additional security • Large support for tools • Dashboards • Easily create reports (Self-service BI) • Know questions
- 34. What is Azure Databricks? A fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure Best of Databricks Best of Microsoft Designed in collaboration with the founders of Apache Spark One-click set up; streamlined workflows Interactive workspace that enables collaboration between data scientists, data engineers, and business analysts. Native integration with Azure services (Power BI, SQL DW, Cosmos DB, Blob Storage) Enterprise-grade Azure security (Active Directory integration, compliance, enterprise -grade SLAs)
- 35. Azure HDInsight Hadoop and Spark as a Service on Azure Fully-managed Hadoop and Spark for the cloud 100% Open Source Hortonworks data platform Clusters up and running in minutes Managed, monitored and supported by Microsoft with the industry’s best SLA Familiar BI tools for analysis, or open source notebooks for interactive data science 63% lower TCO than deploy your own Hadoop on-premises* *IDC study “The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight”
- 36. Hortonworks Data Platform (HDP) 3.0 Simply put, Hortonworks ties all the open source products together (20) (under the covers of HDInsight 4.0 – public preview)
- 39. Compute pool SQL Compute Node SQL Compute Node SQL Compute Node … Compute pool SQL Compute Node IoT data Directly read from HDFS Persistent storage … Storage pool SQL Server Spark HDFS Data Node SQL Server Spark HDFS Data Node SQL Server Spark HDFS Data Node Kubernetes pod Analytics Custom apps BI SQL Server master instance Node Node Node Node Node Node Node SQL Data mart SQL Data Node SQL Data Node Compute pool SQL Compute Node Storage Storage
- 41. © Microsoft Corporation Machine learning and AI portfolio What engines do you want to use? Deployment target Which experience do you want? Build your own or consume pre-trained models? Microsoft ML & AI products Build your own Azure Machine Learning Code first (On-prem) ML Server On-prem Hadoop SQL Server (cloud) BYOT SQL Server Hadoop Azure Batch DSVM Spark Visual tooling (cloud) AML Studio Consume Cognitive services, bots Spark ML, SparkR, SparklyR Notebooks Jobs Azure Databricks Spark When to use what
- 42. © Microsoft Corporation Advanced analytics pattern in Azure Azure Data Lake store Azure Storage HDInsightAzure Databricks Azure ML Services ML server Model training Long-term storage Data processing Azure Data Lake Analytics Azure ML Studio SQL Server (in-database ML) Azure Databricks (Spark ML) Data Science VM Cosmos DB Serving storage SQL DB SQL DW Azure Analysis Services Cosmos DB Batch AI SQL DB Azure Data Factory Orchestration Azure Container Service Trained model hosting SQL Server (in-database ML) Data collection and understanding, modeling, and deployment Sensors and IoT (unstructured) Logs, files, and media (unstructured) Business/custom apps (structured) Applications Dashboards Power BI
- 45. Artificial Intelligence Decision Tree Big Data Decision Tree v4 Business Intelligence Solutions Decision Tree
- 46. Q & A ? James Serra, Big Data Evangelist Email me at: jamesserra3@gmail.com Follow me at: @JamesSerra Link to me at: www.linkedin.com/in/JamesSerra Visit my blog at: JamesSerra.com (where this slide deck is posted via the “Presentations” link on the top menu)
- 49. INGEST STORE PREP & TRAIN MODEL & SERVE C L O U D D A T A W A R E H O U S E Azure Data Lake Store Gen2 Logs (unstructured) Azure Data Factory Microsoft Azure also supports other Big Data services like Azure HDInsight to allow customers to tailor the above architecture to meet their unique needs. Media (unstructured) Files (unstructured) PolyBase Business/custom apps (structured) Azure SQL Data Warehouse Azure Analysis Services Power BI
- 50. INGEST STORE PREP & TRAIN MODEL & SERVE M O D E R N D A T A W A R E H O U S E Azure Data Lake Store Gen2 Logs (unstructured) Azure Data Factory Azure Databricks Microsoft Azure also supports other Big Data services like Azure HDInsight to allow customers to tailor the above architecture to meet their unique needs. Media (unstructured) Files (unstructured) PolyBase Business/custom apps (structured) Azure SQL Data Warehouse Azure Analysis Services Power BI
- 51. A D V A N C E D A N A L Y T I C S O N B I G D A T A INGEST STORE PREP & TRAIN MODEL & SERVE Cosmos DB Business/custom apps (structured) Files (unstructured) Media (unstructured) Logs (unstructured) Azure Data Lake Store Gen2Azure Data Factory Azure SQL Data Warehouse Azure Analysis Services Power BI PolyBase SparkR Azure Databricks Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure Machine Learning to allow customers to tailor the above architecture to meet their unique needs. Real-time apps
- 52. INGEST STORE PREP & TRAIN MODEL & SERVE R E A L T I M E A N A L Y T I C S Sensors and IoT (unstructured) Apache Kafka for HDInsight Cosmos DB Files (unstructured) Media (unstructured) Logs (unstructured) Azure Data Lake Store Gen2Azure Data Factory Azure Databricks Real-time apps Business/custom apps (structured) Azure SQL Data Warehouse Azure Analysis Services Power BI Microsoft Azure also supports other Big Data services like Azure IoT Hub, Azure Event Hubs, Azure Machine Learning to allow customers to tailor the above architecture to meet their unique needs. PolyBase
- 53. INGEST STORE MODEL & SERVE D A T A M A R T C O N S O L I D A T I O N Azure Data Lake Store Gen2 Azure SQL Data Warehouse Azure Data Factory Azure Analysis Services Power BI RDBMS data marts Hadoop Microsoft Azure also supports other Big Data services like Azure HDInsight to allow customers to tailor the architecture to meet their unique needs. PolyBase
- 54. INGEST STORE PREP & TRAIN MODEL & SERVE H U B & S P O K E A R C H I T E C T U R E F O R B I Azure SQL Data Warehouse PolyBase Business/custom apps (structured) Power BI Microsoft Azure supports other services like Azure HDInsight to allow customers a truly customized solution. Multiple Azure Analysis Services instances SQL Multiple Azure SQL Database instances Data Marts Data Cubes Azure Databricks Logs (unstructured) Media (unstructured) Files (unstructured) Azure Data Lake Store Gen2Azure Data Factory
- 55. INGEST STORE PREP & TRAIN MODEL & SERVE A U T O S C A L I N G D A T A W A R E H O U S E Microsoft Azure supports other services like Azure HDInsight to allow customers a truly customized solution. Azure Analysis Services Azure Functions (Auto-scaling) Business/custom apps (structured) Logs (unstructured) Media (unstructured) Files (unstructured) Azure SQL Data Warehouse PolyBase Power BIAzure Data Lake Store Gen2Azure Data Factory Azure Databricks
- 56. D A T A W A R E H O U S E M I G R A T I O N INGEST STORE PREP & TRAIN MODEL & SERVE Azure also supports other Big Data services like Azure HDInsight to allow customers to tailor the architecture to meet their unique needs. Business/custom apps (structured) Azure SQL Data Warehouse Business/custom apps Azure Data Lake Store Gen2 Logs (unstructured) Azure Data Factory Azure Databricks Media (unstructured) Files (unstructured) Azure Analysis Services Power BI PolyBase
Editor's Notes
- Data Platform Overview Want to see a high-level overview of the products in the Microsoft data platform portfolio in Azure? I’ll cover products in the categories of OLTP, OLAP, data warehouse, storage, data transport, data prep, data lake, IaaS, PaaS, SMP/MPP, NoSQL, Hadoop, open source, reporting, machine learning, and AI. It’s a lot to digest but I’ll categorize the products and discuss their use cases to help you narrow down the best products for the solution you want to build.
- http://www.ispot.tv/ad/7f64/directv-hang-gliding Mention WWF, then WWE What is Randy Savage’s nickname (Macho Man)? Who won the final match of the first WrestleMania Hulk Hogan and Mr. T def. Roddy Piper and Paul Orndorff. Who won in the final match for WrestleMania III: Hulk Hogan (c) def André the Giant for the WWF World Heavyweight Championship. What was the name of the show starring "Rowdy" Roddy Piper: Piper’s Pit. What was the nickname for announcer Gene Okerlund: Mean Gene. Who did Hulk Hogan defeat in 1984 to become World Champion? The Iron Sheik Who did Andre the Giant eliminate to win the battle royal at WrestleMania 2? Bret "The Hitman" Hart What was the nickname of Jimmy Snuka? "Superfly"
- One of the first things to understand in any discussion of Azure versus on-premises SQL Server databases is that you can use it all. Microsoft’s Data Platform leverages SQL Server technology and makes it available across physical on-premises machines, private cloud environments, third party hosted private cloud environments, and public cloud. This enables you to meet unique and diverse business needs through a combination of on-premises and cloud-hosted deployments, while using the same set of server products, development tools, and expertise across these environments. As seen in the diagram, each offering can be characterized by the level of administration you have over the infrastructure (on the X axis), and by the degree of cost efficiency achieved by database level consolidation and automation (on the Y axis). When designing an application, four basic options are available for hosting the SQL Server part of the application: SQL Server on nonvirtualized physical machines SQL Server in on-premises virtualized machines (private cloud) SQL Server in Azure Virtual Machine (public cloud) Azure SQL Database (public cloud)
- Before the advent of the cloud. FInd someone to provision physical hardware, servers on the hardware, and applications on those physical servers. So many! Lots of steps, parts, counter productive. All good things begin with a story. So before I talk to you about serverless, I wanted to discuss some history of how apps have been built through the years and how that has led us to the computing options we have today. Buy your own hardware and software Run it in your own data-centers Deal with a multitude of questions # of servers Physical security Hardware failure Software patching Backup Essentially a lot of questions, not all of which might be directly related to running your business. For example, if you are running an insurance company, you want to spend your time growing and innovating in the insurance domain, not worrying about how to handle servers.
- Along came the cloud, way faster than provisioning hardware. So much innovation and rapid appilcation building. BUt Still have questions about servers and scaling. Along came the cloud, and provided some relief from some of those questions. IaaS in the cloud, allowed you to forget about managing all that physical hardware. Instead of hosting it yourself, you could now utilize servers hosted by someone else. While the worries around managing physical infrastructure are gone, there are a number of questions that still need to thought about: How many virtual machines do I need? How many would I need in future when my usage grows? How to handle updating the servers for the latest security patches How to think about deploying my code and the various dependencies, such as application runtimes, to the server Who monitors my apps and so on This means that you still haven’t reached that ideal state where you could focus purely on your business and its problems.
- Trying to remove infrastructure. Don't care about patching OS or manging instance of your VM. All you care about is your web app – still have chores / non-business questions. Well, the cloud continue to help with that problem and with the advent of PaaS, there was even more simplification. PaaS allows you to just worry about your app, and let the cloud vendor take care of not only the physical hardware, but also the ongoing management of the operating environment like Operating Systems, Virtual Machines or Containers, as well as the various application runtimes, whether it is Java Runtime, .NET, Node.js, and so on. You are not completely free from thinking about servers however, since you still have to worry about how to scale your app and accordingly specify the number and size of the various compute resources your apps would need at various times.
- The server exists, it's not yours. NO questions about the server. The latest in this series of evolution is Serverless – which further abstracts the underlying infrastructure from you, and allows you to focus only on a single question – how are you going to build your app to achieve the best results for your business, or whatever it is that you are trying to do with your apps. Whether it is the code to delight your customers through a great experience, or the logic to communicate securely with a business partner, that is where most organization want to spend their creative energies on. And serverless allows you to do just that, focus on your code and forget about the infrastructure. This is why we believe it is the platform for next generation of apps. Now before we jump into explaining the benefits of serverless, let’s call out what it’s not. It’s not the total lack of servers, that would a bit difficult. Instead, it’s a system where you don’t have to think about Servers at all, so for you, they are invisible.
- Our portfolio of products provides customers with the power to deploy the solution that suits their business needs. Your choice of platform, whether on-premises, hybrid or private or public cloud, doesn’t limit you now or in the future. Migrating or expanding becomes an easy process and doesn’t require excessive downtime or introduce potential threats to your business success. With Microsoft, you can seamlessly scale up to larger processing and storage capabilities, or scale out by adding additional servers in parallel arrangement. T: SQL Server is a trusted market leader, and it’s the cornerstone of our data warehouse offering.
- With digital transformation in mind, let’s focus on how SQL Database provides the low-cost, low-friction option to migrating your SQL Server data at scale to SQL Database – without having to re-architect your apps. Introducing Azure SQL Database Managed Instance SQL Database Managed Instance is an expansion of the existing SQL Database service designed to enable database lift-and-shift to a fully-managed PaaS, without re-designing the application. SQL Database Managed Instance provides high compatibility with the on-premises SQL Server programming model and out-of-box support for the large majority of SQL Server features and accompanying tools and services. It’s important to note that Managed Instance isn’t a new service – it is a third deployment option within Azure SQL Database, sitting alongside single databases and elastic pools. As part of Azure SQL Database, Microsoft’s fully managed cloud database service, it inherits all its built-in PaaS features.
- The Hyperscale service tier is a highly scalable performance tier that leverages the Azure architecture to scale out resources for SQL Database substantially beyond the limits available for the General Purpose and Business Critical service tiers. Hyperscale provides the following additional capabilities: Reliable and available Because it’s part of SQL Database, it has multiple levels of redundancy and no single points of failure. It also provides a 99.99% availability SLA. Scalable Support for up to 100 TB of database size Compute and storage scale independently Nearly instantaneous database backups (based on file snapshots stored in Azure Blob storage) regardless of size with no IO impact on Compute High performance Fast database restores (based on file snapshots) in minutes rather than hours or days (not a size of data operation) Higher overall performance due to higher log throughput and faster transaction commit times regardless of data volumes Rapid Scale up - you can, in constant time, scale up your compute resources to accommodate heavy workloads when needed, and then scale the resources back down when not needed. Rapid scale out - you can provision one or more read-only nodes for offloading your read workload and for use as hot-standbys
- Azure Database for MySQL provides fully managed enterprise ready community MySQL database as a service for service for app development and deployment. Being community MySQL allows you to easily lift and shift to the cloud and use languages and frameworks for your choice. On top of that you get built-in high availability and capability to scale in seconds, helping you easily adjust to changes in customer demands. Additionally, you benefit from the unparalleled security and compliance, including Azure IP advantage, as well as Azure’s industry leading reach with more datacenters than any other cloud provider. All this with a flexible pricing model so you can choose resources for your workload with no hidden cost. Languages and Frameworks of your choice Being based on the community-editions of MySQL, PostgreSQL and MariaDB mean you can use the existing development languages frameworks and tools you already use for your apps. In addition, Azure Database Services are deeply integrated with Azure Web Apps to provide a streamlined provisioning and management experience for common frameworks (like WordPress, Drupal, Joomla) and languages (PHP, Node.js, Ruby) to provide a best-in-class PaaS experience. Scale in Seconds with built-in HA Azure Database Services are built upon the same service fabric framework that has been powering SQL Database for years. Unlike an VM-based PaaS offering like AWS RDS, Azure Database Services do not have the overhead a full VM stack has (e.g.; Linux OS + DB). Running on in a secured container implementation (SQLPAL, a very light-weight SQL OS), Azure Database Services can provision a new server in seconds in the event that a primary server hangs or crashes whereas in a traditional VM-based implementation the entire Linux (or Windows) OS stack has to bootstrap before the DB service loads. This means the entire experience of a failover can happen in as little as 30-45 seconds – and most importantly WITHOUT the need for a replica. AWS RDS requires deployment in Multi-AZ in order to achieve 99.95% SLA, which doubles your costs as you have 2 DB servers running at all times. With Azure Database Services, no replicas are needed which means no additional cost, or maintenance, by the customer. Additionally, this HA infrastructure enables the ability for Azure Database Services to scale performance on the fly. When a customer needs to scale-up for workload spikes, by simply changing a slider in the portal, a new server is provisioned at a higher performance level and the previous server’s DNS name and storage is connected to the new instance. Scaling can take a little time as 20 seconds meaning customers can scale performance, up or down, with little/no downtime to the application. Secure and Compliant “Secure by default” is the standard for any Azure service, meaning elements such as SSL encryption between the database and application are turned on by default. Additionally, all data at-rest is encrypted by default in Azure storage using AES 256 bit encryption. And since Azure Database Services are using OSS database engines, the Azure IP Advantage means that customers do not have to worry about litigation using an OSS product in Azure. Microsoft provides indemnification for any OSS first party workload in Azure. Industry-leading global reach With more regions across the globe than any other public cloud provider, Azure offers the ability to have the most globally distributed MySQL, PostgreSQL or MariaDB-based application in the world.
- SMP is one server where each CPU in the server shares the same memory, disk, and network controllers (scale-up). MPP means data is distributed among many independent servers running in parallel and is a shared-nothing architecture, where each server operates self-sufficiently and controls its own memory and disk (scale-out).
- SMP is one server where each CPU in the server shares the same memory, disk, and network controllers (scale-up). MPP means data is distributed among many independent servers running in parallel and is a shared-nothing architecture, where each server operates self-sufficiently and controls its own memory and disk (scale-out).
- 19
- LAMP stack (Linux, Apache, MySQL, PHP/ Python/ Perl)
- Azure Cosmos DB offers the first globally distributed, multi-model database service for building planet scale apps. It’s been powering Microsoft’s internet-scale services for years, and now it’s ready to launch yours. Only Azure Cosmos DB makes global distribution turn-key. You can add Azure locations to your database anywhere across the world, at any time, with a single click. Cosmos DB will seamlessly replicate your data and make it highly available. Cosmos DB allows you to scale throughput and storage elastically, and globally! You only pay for the throughput and storage you need – anywhere in the world, at any time.
- 24
- https://azure.microsoft.com/en-us/support/legal/sla/storage/v1_3/
- Transfer options: https://www.microsoft.com/en-us/garage/profiles/fast-data-transfer/ Azure data box
- Also called bit bucket, staging area, landing zone or enterprise data hub (Cloudera) http://www.jamesserra.com/archive/2014/05/hadoop-and-data-warehouses/ http://www.jamesserra.com/archive/2014/12/the-modern-data-warehouse/ http://adtmag.com/articles/2014/07/28/gartner-warns-on-data-lakes.aspx http://intellyx.com/2015/01/30/make-sure-your-data-lake-is-both-just-in-case-and-just-in-time/ http://www.blue-granite.com/blog/bid/402596/Top-Five-Differences-between-Data-Lakes-and-Data-Warehouses http://www.martinsights.com/?p=1088 http://data-informed.com/hadoop-vs-data-warehouse-comparing-apples-oranges/ http://www.martinsights.com/?p=1082 http://www.martinsights.com/?p=1094 http://www.martinsights.com/?p=1102
- https://www.sqlchick.com/entries/2017/12/30/zones-in-a-data-lake https://www.sqlchick.com/entries/2016/7/31/data-lake-use-cases-and-planning Question: Do you see many companies building data lakes? Raw: Raw events are stored for historical reference. Also called staging layer or landing area Cleansed: Raw events are transformed (cleaned and mastered) into directly consumable data sets. Aim is to uniform the way files are stored in terms of encoding, format, data types and content (i.e. strings). Also called conformed layer Application: Business logic is applied to the cleansed data to produce data ready to be consumed by applications (i.e. DW application, advanced analysis process, etc). This is also called by a lot of other names: workspace, trusted, gold, secure, production ready, governed, presentation Sandbox: Optional layer to be used to “play” in. Also called exploration layer or data science workspace
- Azure Databricks features – Enhance your teams’ productivity Get started quickly by launching your new Spark environment with one click. Share your insights in powerful ways through rich integration with PowerBI. Improve collaboration amongst your analytics team through a unified workspace. Innovate faster with native integration with rest of Azure platform. Build on the most compliant and trusted cloud Simplify security and identity control with built-in integration with Active Directory. Regulate access with fine-grained user permissions to Azure Databricks’ notebooks, clusters, jobs and data. Build with confidence on the trusted cloud backed by unmatched support, compliance and SLAs. Scale without limits Operate at massive scale without limits globally. Accelerate data processing with the fastest Spark engine.
- https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-component-versioning https://azure.microsoft.com/en-us/blog/enterprises-get-deeper-insights-with-hadoop-and-spark-updates-on-azure-hdinsight/ https://azure.microsoft.com/en-us/blog/top-8-reasons-to-choose-azure-hdinsight/
- Key points: Summarize key benefits for Azure Analysis Services Azure Analysis Services helps you transform complex data from different data sources into a BI semantic model, so users in your organization can easily gain insights by connecting to the data models using tools like Excel, Power BI, and others to create reports and perform ad hoc data analysis Talk Track Transform Complex Data into rich BI semantic models: Azure Analysis Services Analysis Services helps you transform complex data into a single business user friendly data model making it easy for business users to understand and analyze data across different data sources. Gain instant insights with in-memory cache using your preferred visualization tools : Not only can business users get insights from data easily using their preferred data visualization tool, whether it is Power BI, Excel or other major data visualization tools, but with the in-memory cache capabilities of Azure Analysis Services, users can gain insights over billions of rows of data at the speed of thought Proven Technology: Azure Analysis Services is based on the proven analytics engine in SQL Server 2016 Analysis Services, that has helped organizations turn complex data into a trusted, single source of truth for years. This means that BI professionals who are familiar with SQL Server Analysis Services, tabular models can get started quickly and do not need to learn new tools or skills. Analytics engine as-a-service (provision fast, scale faster): The same proven enterprise grade BI platform is now available as a fully managed service in Azure. With the power of the trusted Microsoft Cloud, you do not need to manage infrastructure on-premises and can benefit from the scalability of the cloud. Additionally you can use Azure Resource Manager to create and deploy an Azure Analysis Services instance within seconds, and use backup restore to quickly move your existing models to Azure Analysis Services and take advantage of the scale, flexibility and management benefits of the cloud. Scale up, scale down, or pause the service and pay only for what you use. Azure Analysis Services is built for Hybrid BI - Organizations store data in the cloud and on-premises. Azure Analysis Services is built for hybrid data. Data can be access in the cloud, on-premises or a combination of both, enabling a hybrid solution. So - you do not have to move on-premises data to the cloud. To summarize, Azure Analysis Services is simple to use – it is easy to get started, you can use your existing skills to create BI semantic models, and your favorite data visualizations tools to analyze your data.
- Increase analytics and apps performance with scale out data marts
- 3/17/2019
- Microsoft Azure supports other services like Azure HDInsight, Azure Data Lake, Azure IoT Hub, Azure Events Hub in various layers of the architecture above to allow customers a truly customized solution.
- 1) Copy source data into the Azure Data Lake Store (twitter data example) 2) Massage/filter the data using Hadoop (or skip using Hadoop and use stored procedures in SQL DW/DB to massage data after step #5) 3) Pass data into Azure ML to build models using Hive query (or pass in directly from Azure Data Lake Store) 4) Azure ML feeds prediction results into the data warehouse 5) Non-relational data in Azure Data Lake Store copied to data warehouse in relational format (optionally use PolyBase with external tables to avoid copying data) 6) Power BI pulls data from data warehouse to build dashboards and reports 7) Azure Data Catalog captures metadata from Azure Data Lake Store and SQL DW/DB 8) Power BI and Excel can pull data from the Azure Data Lake Store via HDInsight 9) To support high concurrency if using SQL DW, or for easier end-user data layer, create an SSAS cube