Understanding AWS Managed Database and Analytics Services | AWS Public Sector Summit 2017
•Download as PPTX, PDF•
2 likes•677 views
The world is creating more data in more ways than ever before. The average internet user in 2017 generates 1.5GB of data per day, with the rate doubling every 18 months. A single autonomous vehicle can generate 4TB per day. Each smart manufacturing plant generates 1PB per day. Storing, managing, and analyzing this data requires integrated database and analytic services that provide reliability and security at scale. AWS offers a range of managed data services that let customers focus on making data useful, including Amazon Aurora, RDS, DynamoDB, Redshift, Spectrum, ElastiCache, Kinesis, EMR, Elasticsearch Service, and Glue. In this session, we discuss these services, share our vision for innovation, and show how our customers use these services today. Learn More: https://aws.amazon.com/government-education/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Understanding AWS Managed Database and Analytics Services | AWS Public Sector Summit 2017
Similar to Understanding AWS Managed Database and Analytics Services | AWS Public Sector Summit 2017 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
Understanding AWS Managed Database and Analytics Services | AWS Public Sector Summit 2017
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dan Neault, AWS DB, Analytics, & AI Customer Programs Scott Donaldson, Senior Director, FINRA June 13, 2017 Understanding AWS Managed Databases and Analytics Services
- 2. AWS Data Services to Accelerate Your Move to the Cloud RDS Open Source RDS Commercial Aurora Migration for DB Freedom DynamoDB & DAX ElastiCache EMR Amazon Redshift Redshift Spectrum AthenaElasticsearch Service Amazon QuickSight Glue Lex Polly Rekognition Machine Learning Databases to Elevate your Apps Relational Non-Relational & In-Memory Analytics to Engage your Data Inline Data Warehousing Reporting Data Lake Amazon AI to Drive the Future Deep Learning, MXNet Database Migration Schema Conversion
- 3. Public Sector Customers Use AWS Database, Analytic, and AI Services
- 4. AWS Data Services to Accelerate Your Move to the Cloud RDS Open Source RDS Commercial Aurora Migration for DB Freedom DynamoDB & DAX ElastiCache EMR Amazon Redshift Redshift Spectrum AthenaElasticsearch Service Amazon QuickSight Glue Lex Polly Rekognition Machine Learning Databases to Elevate your Apps Relational Non-Relational & In-Memory Analytics to Engage your Data Inline Data Warehousing Reporting Data Lake Amazon AI to Drive the Future Deep Learning, MXNet Database Migration Schema Conversion
- 5. Multi-engine support Open Source Commercial Amazon Aurora Automated provisioning, patching, scaling, backup/restore, failover Use with General Purpose SSD or Provisioned IOPS SSD storage High availability with RDS Multi-AZ Amazon RDS: Cheaper, Easier, and Better
- 6. Enterprise-grade fault tolerant solution for production databases Automatic failover Synchronous replication Inexpensive & enabled with one click High Availability Multi-AZ Deployments
- 7. Up To 5x Performance Of High-end MySQL Highly Available and Durable MySQL Compatible* 1/10th The Cost Of Commercial Grade Databases Fastest Growing AWS Service, Ever Amazon Aurora: Speed and Availability of Commercial Databases, with Cost-Effectiveness of Open Source *PostgreSQL compatibility in Open Preview
- 8. BINLOG DATA DOUBLE-WRITELOG FRM FILES TYPE OF WRITE MySQL with Replica Storage MirrorStorage Mirror DC 1 DC 2 StorageStorage Primary Instance Replica Instance AZ 1 AZ 3 Primary Instance Amazon S3 AZ 2 Replica Instance ASYNC 4/6 QUORUM DISTRIBUTED WRITES Replica Instance Amazon Aurora 780K transactions 7,388K I/Os per million txns (excludes mirroring, standby) Average 7.4 I/Os per transaction MySQL IO profile for 30 min. Sysbench run 27,378K transactions 35X MORE 0.95 I/Os per transaction 7.7X LESS Aurora IO profile for 30 min. Sysbench run Aurora, Faster Because it is Built for AWS
- 9. DynamoDB: Non-Relational Managed NoSQL Database Service Schemaless data model Consistent low latency performance Predictable provisioned throughput Seamless scalability with no storage limits High durability & availability (replication across 3 facilities) Easy administration – we scale for you! Low cost DynamoDB DAXApp DynamoDB Accelerator (DAX) offers caching without coding for sub-millisecond read latency and up to 10x throughput
- 10. DynamoDB at Amtrak Built and deployed an operational database and data mart for near-real-time reporting of sales data Developed and released the solution in six months Used cloud native technologies: DynamoDB, Kinesis, Lambda, and S3 Benefits Improved accuracy and single source of truth for sales data Allows decommissioning of four legacy systems Low maintenance and operational costs. No servers to manage.
- 11. Make Almost Any Database Faster and Less Expensive In-Memory Cache Memcached and Redis Fully managed High Speed In-Memory Data Store Persistent high availability Clusters up to 3.5TB Average read and write time of under 500µs (0.5ms) Amazon ElastiCache Provides Sub-millisecond Caching and In-Memory Data
- 12. AWS Data Services to Accelerate Your Move to the Cloud RDS Open Source RDS Commercial Aurora Migration for DB Freedom DynamoDB & DAX ElastiCache EMR Amazon Redshift Redshift Spectrum AthenaElasticsearch Service Amazon QuickSight Glue Lex Polly Rekognition Machine Learning Databases to Elevate your Apps Relational Non-Relational & In-Memory Analytics to Engage your Data Inline Data Warehousing Reporting Data Lake Amazon AI to Drive the Future Deep Learning, MXNet Database Migration Schema Conversion
- 13. Amazon EMR: the Hadoop and Spark Ecosystem, Without the Chaos Design Patterns Amazon S3 as HDFS Core Nodes and Task Nodes Elastic Clusters Transient + Always On Clusters Leverage the Hadoop ecosystem Use Cases Recommendation Engines Personalization Engines Semi-structured/unstructured data Combine disparate data sets Next generation ETL Sentiment analysis Batch analytics Taming Big Data in the Cloud Hadoop, Spark, Presto, Hive and more Easy to use, fully managed Launch a cluster in minutes Baked in security features Pay by the hour and save with Spot
- 14. Amazon Elasticsearch Service Log Analytics & Operational Monitoring Monitor the performance of your apps, web servers, and infrastructure Easy to use, yet powerful data visualization tools to detect issues in near real-time Ability to dig into your logs in an intuitive, fine-grained way Kibana provides fast, easy visualization Search Application or website provides search capabilities over diverse documents Tasked with making this knowledge base searchable and accessible Key search features including text matching, faceting, filtering, fuzzy search, auto complete, and highlighting Query API to support application search
- 15. Amazon Redshift: Cloud Data Warehousing Leader Node Simple SQL endpoint Stores metadata Optimizes query plan Coordinates query execution Compute Nodes Local columnar storage Parallel/distributed execution of all queries, loads, backups, restores, resizes Up to 2 petabytes of managed data Automated ingestion from S3, Kinesis, EMR and DynamoDB Leader Node Compute Nodes S3 EMR DynamoDB EC2
- 16. Large Data Lakes: PB and XB Run SQL queries directly against data in S3 Fast @ exabyte scale Elastic & highly available On-demand, pay-per-queryHigh concurrency: Multiple clusters access same data No ETL: Query data in-place using open file formats Full SQL support S3 SQL Amazon Redshift Spectrum Run SQL queries directly against data in S3 using thousands of nodes Amazon Athena Serverless interactive query service Query an Exabyte of data in under 3 minutes
- 17. Data Catalog Hive metastore compatible metadata repository of data sources Crawls data source to infer table, data type, partition format Job Execution Runs jobs in Spark containers – automatic scaling based on SLA Glue is serverless – only pay for the resources you consume Job Authoring Generates Python code to move data from source to destination Edit with your favorite IDE; share code snippets using Git AWS Glue for Automated, Serverless ETL
- 18. Amazon QuickSight: Fast Business Analytics Data from Many Sources AWS Managed Databases Amazon S3 Databases on Amazon EC2 On-premises databases Excel and CSV Files Salesforce and other SaaS Mobile and Web Access iPhone, Android and Tablet Most popular web browsers Powered by SPICE Super-fast, Parallel, In-memory Calculation Engine Run fast interactive queries on large datasets Low monthly cost per user
- 19. Old-World Vendors and Old-World Policies… You’ve Got Mail! AUDIT Very Expensive Proprietary Lock-In Punitive Licensing Unshackle From H stile Database Vendors
- 20. Freedom Begins with Choice; Migrating Data and Schema AWS Schema Conversion Tool Automatically convert & move tables, views, stored procedures, metadata Highlights and recommends custom actions as needed AWS Database Migration Service Start a migration in literally a few minutes Keep apps running during the migration Replicate from, within, or to Amazon EC2 or managed database services or on-premises 0 1 2 3 4 5 WorkloadQualification Framework Assess workloads by complexity, technology, effort, and other factors Recommends strategy and plans for migration AWS Workload Qualification Framework
- 21. Heterogeneous Migration Oracle private DC to RDS PostgreSQL migration Used the AWS Schema Conversion Tool (AWS SCT) to convert their database schema Used on-going Change Data Capture (CDC) replication to keep databases in sync until they reached the cutover window Benefits Improved reliability of the cloud environment Savings on Oracle licensing costs SCT Assessment Report showed the scope of the migration
- 22. Amazon AI Intelligent Services Powered By Deep Learning
- 23. 23 231
- 24. FINRA: Data Sharing Pre-Cloud Built a data hub of to deal with growing problem of point-to- point dependencies between databases in the data center. FINRA data center App 1 DB App 2 DB App 3 DB App N DB HUB DB FINRA data center App 1 DB App 2 DB App 3 DB App N DB
- 25. FINRA: Data Replication Services on AWS
- 26. FINRA: Varied Analytic Use Cases
- 27. FINRA: Analytics Architecture Validation Data Management Linkage Data Analytics Normalization Amazon EC2 Amazon S3 Amazon Glacier Amazon Redshift Amazon EMR VPC Amazon EMR Amazon RDS Amazon Machine Learning AWS KMS 12 Batch Analytics Interactive & Visualizations Data Science
- 29. FINRA: Universal Data Science Platform 16
- 30. FINRA: Evolution of the Analytics Portfolio
- 31. FINRA: Analytics Impacts • Removed obstacles “Before data analysis of this magnitude required intervention from technology.” “We are now able to see underlying data and visual representation of summaries together with outliers and anomalies. This reduces our time to market on examinations.” “We moved away from requesting raw reports to requesting dashboards that provide meaningful information and tell a story…” • Lowered the cost of curiosity “Analysts are able to quickly obtain a full picture of what happens to an order over time, helping to inform decision making as to whether a rule violation has occurred.” “[W]ith a click we can now compare firms of our choice or defined peer groups. This helps use by reducing a lot of noise…” “Using machine learning algorithms validates our assumptions and makes us data driven” • Optimize batch and interactive workloads without compromise • Greater innovation and more engaged staff 21
- 32. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Questions?
- 33. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank You!
Editor's Notes
- I lead the AWS DB, Analytics, and Artificial Intelligence Customer Programs team. Joining me is Scott Donaldson, a Sr. Dir. of Tech at Financial Industry Regulatory Authority, or FINRA. FINRA is an independent, non-govt regulator for all securities comp’s that do biz with the investor public in US - 90 mill’n of us FINRA reconstructs the market from tens of billions of events, detecting wrongdoing, & then the consequences from that. Scott was a principal in moving FINRA’s DB and analytics onto the cloud over the last 3 years, a strong statement. I appreciate his teamwork with me today.
- gI will start with how AWS offers a broad portf. of Data Svcs across DB, Analytics, & AI to help cust. accelerate Your Move to the cloud. At AWS > 90% of new features & Svcs are driven by cust f’back, and most requests are driven by a desire to make app experiences better. Better includes faster to deploy & run, more responsive & reliable, and easier & less expensive to operate. For DB, we have a strong mix of rel. Open Source and Commer. Engine choices, and Amzn-created Aurora. For the highest scalability & perf. we have non-rel. Amazon-created DynamoDB and Open Source choices. Moving right, our cust use a range of analytics Svcs to Engage your Data. I will talk about a some popular Open Source-based Svcs. And many cust. are migrating from DW appliances to on-demand Cloud services w/decoupled data, using Redshift and Redshift Spectrum. I’ll also cover integ. Svcs for Extract-Transform & Load, serverless query, & visualization. You might know that AI is also a growing part of the AWS Data portf., with deep learning engines, an ML platforms, and Svcs to enable dev’s to easily add voice and pictorial capabilities to applications. Completing this portfolio view, our cust use migration for a broad range of use cases across modernization of existing apps and moving between DB engines, and both for rel. and non-rel. DB and analytics Svcs.
- Looking at momentum today, more than 2,300 gov’t agencies, 7,000 education institutions, and more than 22,000 nonprofit organizations are using our Svcs. Here are the logos of some of the Public Sector agencies and orginaizations s using AWS DB, analytics, & AI services. You might recognize severof them.
- I will first going deeper in our DB portf., both Rel. and Non-Rel. and In-Memory
- For rel DB, our foundation is Amzn’s Rel Data Svcs. RDS is a TCO win, and is Easier, and Better - providing cust. ever-inc. choice, autom. & innov. 5 DB eng. are avail w/RDS, the Open Source eng. of MySQL, PostgreSQL, and MariaDB; & the commercial eng. of Oracle RDBMS and Msft SQL Server. For even higher levels of reliability, scalability & perf, you can also use Amzn Aurora w/MySQL compat & PG compat in open preview. For RDS commercial, you can bring your own licenses or rent them by-the-hour from AWS. The others are provided as part of our services. For every DB engine, RDS automates the heavy lifting, including maint. high perf and HA. Inadequately managed DB are a leading cause of downtime in IT, and lost sleep. For the typical DB team, perhaps 25% of the time goes to infra, 65% to mgmt., and only 10% to biz and app optimization. With DB from AWS, you can get more leverage from your teams and focus in areas that differentiate you, while Amazon takes care of the infras and DB mgmt. On the final point, we have built-in HA and Cross Region replication across multiple availability zones and geographic regions. These are for all engines, including standard editions, not just for Enterprise editions, with over 99.95% availability. Our experience running Amazon.com taught us what it takes to manage and operate relational DB with HA. We bring that learning to you.
- On HA, to give sense of how simple it is, creating fault tol. w/RDS is just checking a box. AWS does to rest the create a highly reliable, auto failover system with replication. If a primary fails, the application keeps running while the standby gets promoted to primary and new standby is created.
- Now I want to talk about Amazon Aurora, which brings you (title). Most rel. DB technologies were built to scale vert. and assume a fairly monolithic arch for each instance. Making that type of arch perform at high Xactional volumes while providing HA and data durability is non-trivial. It often takes a lot of engineering work and maintenance to sustain the target levels of perf, avail, and durability. To solve that problem AWS built Aurora – a fully managed rel. DB taking full advantage of Amazon’s cloud infra. Aurora is compat. with MySQL schema now, and compat with PostgreS in Open Preview is particularly interesting to Oracle B/C semantic compatibility of PostgreSQL with Oracle makes migration easier. It has up to 5x times the perf of high-end MySQL, HA, and 1/10 the cost of commercial DB. Since release in July 2015 Aurora is the fastest growing AWS service. Compelling to get ent class perf without having to purchase an expensive 3rd party solution. It is completely consumption based and has no license costs.
- Here is an illustration of why Aurora performs so much better than native MYSQL. The key is in how Aurora uses AWS infra to its advantage. With a typical DB you need to write everything to a disk which is mirrored to another disk for durability. Then all of that data must be replicated to another instance where it is written to disk, twice, in order to achieve HA. With Aurora the log data is the only thing that must be written to disk. It is asynchronously written in a dist. manner 6 times, but 4 successful writes is a quorum for a successful write. Data is then asynchronously written to S3 for durability/backup purposes. Aurora keeps 3 instances in synch for failover purposes and for offloading read traffic. The result is a dramatic increase in throughput and reduction in I/O while providing superior avail & durability.
- Stepping over from Rel. BD to NoSQL DB, AWS built DynamoDB to provide very stable low latency perf. regardless of how many Xactional requests you might need to support. As with many of our services we provide durability by replicating across 3 physical facilities. Provisioning DynamoDB is simple – you essentially define your table name, set up your key fields, and the R&W throughput you need. We scale the back end for you. If your needs change, you can change the R&W traffic you require and we will dynamically adjust the infra to meet your needs. To provide further perf enhancements we recently announced the Dynamo DB Accel., or DAX. DAX adds a caching mechanism that dramatically increases perf. You can enable a DAX cluster for a DynamoDB tables and offload read traffic from the DB. For workloads that repeatedly read the same key you can get up to 10X perf. It is completely API compatible with Dynamo DB – you enable it by routing API calls to DAX instead of directly to Dynamo. No coding required. DAX is in Preview in US East 1 and US West 2, and eu-West-1 in Ireland. Now lets look at a customer example on Dynamo…..
- Amtrak teamed up w/Deloitte to overhaul their sales DW. It was built using typical legacy tech that was becoming increasingly expensive to maintain. And it was also not providing them the accuracy and speed they were looking for. They went from concept to deployment in 6 mos. They leveraged AWS managed Svcs of DynamoDB, Kinesis, Lambda and S3 for the entire arch with DynamoDB as the core Svc to stores the data. The result is a solution that will save Amtrak significant operational costs and that allows them to retire four legacy apps. The new solution is much simpler to maintain because there are no servers to manage, and it also increases the accuracy of the data and provides it in near real time instead of daily batch loads. If you want to learn more about this solution, they have a separate session at this summit this afternoon.
- To complete our non-rel. DB portf, ElastiCache provides an in-memory cache for apps. It provides an AWS managed implementation of Memcached or Redis. You can think of it a bit like a “shock absorber” for your DB. It functions similar to DAX, and will work with just about any DB, integrated into the persistence layer of an app. It can speed up read heavy workloads to provide sub-millisecond response while reducing load on expensive DB resources. And that concludes our tour through the DB portion of the AWS Data portfolio….
- We will talk next about Analytics. In addition to a comprehensive set of DB solutions, AWS provides a rich set of tools for big data and Analytics These can be provisioned dynamically to allow you to quickly begin analyzing very large volumes of data.
- To start with, Elastic Map Reduce has been in our analytics port. the longest of any tool. We launched it in 2009. Hadoop, Spark, Presto, and Hive are natural fits for the cloud because you can apply vast resources to a data analytics problem quickly. So, we built a managed service to make it easy for customers to use the cloud for big data analytic purposes. You might use EMR to analyze unstruct. or semi-struct. data sets, mult. data sets in various formats, for large scale batch anal. jobs, or ETL at scale. EMR provisions a fully mang’d infras in a few mouse clicks. You can quickly launch clusters, then use tools of your choice from the Hadoop Ecosys. You can have clusters that are persistent in nature that stay on 24X7, or you can have clusters that are transient in nature – just shut down the cluster when you are done and you stop paying for it. For very large data volumes, EMR works seamlessly with S3. EMR can mount S3 as a FS to ingest data into the cluster and utilize tech such as Presto to query data directly in S3. This allows you to scale to access virtually limitless amounts of data. EMR can take advantage of all of the EC2 pricing models such as reserved instances and spot instances to reduce cost.
- Next, Elasticsearch Service is a relatively new addition to our analytics portfolio. It is a managed version of the Open Source product of the same name and also includes Kibana, and integrates with Logstash. It also integ. with many other AWS svcs incl. AWS IoT, AWS Kinesis Firehose, and also connectors for S3, CloudWatch, and CloudTrail. With this service you can quickly stand up an ELK stack to ingest and analyze large volumes of data. Typical use cases include analytics on log files with Kibana built in to quickly build dashboards to visualize data. Another use area is for application search. Elasticsearch ingests your data and indexes it for easy searching.
- Now we will talk about Amazon Redshift. Like Hadoop, DW is a natural fit for cloud computing. The ability to scale out across many compute nodes to query and analyze data is our sweet spot. Redshift is a Massively Parallel Processing, or MPP, DW that uses columnar storage which makes it optimal for analytics processing. There are many DW solutions on the market, many of them HW based, but Redshift is different 1st, Redshift is fully elastic – you can dynamically provision a warehouse from <1TB to 2PB in size, and can grow and shrink elastically with your data volumes. If you need more capacity, just go into the console and change the size of the cluster. We will build a new cluster, copy your data over and switch over to the new cluster for you. 2nd, Redshift is completely utility-based in its pricing – there are no licenses to buy, you pay for what you use. Similar to EMR, you can have persistent warehouses that are up 24X7 or have transient warehouses that you use for a short period of time and then bring down. 3rd it is fully managed, we do the backups, resizing, patching, etc. for you. You just provision it, load your data and start analyzing your data. Redshift is available with several instance types, ranging from under $1000/TB/year using DS2 nodes with mag storage to 10x the perf of other DW options using DC1 nodes with SSD storage.
- We just covered tools to analyze many TB or even PB of data. But we know sometimes you want to query and analyze at an even higher scale. Ingesting many PB into a Hadoop cluster or DW can quickly become impractical, so AWS recently launched 2 tools to help with this problem – Amazon Athena and Amazon Redshift Spectrum. They do similar things in different ways. Athena is a GP serverless query engine for S3. It uses Presto inside to do rel. queries against data in S3 in struct formats such as JSON, Parquet or Comma-separated values. Using Athena is straightforward. You create table definitions for data stored in S3, then use your favorite BI or analytics tool to query & analyze the data. Beyond storing the data, you only pay for the volume of data you scan during the query. Next, Spectrum is a new Svc that extends Redshift beyond the data stored in Redshift to query a data lake in S3. Unlike Athena, you provision a cluster of servers that Redshift uses to query & join data in S3 w/data stored w/in the Redshift Cluster. With Spectrum we were able to query an Exabyte of book sales data stored by Amazon.com to do analytics on projected sales of a given book title in under 3 minutes. So with these tech’s, you can scale to handle massive volumes of data without having to buy a DC full of HW.
- Now I’ll talk about AWS Glue, a new service that is still in preview. It is a serverless ETL Svc. It creates a Data Catalog across a metadata repository by crawling your data sources to understand formats & relationships To Author Jobs, glue generates Python code (which you can then edit) to move or Xform the data from source to destination Job Execution is done in Spark containers, and we ramp up the number of servers to meet your SLA. Because it is serverless, you just pay for what you use.
- And for reporting, Amazon Quicksight is a cloud-based biz analytics tool you can think of as the AWS UI into the data in the DB & analytic toolsets. Quicksight understands a wide variety of DB formats, whether they are in the cloud or on prem. It also understands formats such as Excel and CSV, and it can query Athena or even SaaS vendors such as Salesforce. It can query the data directly from the source or load data into SPICE – QuickSight’s in memory query engine. Literally within minutes you can connect to a datasource and start visualizing and analyzing data. QuickSight allows you to save queries and analysis into dashboards which can be shared and also has a mobile client. And that wraps up our DB and Analytics portfolio.
- Want to shift to a theme we hear a lot from DB cust: That is they like what they hear from us a lot more than what they are used to. They want to be free from the “old-world” policies – about economics, choice & the ability to embrace leading edge innovation, & biz practices. Old-world DB vendors are very expensive, proprietary, design for lock in, have punitive licensing terms. And too often send untimely email that you're being audited Many customers tell us they have just had enough of this.
- In 2015 we announced our DB migration Svc. Since we opened it up the beginning of this year, over 28K DB have migrated. DMS let you migrate to, from, or within the cloud safely & securely. And the source DB keeps running while the data is copied. It takes only minutes to set up, and it runs at very low cost. There are dozens of DB supported as sources & targets, including most widely-used commercial and Open Source DB, like MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, Redshift, DynamoDB, MongoDB, and others. When you want to change your DB from one engine to another, the Schema Conversion Tool, or SCT, creates an assessment report to guide you on your migration. Then SCT will read the metadata from your source and automatically convert to the right format for your target. SCT is free and also non-disruptive, so it makes it easy to determine how easy it will be to move your DB. As our customers examine their options for DB freedom, they can use our Workload Qualification Framework to look at each workload, assess the complexity and help determine which workloads are easiest to migrate, and which may need extra effort. Where WQF indicates higher complexity, AWS has programs that can help provide technology and expertise to make the migration easier and minimize risks. Now lets look at a customer example of DMS and SCT…..
- Located in the UK, Trimble is a global leader in geolocation svcs. Their gov’t, health care, construction, & other clients depend upon Trimble to help know where critical items are located when needed. Trimble’s Oracle solution was built over a decade ago. It was no longer flexible nor cost-effective for Trimble’s growing needs. Using SCT and DMS, Trimble migrated their key apps from Oracle to RDS for PostgreSQL. In 6 weeks, for £40,000, or about US$55,000. Immediate annual savings for Trimble of US$165,000, while allowing the app to grow to meet needs without requiring expensive licensing.
- Completing our Portfolio view, there are other sessions at this summit where we will be diving deeper into AI. I just want to make a few points. At Amazon, we’ve been using AI to better serve our cust for over 20 years. It’s a key part of our ops, from AWS to logistics, to new svcs like Alexa. We anticipate a few years from now AI will be bigger than the rest of AWS combined. Cloud computing, new low-cost high-speed processors, and new prog. frameworks combine to move AI from science fiction into everyday reality. The AWS approach is to support a range of Svcs for our customers’ differing needs. Starting at the bottom. Engines are for use by AI eng’rs, implementing new W/L. We’re contributing to the Apache MXNet project, and we also support a range of engines. Platforms are for Data Scientists and Data Analysts, allowing the application of algorithms to data without requiring deep AI expertise. On the top, Svcs let Devs to add facial recog, foto tagging, nat lang underst, text2voice, & other capab to apps w/o need’g any AI expertise. And that covers our DB, analytics, and AI Portfolio. Now I want to ask Scott to talk about FINRA’s use of this portfolio, followed by a Q&A session.
- The Financial Industry Regulatory Authority is an independent, non-governmental regulator for all securities firms during business with the public in United States. FINRA’s 3-D mission is investor protection and market integrity. We protect 90 million investors every single day! Everyday we work to: deter misconduct by enforcing the rules; detect and prevent wrongdoing in the U.S. markets; and discipline those who break the rules. We get billions of trades, quotes and order data daily. We conduct surveillance and monitor majority of the equities and options markets in the United States.
- The point to points were proving to be problematic in terms of Potential performance impacts on operational stores from heavy external query usage Tight coupling of SLAs If system A had an SLA of 8 – 8 and depended on System B which only had an SLA of 9 – 5 what do you do for planned maintenance So we moved to a Hub to decouple the systems
- Main Idea: Challenge to support migration from on premise to the cloud. Need a data migration/replication services. Built this on top of AWS Data Migration Services (DMS) to expedite application migrations Challenge: Move our RDBMS-backed applications in our data center to the cloud on RDS PostgreSQL Rely heavily on a data sharing hub in the center leveraging materialized views to share sets of data between applications We want to support applications executing on their migration independent of their upstream / downstream partner’s cloud migration schedules while minimizing rework Problems We can’t expect 100+ databases and applications to move to the cloud all at once at the same time. We need a way to try and provide a way for applications and databases to move without forcing their upstream/downstream dependencies to move with them or make coordinated changes. Solution: Automation / lights-out operations is first class consideration for FINRA So our implementation is exposed through an API that wraps the DMS API API covers DMS actions as well as non-DMS actions Allows our partners to include as part of their deployment and go through normal SDLC
- Business Challenges: Exchanges are dynamically evolving Regulatory landscape is changing Market manipulators innovating Normalizing data sets for common analytics and keep fidelity of original data Main Idea: AWS cloud has allowed us to separate infrastructure services from analytic services Standardized infrastructure services Focus greater investment on value add analytics rather than time consuming infrastructure items
- Where We Started: HBase – fast, interactive fetches based on keys Hive & Tez - Batch analytics for surveillance patterns and creation of data marts for interactive exploration Redshift – Data Marts & Interactive queries using power of MPP and Columnar access RDS – workflow transactions, preferences, history, etc. Where We Are Now & Where We are Going: EMR: Hive (decreasing) Presto (increasing) planning to move to Athena (service rather than own cluster) Spark (increasing) Batch analytics using Scala & Spark SQL Interactive analytics for fast access to cached data setts HBase (increasing) Moved from static clusters to highly resilient Hbase on S3 Redshift (maintain) Plan to move towards Spectrum as we are storage bound not compute RDS (increasing) Remainder of portfolio now moving to the cloud Data Science/Machine Learning (increasing) MetaStore (increasing) Plan to look at Glue as complementary or alternative to our own Metastore
- Main Idea: AWS analytic services allows us to flexibility to customize interactive analytics Use Cases: Interactive Summaries Market Reconstruction Audit Trail Data Science (e.g. regression analysis for purposeful sampling to hone firm examinations) AWS Analytics Tech Stack: EMR: Presto, Spark, Hive Redshift: data warehouse and data marts created from EMR queries (Hive & Presto) RDS: History, Preferences, etc. Moving towards use of Spectrum, Athena, Glue, QuickSite/SPICE, Notebooks
- R 3.2.5, Python (2.7.12 and 3.4.3) Packages R: 300+ Python: 100+ Tools for Building Packages gcc, gfortran, make, java, maven, ant… IDEs Jupyter, RStudio Server Deep Learning CUDA, CuDNN (if GPU present) Theano, Caffe, Torch TensorFlow
- Where we started: HBase – fast, interactive fetches based on keys on static cluster Hive – Batch analytics for surveillance patterns and creation of data marts for interactive exploration Redshift – Data Marts & Interactive queries using power of MPP and Columnar access RDS – workflow transactions, preferences, history, etc. How We Evolved EMR: HBase on S3 Hive & Tez (decreasing) Presto (increasing) Spark (increasing) Batch analytics using Scala & Spark SQL Redshift (maintain) Universal Data Catalog MetaStore (increasing) Data Science/Machine Learning (increasing) Now and Future Athena Use service instead of own cluster EMR: Spark (increasing) Batch analytics using Scala & Spark SQL Interactive analytics for fast access to cached data setts Spectrum Plan to move towards Spectrum as we are storage bound not compute RDS (increasing) Remainder of portfolio now moving to the cloud Data Science/Machine Learning (increasing) Glue Look as complement or alternative to Metastore QuickSite / BI More advanced interactive analytics with SPICE