Databases & Analytics AWS re:invent 2019 Recap

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Databases & Analytics
Sungmin Kim
Solutions Architect, AWS

Data
The world’s most
valuable resource is
no longer oil, but data.*
*Copyright: The Economist, 2017, David Parkins
“
”

Hardware and software installation
Configuration, patching, and backups
Cluster setup and data replication for high availability
Capacity planning, and scaling clusters for compute and storage
Managing software on-premises
is time consuming and complex

Customers moving to fully managed services
Relational databases
Aurora RDS EMR
Hadoop
and Spark
Elasticsearch
Service
Operational
analytics
Managed
Streaming
for Kafka
Real-time
analytics
DynamoDB DocumentDB ElastiCache
Managed
Cassandra
Service
Non-relational databases

AWS Databases – for all your application use cases

Amazon RDS
Managed relational database service with a choice of popular databases
Easy to administer
No infrastructure provisioning
No software installation and
patching
Built-in monitoring
Performant & scalable
Scale with an API call or a few
clicks
Read replicas for increased
throughput
Automatic Multi-AZ
data replication
Automated backup,
snapshots, and failover
Available & durable Secure and compliant
Encryption at rest and in transit
Network isolation and
resource-level permissions

How do you scale your relational database to support
tens of thousands of connections?
Serverless applications
open and close tens of
thousands of connections
within seconds
Leads to longer query
response times that limits
application scalability
A database proxy server
are difficult to deploy,
patch, and manage

Amazon RDS를 위한 완전 관리형 고-가용성 데이터베이스 프록시 기능. 커넥션 풀 및
공유 연결을 통해 애플리케이션 확장성, 데이터베이스 장애에 대한 탄력성 및 보안 향상
DRAFTDatabases
Preview – December 3
L E A R N M O R E DAT368: Setting up database proxy servers with RDS Proxy
Amazon RDS Proxy
RDS
Proxy
Applications
RDS
Database Instance
Connection Pooling

Amazon Aurora
MySQL and PostgreSQL compatible relational database built for the cloud
Performance and availability of commercial-grade databases at 1/10th the cost
Performance
and scalability
5x throughput of MySQL
3x throughput of PostgreSQL
Up to15 read replicas
Scale out reads and writes
across multiple data centers
Fully managed
Managed by RDS:
no hardware provisioning,
software patching, setup,
configuration, or backups
Availability
and durability
Fault-tolerant self-healing storage
Six copies of data across three AZs
Continuous backup to S3
Single Global database with cross-
region replication
Network isolation
Encryption at rest/transit
Highly secure

Generate predictions directly from Aurora queries
Models run in SageMaker & Comprehend
Use standard SQL, no ML expertise required
Suitable for low-latency, high-volume use cases
Amazon
SageMaker
ML
Aurora
Database
Athena
Interactive
analytics
SQL
Select
from
Where
ML in Amazon Aurora and Athena
Bringing machine learning to data developers and data analysts
L E A R N M O R E DAT369 - Deep dive on Amazon Aurora machine learning integration
General Availability – November 26
Databases

Amazon Managed Apache Cassandra Service
높은 가용성과 확장성을 제공하는 서버리스 Apache Cassandra 호환 데이터베이스
완전 관리 서비스
Apache Cassandra
호환성 제공
성능 및
확장성 보장
높은 가용성 및
보안 제공
관리할
서버가 없음
DRAFTDatabases
L E A R N M O R E DAT324: Overview of Amazon Managed Apache Cassandra Service

Common data categories and use cases

Amazon
Athena
Interactive
Query
Any type of analytics on the data lake
Most comprehensive analytics platform
Amazon S3 | AWS Glue
Lake Formation
Data lake
Amazon
Elasticsearch
Service
Amazon
Kinesis
Amazon
MSK
Amazon
SageMaker
Amazon
Personalize
Amazon
QuickSight
AWS Data
Exchange
Operational
Analytics
Real time
Analytics
Predictive
Analytics
RecommendationsVisualizations
Data
Exchange
Amazon
Redshift
Data
Warehousing
Amazon
EMR
Big Data
Processing

Amazon
Athena
Interactive
Query
Any type of analytics on the data lake
Amazon S3 | AWS Glue
Lake Formation
Data lake
Amazon
Elasticsearch
Service
Amazon
Kinesis
Amazon
MSK
Amazon
SageMaker
Amazon
Personalize
Amazon
QuickSight
AWS Data
Exchange
Operational
Analytics
Real time
Analytics
Predictive
Analytics
RecommendationsVisualizations
Data
Exchange
Amazon
Redshift
Data
Warehousing
Amazon
EMR
Big Data
Processing

Amazon EMR
Easily Run Spark, Hadoop, Hive, Presto, HBase, and more big data apps on AWS
Low cost
50–80% reduction in costs with
EC2 Spot and Reserved Instances
Per-second billing for flexibility
Use S3 storage
Process data in S3
securely with high performance
using the EMRFS connector
Latest versions
Updated with latest open source
frameworks within 30 days
Fully managed no cluster
setup, node provisioning,
cluster tuning
Easy

Performance Improvements in Spark for Amazon EMR
Performance optimized runtime for Apache Spark, 2.6x faster performance at 1/10th the cost
*Based on TPC-DS 3 TB Benchmarking running 6 node
C4x8 extra large clusters and EMR 5.28, Spark 2.4
10,164
16,478
26,478
0 5,000 10,000 15,000 20,000 25,000 30,000
Spark with EMR (with runtime)
3rd party Managed Spark (with their
runtime)
Spark with EMR (without runtime)
Runtime total on 104 queries
(seconds—lower is better)
Runtime optimized for Apache Spark performance
100% compliant with Apache Spark APIs
Best performance
2.6x faster than Spark with EMR without runtime
1.6x faster than 3rd party Managed Spark (with their runtime)
Lowest price
1/10th the cost of 3rd party Managed Spark (with their runtime)
NEW

Amazon Athena
Pay per query
Pay only for queries run
Save 30–90% on per-query costs
through compression
Use S3 storage
ANSI SQL
JDBC/ODBC drivers
Multiple formats,
compression types, and
complex joins and data types
SQL
Serverless: zero infrastructure,
zero administration
Integrated with QuickSight
EasyQuery instantly
Zero setup cost
Point to S3 and start querying
Serverless, interactive query service

Preview – November 26Amazon Athena Federated Query
Run SQL queries on data spanning multiple data stores
Redshift
Data warehousing
ElastiCache
Redis
Aurora
MySQL, PostgreSQL
DynamoDB
Key value, Document
DocumentDB
Document
S3/Glacier
Run connectors in AWS Lambda: no servers to manage
Run SQL queries on relational, non-relational, object,
or custom data sources; in the cloud or on-premises
Open Source Connectors for common data sources
Build connectors to custom data sources
Analytics

Amazon Redshift
Best performance,
most scalable
3x faster with RA3*
10x faster with AQUA*
Adds unlimited compute capacity
on-demand to meet unlimited
concurrent access
Lowest cost
Cost-optimized workloads
by paying compute and
storage separately
1/10th cost of Traditional
DW at $1000/TB/year
Up to 75% less than other
cloud data warehouses &
predictable costs
Data lake &
AWS integration
Analyze exabytes of data across
data warehouse, data lakes, and
operational database
Query data across various
analytics services
Most secure
& compliant
AWS-grade security (eg. VPC,
encryption with KMS, CloudTrail)
All major certifications such
as SOC, PCI, DSS, ISO,
FedRAMP, HIPPA
First and most popular cloud data warehouse
*vs other cloud DWs

컴퓨팅 및 스토리지 비용을 별도로 지불하여 데이터 웨어하우스 비용 최적화
General Availability – December 3
L E A R N M O R E
ANT213-R1: State of the Art Cloud Data Warehousing
ANT230: Amazon Redshift Reimagined: RA3 and AQUA
기존 클라우드 DW의 3 배 성능 제공
비슷한 가격의 DS2 인스턴스로서 2 배의
성능과 2 배의 스토리지 (주문형)
DW 저장 용량을 자동으로 확장 및 최대
8PB의 워크로드 지원 (압축)
COMPUTE NODE
(RA3/i3en)
SSD Cache
S3 STORAGE
COMPUTE NODE
(RA3/i3en)
SSD Cache
COMPUTE NODE
(RA3/i3en)
SSD Cache
COMPUTE NODE
(RA3/i3en)
SSD Cache
Managed storage
$/node/hour
$/TB/month
Amazon Redshift RA3 instances
SEOUL
Analytics

DRAFTAnalytics
L E A R N M O R E ANT230: Amazon Redshift Reimagined: RA3 and AQUA
AQUA (Advanced Query Accelerator) for Amazon Redshift
새로운 하드웨어 기반 가속 캐시를 통해 비용 증가 없이 타 클라우드 데이터 웨어
하우스보다 10 배 빠르게 질의 실행

DRAFTAnalytics
AQUA는 컴퓨팅을 스토리지를 기반으로 하므로
데이터 이동 필요 없음
S3 기반 고속 캐시는 여러 노드에서 동시에
데이터를 처리하도록 수평 확장
AWS 설계 프로세서로 데이터 압축, 암호화 및
데이터 처리 가속화
Redshift 버전과 100 % 호환
AQUA (Advanced Query Accelerator) for Amazon Redshift

Speed-up queries by orders of magnitude
• Joins, filters, aggregations, and projections
Analytics
Preview – November 27
Simplify and accelerate ETL/BI pipelines
• Incremental refresh
• User triggered maintenance
Easier and faster migration to Amazon Redshift
Amazon Redshift Materialized Views

Amazon Redshift Federated Query
DRAFTAnalytics
Amazon RDS 및 Amazon Aurora 데이터베이스와 Amazon Redshift 데이터 웨어
하우스 및 Amazon S3 테이블 데이터를 결합할 수 있는 통합 SQL 질의 기능
SEOUL

데이터 웨어하우스 기반 데이터를 데이터 레이크로 내보내 다양한 분석 기능을
수행하여 새로운 통찰력 입수 가능
General Availability – December 3
L E A R N M O R E ANT335-R: How to build your data analytics stack at scale with Amazon Redshift
Amazon Redshift Data Lake Export SEOUL
Analytics

Amazon Elasticsearch Service
Fully managed, scalable, secure, Elasticsearch service
Open source Elasticsearch
APIs, Kibana, and
Logstash
Open-source Elasticsearch APIs
Managed Kibana
Integration with Logstash
Scale clusters up/down via a
single API call or a few clicks
Secured network isolation
with VPC, encrypt data
at-rest and in-transit
Compliant: HIPPA, PCI DSS,
and ISO
Scalable, secure,
and compliant
Pay only for
what you use
Cost-optimized workloads
No upfront fee or
usage requirement
Critical features built-in:
encryption, VPC support,
24x7 monitoring
Fully managed
Deploy Elasticsearch clusters
in minutes: simplified hardware
provisioning, software
installation/patching, failure
recovery, backups, and monitoring

Challenges with analyzing high volumes of data in real-time
Storing data is
expensive at scale
Limits the amount of
data retained for analysis
Miss out on
valuable insights

Analytics
Amazon Elasticsearch Service를 위한 저비용의 확장 가능한 웜 스토리지 계층.
UltraWarm for Amazon Elasticsearch Service
L E A R N M O R E
ANT229 - Introducing UltraWarm for Amazon Elasticsearch Service
ANT342 - Scaling Amazon Elasticsearch Service with UltraWarm

Our portfolio
Broad and deep portfolio, purpose-built for builders
S3/Glacier
Glue
ETL & Data Catalog
Lake Formation
Data Lakes
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka
Data Movement
Data Lake
Business Intelligence & Machine Learning
Data Exchange
Data exchange
NEW
QuickSight
Visualizations
SageMaker
ML
Comprehend
NLP
Transcribe
Speech-to-text
Textract
Extract text
Personalize
Recommendation
Forecast
Forecasts
Translate
Translation
CodeGuru
Code reviews
Kendra
Enterprise search
NEW NEW
RDS
MySQL, PostgreSQL,
MariaDB, Oracle, SQL Server,
RDS on VMware
Aurora
MySQL, PostgreSQL
DynamoDB
Key value, Document
ElastiCache
Redis, Memcached
Neptune
Graph
Timestream
Time Series
QLDB
Ledger Database
Analytics Databases
Managed
Blockchain
Blockchain
Templates
Blockchain
Managed Apache
Cassandra Service
Wide column
NEW
DocumentDB
Document
Redshift
Data warehousing
EMR
Hadoop + Spark
Kinesis Data Analytics
Real time
Elasticsearch Service
Operational Analytics
Athena
Interactive analytics
NEW
NEW
NEW
NEW
NEW
AQUA EMR on Outposts
UltraWarm
RDS Proxy
RDS on Outposts

2019년 출시 서비스 한눈에 보기
ANALYTICS COMPUTE DATABASE MANAGEMENT &GOVERNANCE
Amazon EMR on AWS Outposts
Deploy Apache Hadoop & Spark Clusters
in AWS Outposts.
Announcement
Amazon Redshift Data Lake Export
Unload Amazon Redshift query result to
Amazon S3 data lake as Apache
Parquet. Announcement
Amazon Redshift Federated Query
(Preview) Query, analyze data across
operational databases, data warehouses,
and data lakes. Announcement|
Documentation
RA3 with managed storage
Scale & pay for compute and storage
independently to cluster size based
on compute needs.
Announcement
Advanced Query Accelerator (AQUA)
Enables Redshift to run up to 10x faster.
Amazon UltraWarm for
Amazon Elasticsearch (Preview)
Store and analyze data using
Elasticsearch and Kibana while reducing
your cost.
Announcement |Documentation
Kinesis Video Streams RTC
Stream live media with ultra-low
latency and enable two-way
interactivity for millions of camera
devices.
Announcement
Amazon EC2 Inf1 Instances
High performance and the lowest cost
machine learning inference in the cloud.
Announcement |ProductPage
Amazon EC2 M6g, C6g, and R6g
Instances powered by AWS Graviton2
Processors Deliver up to 40%
price/performance improvement over
comparable x86-based instances for a
broad spectrum of workloads.
Amazon EC2 Image Builder
EC2 Image Builder simplifies creation,
maintenance, validation, sharing, and
deployment of Linux or Windows Server
images for use with Amazon EC2 and on-
premises. Announcement|Product Page
Amazon EKS for AWS Fargate
Amazon Elastic Kubernetes Service (EKS)
can now run Kubernetes pods on AWS
Fargate, the serverless compute engine
built for containers on AWS.
Announcement |Blog
AWS Wavelength
Build applications with single-digit
millisecond latencies delivery to mobile
devices on 5G. Announcement |Product
Page
AWS Lambda Provisioned Concurrency
Keep functions initialized and hyper-ready
to respond in milliseconds.
Announcement |Documentation
HTTP APIs for Amazon API
Gateway (Preview)
Make it easy to create, publish, maintain,
monitor, and secure APIs at scale.
Announcement |Blog
AWS Compute Optimizer
Uses ML to recommend optimal resources
for customer workloads to reduce costs, and
improve performance.
Announcement |Blog
AWS Fargate Spot
A new option to run fault-tolerant
applications on AWS Fargate with up to
70% discount compared to Fargate
prices. Announcement
AWS Local Zones
A new type of AWS infrastructure
deployment that places compute, storage,
database and other services closer to
customers.
Announcement
AWSNitro Enclaves
AWS Nitro Enclaves enables customers to
create isolated compute environments to
further protect and securely process highly
sensitive data. E.g. PII, health, finance.
Product Page
AWSOutposts
AWS Outposts GA -- a new fully
managed service that extends AWS
infrastructure, services, APIs, and tools to
on-premises environments.
Announcement|Blog
Amazon Managed Apache Cassandra
Service (Preview)
Scalable, highly available, managed
Apache Cassandra-compatible database
service.
Amazon RDS on Outposts (Preview)
Deploy managed databases in on-
premises AWS Outposts environments.
Announcement |FeaturePage
Amazon RDSProxy
Fully managed, highly available
database proxy for Amazon Relational
Database Service.
Simplified Windows and SQL Server
BYOL
New BYOL experience for software licenses,
such as Windows and SQL Server that
require dedicated physical servers.
Announcement |Product Page |
Documentation
MIGRATION &TRANSFER
End of Support Migration Program
(EMP) for WindowsServer
Helps customers package legacy
applications allowing them to run on
newer windows server OS without any
code changes.
INTERNET OF THINGS
AWS IoT SiteWise (Preview)
IoT SiteWise new features and pricing,
additional capabilities to collect data, add
context to the data, use the data in their
custom applications, and view the data.
QUANTUM TECHNOLOGIES
Amazon Braket(Preview)
Build, test, and run quantum computing
applications.
Product Page |Blog|Announcement

MACHINE LEARNING NETWORKING STORAGE
Amazon Augmented AI (Amazon A2I)
Makes it easy to build workflows
required for human review of ML
predictions. Announcement |Product
Page |Documentation |
Amazon CodeGuru(Preview)
Machine learning service for automated
code review and application performance
profiling. Announcement |Product Page |
Amazon Fraud Detector (Preview)
Easily identify potentially fraudulent
online activities such as online payment
fraud and the creation of fake accounts.
|Announcement |Product Page
Amazon Kendra(Preview)
Transform the way you search and interact
with enterprise data using AI.
|Documentation
Amazon Transcribe Medical
Quickly create accurate transcriptions from
medical consultations between patients
and physicians. Announcement|Product
Page |Documentation
Amazon Rekognition Custom Labels
Train custom models to identify objects
and scenes in images that are specific to
your business needs.
AWS DeepComposer
Gives developers of all skill levels a
creative way to experience machine
learning with a musical keyboard.
Announcement|Product Page
AWS DeepRacer Evo
Multi-car racing and object avoidance
capabilities.
Announcement |Product Page |
Amazon SageMaker Studio
Provides a single, web-based visual
interface to perform all ML development
steps. Announcement
Amazon SageMakerNotebooks
One-click Jupyter notebooks that you can
start working within seconds.
Announcement
Amazon SageMaker Experiments
Help developers manage and track
iterations for machine learning models by
automatically storing input parameters,
configuration, and results as experiments.
Announcement
Amazon SageMakerDebugger
Debug and profile model training to
improve accuracy, reduce training times,
and facilitate a greater understanding of
machine learningmodels.
Announcement
Amazon SageMakerAutopilot
The industry’s first automated machine
learning capability that gives you
complete control and visibility into your
ML models. Announcement
Amazon SageMaker Model Monitor
Automatically detect concept drift in
deployed models and identify the source
of the problem.
Announcement
SageMaker Operators for Kubernetes
Make it easier to use Kubernetes to train,
tune, and deploy machine learning (ML)
models in Amazon SageMaker.
Announcement |Documentation |GitHub|
Contact Lens for Amazon Connect
Customer centers use of ML to better
understand sentiment, trends, and
compliance risks of customer
conversations.
Announcement
AWS Machine Learning Embark
program Combines training, coaching,
and implementation support to launch
the company’s machine learning
journey.
Announcement
Accelerated Site-to-Site VPN
Improve the performance of VPN
connections by intelligently routing traffic
through the AWS Global Network and
AWS edge locations.Announcement
Amazon VPC Ingress Routing
Make it Easy to insert Virtual Appliances in
the Forwarding Path of VPC Traffic.
Announcement |Blog
AWS Transit Gateway Network
Manager
Centrally manage and monitor your
global network across AWS and on
premises.
Announcement
SERVERLESS
Amazon EventBridge with Schema
Registry
Stores event structure - or schema - in a
shared central location and maps those
schema to code. Announcement |
Blog |Documentation
AWS Step Functions Express
Workflows
Workflow type capable of orchestrating
AWS compute, database, and messaging
services at event rates greater than 100,000
state transitions per second.
Announcement |Product Page|Blog
Access Analyzer for AmazonS3
Ensure access policies provide only intended
access to S3 resources.
Announcement |Blog |Documentation
Amazon S3 Access Points
Simplify and manage access at scale for
applications using shared data sets on
S3. Announcement |Feature Page |Blog |
SECURITY, IDENTITY,&
COMPLIANCE
Amazon Detective(Preview)
Easily analyze, investigate, and quickly
identify the root cause of potential security
issues or suspiciousactivities.
AWS Identity and Access Management
(IAM) Access Analyzer
Generates comprehensive findings that
identify resources that can be accessed from
outside an AWS account.
Blog |Announcement |FeaturePage
DEVELOPER TOOLS
Amazon Builders’Library
Articles that detail how Amazon builds
and operates the technologies that
power Amazon.com andAWS.
Builders Library
2019년 출시 서비스 한눈에 보기

Databases & Analytics AWS re:invent 2019 Recap

Databases & Analytics AWS re:invent 2019 Recap

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Databases & Analytics AWS re:invent 2019 Recap

Similar to Databases & Analytics AWS re:invent 2019 Recap (20)

More from Sungmin Kim

More from Sungmin Kim (14)

Recently uploaded

Recently uploaded (20)

Databases & Analytics AWS re:invent 2019 Recap