Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliat...
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
서버리스란?
• 서버를 생성하거나 관리할 필요가 없음
• 요청에 맞추어 확장 가능
• ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS 서버리스 포트폴리오
애플리케이션 빌딩 블록 – 컴퓨팅 및 저장소 서비스
AWS
...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda
이벤트 소스 AWS Lambda Function 다양한 종류의 서비...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HTTP 요청을 받아 AWS Lambda의 이벤트 소스로 사용하도록 합니다.
또한 We...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB (w/Global Tables)
Amazon DynamoDB를 이용하여...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
var AWSXRay = require(‘aws-xray-sdk-core‘);
AWSX...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
X-Ray Trace 예제
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Other AWS services
Amazon Cognito – 페더레이션 및 인증
A...
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
이런 분들에게 추천합니다
웹 서비스를 빠르게 개발, 배포하고 싶어요.
조직 내에 인프라...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
멀리 리전 웹 앱 아키텍처 - Wild Rydes
Users
Amazon
Route 5...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda Layers
함수들 간 코드를 공유할 수 있습니다:
레이어를 업로드 한 후...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda Runtime API
리눅스 호환 Language 런타임을 Lambda에서...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
고려 사항
코드
Q: 저희는 지역적 이슈가
발생하더라도 코드에 접근하여
업데이트 또는 ...
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
이런 분들에게 추천합니다
반복되는 작업을 자동화하고 싶어요.
특정 이벤트에 대해 워크플...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
오퍼레이션 자동화 패턴 - AWS Ops Automator
https://aws.ama...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
더 복잡한 워크플로를 처리하고 싶다면?
Web App
Amazon DynamoDB:
I...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
구축, 실행이 쉽고 안전한 완전 관리되는 자동화된 워크플로
분석 및 ML배치 작업마이크...
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
이런 분들에게 추천합니다
운영중인 서비스의 모든 데이터를 수집, 저장, 처리하고 싶어요...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Catalog & Search Access & User Interfaces
Analyt...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon S3
Bucket(s)
Crawlers Data catalog
AWS Gl...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Athena – 서버리스 대화식 쿼리 서비스
44.66 seconds...Data sc...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
NEW! AWS Lake Formation (Preview)
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
이런 분들에게 추천합니다
기기의 수가 많고 데이터 업로드가 잦아요.
준 실시간성 처리가...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
실시간 IoT Device Monitoring 패턴
https://aws.amazon....
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
고려 사항
• Firehose의 버퍼 크기와 버퍼 인터벌 조정
• 더 큰 objects...
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Minseok Han
DevOps Engineer | SendBi...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is SendBird?
• Chat API/SDK for Mobile Apps...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key milestone
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Log pipelines at SendBird
• Source data for DAU,...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipelines for searching
• Managed Elasticsearch ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipelines for searching
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Kinesis Agent
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Kinesis Agent
Pros
• Easy Installation and C...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Kinesis Agent
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Kinesis Agent
Cons
• Possibility for Duplica...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fluentd (rotation issues resolved!)
• File rotat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fluentd - aggregation
• Kinesis Data Stream and ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fluentd - aggregation
• Aggregate record up to 2...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Accessibility
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Accessibility
• ElasticSearch Service Domain has...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Accessibility
• Amazon Cognito: user sign-up, si...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch index rotation
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch index rotation
• Data pipeline wor...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch index rotation
• Firehose support ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch index rotation
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch index rotation
• Index can be rota...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring
• Kinesis Stream getRecord counts
• K...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary
자주 사용되는 서비스들을 쉽고 빠르게 개발하기 위해 아래와 같은 서버리스...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Next Steps
• 관련 Summit Seoul 2019 세션:
1. AWS Lam...
Upcoming SlideShare
Loading in …5
×

서버리스 아키텍처 패턴 및 로그 처리를 위한 파이프라인 구축기 - 황윤상 솔루션즈 아키텍트, AWS / Matthew Han, SendBird :: AWS Summit Seoul 2019

1,392 views

Published on

서버리스 아키텍처 패턴 및 로그 처리를 위한 파이프라인 구축기
황윤상 솔루션즈 아키텍트, AWS
Matthew Han, SendBird

AWS에서 제공하는 다양한 서비스는 서버리스 아키텍처에서 활용 가능합니다. 본 세션에서는 다양한 시나리오를 위한 서버리스 아키텍처를 소개하고, 각 아키텍처 별 주의해야할 점 및 모범 사례를 살펴봅니다. 특히, 서버리스 아키텍처로 로그 처리 파이프라인을 구축한 SendBird의 사례를 살펴보고, 파이프라인 구축에 있어서 얻게된 팁들을 공유합니다.

Published in: Technology
  • Login to see the comments

서버리스 아키텍처 패턴 및 로그 처리를 위한 파이프라인 구축기 - 황윤상 솔루션즈 아키텍트, AWS / Matthew Han, SendBird :: AWS Summit Seoul 2019

  1. 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 서버리스 아키텍처 패턴 및 로그 처리를 위한 파이프라인 구축기 Yunsang Hwang Solutions Architect | AWS Korea Minseok Han DevOps Engineer | SendBird
  2. 2. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  3. 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 서버리스란? • 서버를 생성하거나 관리할 필요가 없음 • 요청에 맞추어 확장 가능 • 사용한 만큼만 비용 지불 • 고가용성(HA) 및 재해 복구(DR) 내장
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS 서버리스 포트폴리오 애플리케이션 빌딩 블록 – 컴퓨팅 및 저장소 서비스 AWS Lambda AWS Fargate Amazon API Gateway Amazon SNS Amazon MQ Amazon SQS AWS Step Functions 애플리케이션 통합 개발자 도구 보안 및 운영 관리 Amazon Aurora Serverless Amazon S3 Amazon DynamoDB AWS AppSync AWS IAM Amazon Cognito Amazon Inspector Amazon VPC Amazon GuardDuty AWS CloudFormation AWS Cloud9 AWS CloudTrail Amazon CloudWatch AWS X-Ray AWS CodePipeline AWS Config AWS SSO AWS Shield AWS WAF Amazon Kinesis AWS Serverless Application Repository Athena
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Lambda 이벤트 소스 AWS Lambda Function 다양한 종류의 서비스 Node.js Python Java C# Go Ruby Runtime API 데이터 상태 변경 엔드포인트 요청 리소스 상태 변경
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. HTTP 요청을 받아 AWS Lambda의 이벤트 소스로 사용하도록 합니다. 또한 WebSocket을 지원하여, Stateful 한 요청을 Stateless한 백엔드 서비스로 라우팅할 수 있습니다. • 자동적 확장 • 인증 지원 • API 요청 제한 • SSL 적용 Amazon API Gateway
  8. 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB (w/Global Tables) Amazon DynamoDB를 이용하여 서버리스 애플리케이션 데이터를 쉽게 저장할 수 있습니다. • 99.999%의 SLA가 적용되는 NoSQL 데이터베이스 • 멀티 리전 및 멀티 마스터 테이블 지원 • 대규모 글로벌 애플리케이션을 지원 가능한 빠른 읽기 / 쓰기 성능 • 콘솔에서의 클릭 몇 번으로 쉽게 생성하는 글로벌 테이블
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. var AWSXRay = require(‘aws-xray-sdk-core‘); AWSXRay.middleware.setSamplingRules(‘sampling-rules.json’); var AWS = AWSXRay.captureAWS(require(‘aws-sdk’)); S3Client = AWS.S3(); AWS X-Ray Integration with Serverless • Lambda에서 지원하는 모든 언어에 대해 incoming request 처리 • 이 때 Lambda의 SDK를 사용하여 X-Ray daemon을 실행 가능
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray Trace 예제
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Other AWS services Amazon Cognito – 페더레이션 및 인증 Amazon Route 53 – 메트릭 기반 DNS 라우팅 및 헬스 체크 Amazon CloudFront – 컨텐트 배포 및 가속 AWS CloudFormation + SAM – 구성 관리 Amazon S3 buckets – 정적 웹 사이트 호스팅 AWS Identity and Access Management (IAM) – AWS 리소스 권한 관리 Amazon CloudWatch – 모니터링
  12. 12. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 이런 분들에게 추천합니다 웹 서비스를 빠르게 개발, 배포하고 싶어요. 조직 내에 인프라 담당자가 없어요. 서비스 수요에 따라 빠르게 확장 가능했으면 좋겠어요. 추후 글로벌 확장도 고려하고 있어요.
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 멀리 리전 웹 앱 아키텍처 - Wild Rydes Users Amazon Route 53 Primary Region Secondary Region https://aws.amazon.com/serverless-workshops/ Amazon API Gateway (Regional endpoint) Amazon API Gateway (Regional endpoint) Amazon Cognito Amazon S3 Static Website Hosting Amazon DynamoDB AWS Lambda Amazon DynamoDB AWS Lambda
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda Layers 함수들 간 코드를 공유할 수 있습니다: 레이어를 업로드 한 후, 다른 함수들에서 업로드한 레이어를 참조할 수 있습니다. 비즈니스 로직과 레퍼런스를 나눌 수 있고, 개발자들로 하여금 비즈니스 로직에 집중하여 더 빠른 배포를 가능하게 합니다. 또한 레이어는 보안이 유지된 채로 안전하게 공유됩니다.
  16. 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda Runtime API 리눅스 호환 Language 런타임을 Lambda에서 사용 가능합니다. 새로운 Runtime API로 런타임 실행 규약과 통합 지점을 코드화 할 수 있습니다. 기능 발표와 함께 Ruby를 지원하는 커스텀 런타임이 AWS Lambda에 내장되었으며, 파트너들을 통해 더 많은 런타임들이 지원될 예정입니다. (Erlang 등) 커스텀 런타임은 “레이어”로 배포됩니다. Rule Stack
  17. 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 고려 사항 코드 Q: 저희는 지역적 이슈가 발생하더라도 코드에 접근하여 업데이트 또는 배포를 수행할 수 있어야 합니다. A: AWS CodeCommit을 이용하여 코드를 타 리전으로 백업 및 복제할 수 있습니다. 데이터 Q: 저희는 서비스 운영에 차질이 없도록, 보조 리전에서도 데이터 접근이 가능해야 합니다. A: DynamoDB 전역 테이블을 사용하면 지연 시간을 최소화하면서 데이터를 보조 리전으로 쉽게 동기화할 수 있습니다. 템플릿 Q: 서비스가 장애 조치되는 동안 앱을 신속하게 배포하거나 업데이트해야 할 수도 있습니다. A: CloudFormation 또는 SAM 템플릿을 Amazon S3에 저장하고 버킷에서 영역 간 복제를 활성화하여 조치할 수 있습니다.
  18. 18. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 이런 분들에게 추천합니다 반복되는 작업을 자동화하고 싶어요. 특정 이벤트에 대해 워크플로를 수행하도록 하고 싶어요. 인프라 정책을 강제하고 잘 적용되고 있는지 보고 싶어요. 알림이 올 때 특정 조건에 맞게 처리하고 싶어요.
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 오퍼레이션 자동화 패턴 - AWS Ops Automator https://aws.amazon.com/answers/infrastructure-management/ops-automator/ AWS Lambda (AWS Ops Automator microservices) Amazon Simple Notification Service Amazon CloudWatch Amazon DynamoDB (Task configuration & Tracking) AWS CloudFormation Templates (Task Configuration) Task Configuration Stacks Cross-account Roles Amazon S3 Bucket Secondary Account
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 더 복잡한 워크플로를 처리하고 싶다면? Web App Amazon DynamoDB: Image meta-data & tags Amazon Cognito: User authentication Amazon S3: Image uploads AWS Step Functions: Workflow orchestration Start state machine execution 1 Extract image meta-data 2 Amazon Rekognition: Object detection Invoke Amazon Rekognition Generate image thumbnail 3 3Store meta-data and tags 4 https://github.com/awslabs/lambda-refarch-imagerecognition
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 구축, 실행이 쉽고 안전한 완전 관리되는 자동화된 워크플로 분석 및 ML배치 작업마이크로서비스 AWS Step Functions 새로운 서비스 통합
  23. 23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 이런 분들에게 추천합니다 운영중인 서비스의 모든 데이터를 수집, 저장, 처리하고 싶어요. 가지고 있는 데이터의 종류가 다양해요. 쉽고 빠르게 분석해서 BI 로 보고 싶어요. AI/ML로 활용하고 싶어요. ETL 모듈을 따로 만들기 힘들어요.
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Catalog & Search Access & User Interfaces Analytics & Serving S3 Amazon DynamoDB Amazon Elasticsearch Service AWS AppSync Amazon API Gateway Amazon Cognito AWS KMS AWS CloudTrail Manage & Secure AWS IAM Amazon CloudWatch Data Ingestion AWS Snowball AWS Storage Gateway Amazon Kinesis Data Firehose AWS Direct Connect AWS Database Migration Service Amazon Athena Amazon EMR AWS Glue Amazon Redshift Amazon DynamoDB Amazon QuickSight Amazon Kinesis Amazon Elasticsearch Service Amazon Neptune Amazon RDS Central Storage Scalable, secure, cost- effective AWS Glue AWS의 데이터 레이크
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon S3 Bucket(s) Crawlers Data catalog AWS Glue Amazon Athena Amazon Redshift Spectrum Amazon QuickSight 데이터 레이크 패턴
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Athena – 서버리스 대화식 쿼리 서비스 44.66 seconds...Data scanned: 169.53GB 비용: $5/TB or $0.005/GB = $0.85 SELECT gram, year, sum(count) FROM ngram WHERE gram = 'just say no' GROUP BY gram,year ORDER BY year ASC;
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. NEW! AWS Lake Formation (Preview)
  29. 29. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 이런 분들에게 추천합니다 기기의 수가 많고 데이터 업로드가 잦아요. 준 실시간성 처리가 필요해요. 기기가 간헐적으로 네트워크에 접속해서, 갑작스레 많은 요청이 들어와요. 메시지의 순서와 보존이 중요해요.
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 실시간 IoT Device Monitoring 패턴 https://aws.amazon.com/solutions/real-time-iot-device-monitoring-with-kinesis/ AWS IoT Core IoT rule Amazon Kinesis Data Firehose (events) Amazon Kinesis Data Analytics Amazon Kinesis Data Streams (events) AWS Lambda function Amazon S3 Bucket (events) Amazon DynamoDB Table Real-time Dashboard Users Amazon CloudFront Amazon Kinesis Data Firehose (Processed) Amazon S3 Bucket (Processed) Amazon Cognito IoT Devices
  32. 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 고려 사항 • Firehose의 버퍼 크기와 버퍼 인터벌 조정 • 더 큰 objects = 더 적은 Lambda 호출 및 S3 PUT 요청 • 스토리지 비용 절감을 위해 압축 사용 • 데이터 변경을 수행하는 경우 Source Record Backup 사용 • 데이터 변경 에러 발생 시 복구 가능 • Amazon Redshift Best Practices for Loading Data 참고
  33. 33. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Minseok Han DevOps Engineer | SendBird
  34. 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is SendBird? • Chat API/SDK for Mobile Apps, Websites and Games • 10,000+ SDKs installed from 150+ Countries • 1on1 Private Chat, Group Chat, to Large public Chat • Send Text, Image, Video, Push Notifications • Mentions, Read Receipts, Typing Indicators • Moderation, Chatbot Support, Online Presence..More..
  35. 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Key milestone
  36. 36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  37. 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Log pipelines at SendBird • Source data for DAU, MAU calculation • Load testing analytics • Rule generator for AWS WAF
  38. 38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  39. 39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pipelines for searching • Managed Elasticsearch clusters • Well integration with Amazon Kinesis Firehose • Configurable access policy • Cognito authentication • Available for Inplace update • ...Amazon Elasticsearch Service
  40. 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pipelines for searching
  41. 41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  42. 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Kinesis Agent
  43. 43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Kinesis Agent Pros • Easy Installation and Configuration • Support Key Value pair type metadata • Support EC2 metadata (It`s not documented yet. You can have PR!)
  44. 44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Kinesis Agent
  45. 45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Kinesis Agent Cons • Possibility for Duplicated data (high traffic) • Abnormality with logrotate • Known issues but not patched
  46. 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fluentd (rotation issues resolved!) • File rotation detection • Rotate_wait option: preventing the old file from getting lost • More detail configuration • 500+ plugins • Support Kinesis output plugin
  47. 47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fluentd - aggregation • Kinesis Data Stream and Firehose are awesome but the cost is a burden. • Kinesis Data Stream Pricing have two metrics: 1) Shard Hour 2) PUT Payload Units • Fluentd kinesis plugin supports aggregation
  48. 48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fluentd - aggregation • Aggregate record up to 25KB per single put unit • Kinesis Data Firehose de-aggregates automatically the records before it delivers them to AWS Lambda • The smaller the data, the greater the cost savings: 1) The number of shard can be diminished 2) The number of Put unit can be decreased
  49. 49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  50. 50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Accessibility
  51. 51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Accessibility • ElasticSearch Service Domain has two options for network configuration: We have 20+ VPCs.. 1) VPC Access 2) Public Access • IP based policy + Cognito authentication
  52. 52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Accessibility • Amazon Cognito: user sign-up, sign-in, and access control • Manage Authentication and access control for multi ES Kibana • Support multi-factor authentication, compromised credential checking, and adaptive authentication Amazon Cognito
  53. 53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  54. 54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elasticsearch index rotation
  55. 55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elasticsearch index rotation • Data pipeline works well • But how about cluster storage? • How Can I manage ES domain storage?
  56. 56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elasticsearch index rotation • Firehose support time-based index rotation: NoRotation, OneHour, OneDay, OneWeek, or OneMonth
  57. 57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elasticsearch index rotation
  58. 58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elasticsearch index rotation • Index can be rotated automatically But You have to delete index yourself! • Index Deletion on AWS Lambda function • Cloudwatch scheduled rule can trigger AWS Lambda function
  59. 59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  60. 60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Monitoring • Kinesis Stream getRecord counts • Kinesis Stream Incoming bytes , incoming record • Elasticsearch cluster free storage • Index deletion AWS Lambda error rate
  61. 61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Monitoring
  62. 62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  63. 63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary 자주 사용되는 서비스들을 쉽고 빠르게 개발하기 위해 아래와 같은 서버리스 패턴들을 활용하실 수 있습니다: • 웹 애플리케이션 • 데이터 레이크 • IoT 등의 데이터 스트림 처리 • 운영 및 리소스 관리 자동화 또한 서버리스 패턴을 이용한 개발 및 배포를 위해서 DevOps 도구와 SAM을 사용, 편리하게 버전 관리 및 배포를 할 수 있습니다. 관련 예제 등은 AWS Serverless Application Repository에서 찾아보실 수 있습니다: • https://serverlessrepo.aws.amazon.com/applications
  64. 64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Next Steps • 관련 Summit Seoul 2019 세션: 1. AWS Lambda 내부 동작 방식 및 활용 방법 자세히 살펴 보기 (김일호 솔루션즈 아키텍트) 2. Effective Data Lake : 디자인 패턴 및 모범 사례(유다니엘 솔루션즈 아키텍트 외) 3. 서버리스 기반 콘텐츠 추천 서비스 만들기(이상현, Vingle) 4. 타 게임사의 경험으로 본 AWS 핵심 모범 사례 한방에 배우기(이정훈 솔루션즈 아키텍트) 5. 신입 개발자가 스타트업에서 AWS로 살아남는 이야기(조용진, 모두의 캠퍼스) 6. AWS 기반 클라우드 아키텍처 모범사례 - 삼성전자 개발자 포털/개발자 워크스페이스(정영준 솔루션즈 아키텍트 외) • 리소스: 1. AWS re:Invent 2017: Getting Started with Serverless Architectures (CMP211) https://www.youtube.com/watch?v=0ytBy4-fvo4 2. AWS re:Invent 2017: Thirty Serverless Architectures in 30 Minutes (SRV213-R) https://www.youtube.com/watch?v=xJcm9V2jagc

×