Kubernetes Korea User Group에서 주최한 Meetup 발표자료 입니다. (2019. 03. 07)

1. 멀티클라우드 Service Mesh

2. 2
Contents
멀티클라우드
Istio
Service Mesh
Demo : Traffic Routing
Multi-Cloud Service Mesh
Multi-Cloud Service Mesh To-Be Model

3. 3
멀티클라우드란?
§ 2개 이상의 Cloud Provider가 제공하는 Public 또는 Private(On-Premise) 환경에 구성된 클라우드
Public Cloud1 Public Cloud2
Private Cloud On-Premise
§ 고려사항
• Workload
• 적용기술
• 네트워크
§ 구성 기준
• 접근성
• 확장성
• 안정성

4. 4
멀티클라우드 대표 모델
Hybrid Cloud Extend Cloud
HA Cloud Bursting Cloud
Workload2Workload1
Public Cloud
Workload4Workload3
Private Cloud (On-Premise)
Workload2Workload1
Main Cloud
Workload4Workload3
Expend Cloud
Workload2Workload1
Active Cloud
Workload1Workload1
Standby Cloud
Workload1Workload1
Workload1Workload1
Failover
Workload2Workload1
Main Cloud Bursting Cloud
Workload2+Workload1+
Bursting
ExtendConnect

5. 5
Hybrid Cloud
§ Private Cloud(On-Premise)와 Public Cloud간 연결하여 Workload를 통합하는 클라우드 방식
Workload 2Workload 1
Public Cloud
Workload 4Workload 3
Private Cloud (On-Premise)
• 접근성 용이 (Internet 연결)
• 다중 지역 배포 및 자동 확장
• On-Demand 형태의 다양한 서비스 제공
• Compliance 및 보안이 중요한 데이터 관리
• 인프라 및 H/W의 저수준 레벨까지 제어 가능
• 초기 도입 비용 지불후 Pay for use 없이 자원 제공
Connect

6. 6
Extend Cloud
Workload2Workload1
Main Cloud
Workload4Workload3
Expend Cloud
§ Main Cloud의 자원 및 Workload를 Extend Cloud를 통해 확장하는 클라우드 구성 방식
• 추가 Workload 확장
• Expend Cloud내 특화 서비스 사용가능
Extend
• Workload 자원 분할
• 최적의 서비스 배포장소 선택 가능
• 이종 클라우드 상품 조합에 따른 비용 감소

7. 7
High Availability Cloud
Workload2Workload1
Active Cloud
Workload4Workload3
Standby Cloud
Workload2Workload1
Workload4Workload3
Failover
§ 전체 Workload를 동일하게 구성하여 장애 발생시 서비스 무중단 혹은 빠른 복구를 제공하는 클라우드 방식
• 서비스 무중단으로 제공 가능 (Datacenter/Zone/Region 규모의 장애 발생)
• 데이터는 항시 백업 혹은 복제로 RPO(복구시점목표) 최소화
• Standby Cloud 지점에 필수적인 Workload를 일부 기동하여 RTO(복구시간목표) 최소화
• Infrastructure as Code 기반 복구 자동화 (RTO 최소화)

8. 8
Bursting Cloud
Workload2Workload1
Main Cloud Bursting Cloud
Workload2+Workload1+
Bursting
§ 특정 시점에 자원 부족시 자동으로 Scale-out 하여 일시적으로 많은 Workload를 수용할 수있는 환경 제공
• 일시적인 동적 서비스 대상 Workload 자원 확장
• 여유자원 프로비저닝 및 유지 불필요
• 초과용량 기준 지정 불필요
• Infrastructure as Code 기반 자원 확장 자동화

9. 9
Service Mesh
Service1 POD
Container Sidecar Proxy
Service2 POD
Container Sidecar Proxy
Service3 POD
Container Sidecar Proxy
§ Microservice간 통신 및 트래픽을 제어하는 Service Network Layer
• Microservice 간 통신이 서비스 Network 계층의 Proxy를 통해 수행
• 개별 Proxy는 서비스 내부가 아니라 서비스와 분리된 Sidecar 형태로 실행
• Sidecar Proxy들 간의 Mesh Network를 형성
• Sidecar Proxy는 연결된 Container 대신 타 서비스로의 통신 및 트래픽 제어
Service4 POD
Container Sidecar Proxy
Service5 POD
Container Sidecar Proxy

10. 10
Istio
TelemetryTraffic Management Security
• mTLS (양방향 트래픽 암호화)
• Auth (서비스 접근 인증)
• RBAC (권한 제어)
– Namespace
– Service
– Method
• 서비스 Discovery
• Load Balancing
• 트래픽 라우팅
• HTTP/gRPC/TCP 트래픽 처리
• Policy 설정
• Fault Injection
• Failure Handling
• Monitoring (Metrics 기반)
• Logging
• 분산 Tracing
• Service Topology 제공
• Telemetry 항목 시각화
§ Service Mesh 구성 및 관리 플랫폼

11. 11
Istio 컴포넌트
§ Istio Data plane : Service Mesh 구성
§ Envoy : 서비스간 통신 및 트래픽 제어
Kubernetes
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Service1 POD
Container Proxy
Service2 POD
Proxy Container
• Istio Control plane : Service Mesh Controller 구성
• Pilot : 서비스 및 트래픽 관리, Policy 적용
• Mixer : Telemetry 연동, Policy 체크
• Citadel : 보안 관리 및 트래픽 mTLS 적용

12. 12
Service Mesh 구성용 오픈소스
§ Hipster Shop Microservice Demo
(https://github.com/GoogleCloudPlatform/microservices-demo)

13. 13
Service Mesh Architecture
On-Premise
Kubernetes
Users Internet Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Recommendation
Frontend
LoadGenerator Advertise
ProductCatalog
EmailCurrency
Payment
Shipping
Redis-Cart Cart
Checkout
Ingress
Gateway

14. 14
Istio 트래픽 관리 정책 - 전체
Users Internet
kind: Gateway
meta:
name: frontend-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
kind: ServiceEntry
spec:
hosts:
- "metadata.google”
ports:
- number: 80
protocol: HTTP
name: http
kind: DestinationRule
spec:
host: productcatalogservice
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
kind: VirtualService
spec:
hosts:
- productcatalogservice
http:
- route:
- destination:
host: productcatalogservice
subset: v1
weight: 25
- destination:
host: productcatalogservice
subset: v2
weight: 75
kind: VirtualService
spec:
hosts:
- "*"
gateways:
- frontend-gateway
http:
- route:
- destination:
host: frontend
port:
number: 80
Ingress
Gateway
frontend
Container Sidecar Proxy
Productcatalogservice-v1
Container Sidecar Proxy
Productcatalogservice-v2
Container Sidecar Proxy
Egress
Gateway
Internet GCP Meta Server
Istio Data Plane

15. 15
Istio 트래픽 관리 정책 - Gateway
Users Internet
Ingress
Gateway frontend
Container Sidecar Proxy
Productcatalogservice-v1
Container Sidecar Proxy
Productcatalogservice-v2
Container Sidecar Proxy
Istio Data Plane
§ Ingress Gateway에 적용
§ Service Mesh 외부 수신 트래픽 활성화 (기본적으로 Istio는 모든 외부 트래픽을 차단)
§ Edge Load Balancer 연결
§ 활성화 할 프로토콜, 포트, 호스트(서비스) 설정 (HTTP 80, 전체 서비스 대상)
Egress
Gateway
Internet GCP Meta Server
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: frontend-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"

16. 16
Istio 트래픽 관리 정책 – VirtualService
Ingress
Gateway frontend
Container Sidecar Proxy
Productcatalogservice-v1
Container Sidecar Proxy
Productcatalogservice-v2
Container Sidecar Proxy
Egress
Gateway
Istio Data Plane apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: productcatalogservice
spec:
hosts:
- productcatalogservice
http:
- route:
- destination:
host: productcatalogservice
subset: v1
weight: 50
- destination:
host: productcatalogservice
subset: v2
weight: 50
Users
Internet GCP Meta Server
Internet
§ Sidecar Proxy에 적용
§ 라우팅 제어규칙 설정
§ Gateway 정책과 맵핑하여 라우팅 활성화 (frontend 서비스)
§ 트래픽 Weight(가중치 비율) 적용 (productcatalog 서비스)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: frontend-ingress
spec:
hosts:
- "*"
gateways:
- frontend-gateway
http:
- route:
- destination:
host: frontend
port:
number: 80

17. 17
Istio 트래픽 관리 정책 - DestinationRule
Users Internet Ingress
Gateway
frontend
Container Sidecar Proxy
Productcatalogservice-v1
Container Sidecar Proxy
Productcatalogservice-v2
Container Sidecar Proxy
Egress
Gateway
Istio Data Plane
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: productcatalogservice
spec:
host: productcatalogservice
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
Internet GCP Meta Server
§ Sidecar Proxy에 적용
§ 라우팅 발생후 적용할 정책 설정
§ VirtualService 정책과 맵핑하여 라우팅 구현
§ Label별 Subset(부분집합) 적용 (productcatalog v1, v2)

18. 18
Istio 트래픽 관리 정책 - ServiceEntry
Users Internet Ingress
Gateway
frontend
Container Sidecar Proxy
Productcatalogservice-v1
Container Sidecar Proxy
Productcatalogservice-v2
Container Sidecar Proxy
Egress
Gateway
Internet GCP Meta Server
Istio Data Plane
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: whitelist-egress-googleapis
spec:
hosts:
- "169.254.169.254" # GCE metadata server
- "metadata.google" # GCE metadata server
- "metadata.google.internal" # GCE metadata server
- "accounts.google.com" # Used to get token
- "*.googleapis.com"
ports:
- number: 80
protocol: HTTP
name: http
- number: 443
protocol: HTTPS
name: https
§ Egress Gateway에 적용
§ 외부 요청을 하기위한 트래픽을 활성화 (기본적으로 Istio는 모든 외부 트래픽을 차단)
§ 활성화 할 프로토콜, 포트, 호스트(서비스) 설정 (GCP Meta Svr, HTTP:80, HTTPS:443)

19. 19
Multi-Cloud + Service Mesh 통합 구조
Public Cloud
(IaaS – AWS, Azure, Cloud Z)
Service Mesh
(Istio)
Private Cloud
(IaaS – VMware, Openstack)
On-Premise
(IaaS – Bare-metal)
Container Orchestration
(CaaS - Kubernetes)
Common tool chain
(CI/CD, Provisioning, Automation, Registry, Telemetry, Policy, IAM, Security)
Container Orchestration
(CaaS - Kubernetes)
Container Orchestration
(CaaS - Kubernetes)
Application1 Application2 Application3 Application4 Application 5 Application6
§ Multi-Cloud 내 Service Mesh 방식으로 안정적인 서비스 통합운영 및 확장 가능한 구조

20. 20
Multi-Cloud + Service Mesh 통합을 위한 요소
1. CI/CD 기반 Code Pipeline을 통해 빠른 서비스 제공 및 확장
• Github
• Jenkins
• Spinnaker
2. 서비스간 통신 및 트래픽 관리를 통한 대규모의 Microservice 운영/관리
• Envoy
• Istio
3. Application을 Container화 하여 Microservice 단위 Workload 일관성 유지
• Docker
• Kubernetes
4. 자동화 Provisioning시스템을 활용하여 빠른 환경구성 및 Workload 자원 동적 확장
• Terraform
• Packer
• Ansible

21. 21
Multi-Cloud Service Mesh Architecture (Extend Cloud)
On-Premise
Kubernetes
Users Internet Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Recommendation
LoadGenerator Advertise
ProductCatalog-v1
PaymentCart
Ingress Gateway
Cloud Z
Redis-Cart
Currency
ShippingEmail
ProductCatalog-v2
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
Checkout
Frontend
Subnet :10.0.0.0/16, 10.1.0.0/16
172.30.0.0/16, 172.21.0.0/16
Subnet : 172.17.0.0/16
Subnet : 169.56.94.32/27

22. 22
Multi-Cloud Service Mesh 구성/설치 절차
On-Premise Cloud Z
Kubernetes – Main Cluster Kubernetes – Remote Cluster
8. Istio 설치
9. 설치가 완료되면 Control Plane 컴포넌트 및
Ingress/Egress Gateway가 istio-system namespace에
생성
10. Pilot, Policy, Telemetry POD IP 복사
11. Helm으로 Istio-Remote 설치
§ Setting 옵션값중 Pilot, Policy, Telemetry POD
IP 추가
12. 설치가 완료되면 Sidecar-Injector 및 Citadel
컴포넌트가 istio-system namespace에 생성
13. K8s Remote Cluster의 Context config 파일 생성
14. Context config 세팅용 파일 생성
15. Namespace Sidecar Injection Label 추가
§ kubectl label ns default istio-
injection=enabled
16. Context config 파일 및 세팅용 파일을 K8s Main
Cluster로 복사
17. Context config 세팅용 파일을 환경변수로 등록
18. K8s Remote Cluster 연결용 Secret 파일 생성
19. Namespace Sidecar Injection Label 추가
§ kubectl label ns default istio-
injection=enabled
4. Helm Strongswan value.yaml 설정
§ K8s Remote Cluster에서 복사한 Load
balancer와 맵핑된 External-IP를 설정
5. Helm으로 Strongswan 설치
6. ipsec 상태 확인 및 ping 체크
1. Helm Strongswan value.yaml 설정
2. Helm으로 Strongswan 설치
3. Strongswan의 Service 항목중 Load balancer와 맵핑된
External-IP 복사
7. ipsec 상태 확인 및 ping 체크

23. 23
Multi-Cloud Service Mesh Routing Flow 1
On-Premise
Kubernetes
Users Internet
Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Cloud Z
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
productcatalogservice.default.svc.cluster.local
Ingress Gateway frontend
Service Proxy
productCatalog-v1
10.0.61.63:3550
productCatalog-v2
172.21.192.101:3550
Service Proxy
Service Proxy

24. 24
Multi-Cloud Service Mesh Routing Flow 2
On-Premise
Kubernetes
Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Cloud Z
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
Ingress Gateway frontend
Service Proxy
productCatalog-v1
10.0.61.63:3550
productCatalog-v2
172.21.192.101:3550
Users Internet
On-Premise K8s Cluster
• Service Endpoint - 10.0.61.63:3550
• Service Domain Name
productcatalogservice.default.svc.cluster.local
productcatalogservice.default
productcatalogservice
Cloud Z K8s Cluster
• Service Endpoint - 172.21.192.101:3550
• Service Domain Name
productcatalogservice.default.svc.cluster.local
productcatalogservice.default
productcatalogservice
Service Proxy
Service Proxy

25. 25
Multi-Cloud Service Mesh Routing Flow 3
On-Premise
Kubernetes
Users Internet
Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Cloud Z
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
Ingress Gateway frontend
Service Proxy
productCatalog-v1
10.0.61.63:3550
productCatalog-v2
172.21.192.101:3550
productcatalogservice
productcatalogservice.default
productcatalogservice.default.svc.cluster.local
{
10.0.61.63:3550
172.21.192.101:3550
}
Service Proxy
Service Proxy

26. 26
Demo : Traffic Routing
On-Premise
Kubernetes
Users Internet
Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Cloud Z
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
Ingress Gateway frontend
Service Proxy
productCatalog-v1
productCatalog-v2
Service Proxy
Service Proxy
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: productcatalogservice
spec:
host: productcatalogservice
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: productcatalogservice-v1
spec:
template:
metadata:
labels:
app: productcatalogservice
version: v1
spec:
containers:
- name: server
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: productcatalogservice-v2
spec:
template:
metadata:
labels:
app: productcatalogservice
version: v2
spec:
containers:
- name: server

27. 27
Demo : Traffic Routing (100:0)
On-Premise
Kubernetes
Users Internet
Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Cloud Z
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
Ingress Gateway frontend
Service Proxy
productCatalog-v1
productCatalog-v2
Service Proxy
Service Proxy
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: productcatalogservice
spec:
hosts:
- productcatalogservice
http:
- route:
- destination:
host: productcatalogservice
subset: v1

28. 28
Demo : Traffic Routing (50:50)
On-Premise
Kubernetes
Users Internet
Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Cloud Z
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
Ingress Gateway frontend
Service Proxy
productCatalog-v1
productCatalog-v2
Service Proxy
Service Proxy
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: productcatalogservice
spec:
hosts:
- productcatalogservice
http:
- route:
- destination:
host: productcatalogservice
subset: v1
weight: 50
- destination:
host: productcatalogservice
subset: v2
weight: 50

29. 29
Demo : Traffic Routing (0:100)
On-Premise
Kubernetes
Users Internet
Load Balancer
Istio Control Plane
Istio Data Plane
Pilot Mixer Citadel
Cloud Z
Kubernetes
Istio Data Plane
Internet
VPN VPN
Strong
swan
Strong
swan
Ingress Gateway frontend
Service Proxy
productCatalog-v1
productCatalog-v2
Service Proxy
Service Proxy
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: productcatalogservice
spec:
hosts:
- productcatalogservice
http:
- route:
- destination:
host: productcatalogservice
subset: v2

30. 30
Multi-Cloud Service Mesh To-Be Model
AWS Cloud
Cloud Z
Azure Cloud
DNS or Global Server Load Balancer
VPN Gateway
Elastic Kubernetes Service Azure Kubernetes Service
Kubernetes
Kubernetes
Istio Control Plane
Istio Control Plane
Istio Data Plane
Istio Data Plane
Istio Data Plane
Istio Data Plane
Pilot Mixer Citadel
Recommendation
ProductCatalogFrontend
CheckoutAdvertiseEmail
Ingress Gateway
Load Balancer
CartRedis-Cart
CurrencyPayment
Shipping
AI Service Blockchain
Bigdata,ML,DL Service Serverless
VPN
VPN
Direct Connect
Storage Gateway
VPN
Object StgVolumeRepositoryMonitoring TracingLogging Registry
Volume RDB NoSQL CI CDServerless Github
Watson
Ethereum
Tensorflow
Load Balancer
Recommendation
ProductCatalogFrontend
Checkout
Advertise
Email
CartRedis-Cart
Currency
Payment Shipping
Telemetries Registries Storages
Databases Code Pipelines
Ingress Gateway
Egress
Gateway
Egress
Gateway
Egress
Gateway
Pilot Mixer Citadel
VPN Egress Gateway
Volume
1. Hybrid Service Mesh
• AWS Cloud : Public Cloud
• On-Premise : Private Cloud
2. HA Service Mesh
• Hybrid Cloud : Active Cloud
• Azure Cloud : Standby Cloud
3. Extend Service Mesh
• Hybrid Cloud : Main Cloud
• Cloud Z : Extend Cloud
VolumeRDBNoSQLLDAPPolicy MgmtCluster Meta
Managements Databases
Gitlab
On-Premise

31. Q & A

32. End Of Document
나정호 (skwjdgh1@gmail.com)