Presentation at StackStorm meetup SF 2018-06-12

1. INTERNET MULTIFEED CO.Copyright ©
StackStorm on AWS EKS demo
Shu Sugimoto
Software Development Manager, JPNAP
2018-06-12(Tue)

2. INTERNET MULTIFEED CO.Copyright ©
Who am I
• Shu Sugimoto
• handles: shu, shusgmt
• Software Development Manager, JPNAP
• Agile, DevOps
• based in Tokyo
• JPNAP
• IXP: Internet Exchange Provider
• one of the largest in APAC
• JPNAP Tokyo
• Peak traffic: 1.09Tbps
2

3. INTERNET MULTIFEED CO.Copyright ©
st2 “facts” at JPNAP
• Use case
• Auto-provisioning of customer ports
• Involves switches, routers, servers
• st2 = Workflow engine
• kicks by executions API call
• no sensors
• Packs/Actions/Workflows development
• Mostly develop our own: nothing from exchange packs
• 99% of actions are `remote-shell-cmd` runner
• ~10 private packs, ~50 actions + workflows
• A very few python code
• 100% Mistral
• Heavily dependent on `core.ask`/inquiries
3

4. INTERNET MULTIFEED CO.Copyright ©
st2 “facts” at JPNAP
• “Current” deployment
• st2-docker
• Multi-process mode
• No HA yet
• Principle: Keep important data out of st2
• No Kubernetes yet
• Community LDAP Auth backend enabled
• Packs CI/CD
• CI: Not yet
• CD: Jenkins
4

6. INTERNET MULTIFEED CO.Copyright ©
Demo scenario
• Run several workflows
• Bring down two nodes in same AZ
• See what happens
6

7. INTERNET MULTIFEED CO.Copyright ©
Demo info
• All in AWS Oregon(us-west-2) region
• Amazon EKS
• k8s version: 1.10
• 6x m5.large instances for nodes
• across 3 AZ by autoscaling group
• Amazon EFS for efs-provisioner
• gives RWX storage backend
• Middlewares
• MongoDB: helm chart: stable/mongodb-replicaset
• 3 replicas (default)
• RabbitMQ: helm chart: stable/rabbitmq-ha
• 3 replicas (default)
• PostgreSQL: Amazon RDS Aurora PostgreSQL
• db.r4.large
• 2 AZ HA Cluster
• Redis: Amazon ElastiCache Redis
• cache.m4.large
• 3 AZ HA Cluster
• StackStorm
• All components are scaled to 6 pods
• …except following components:
• st2sensorcontainer
• st2rulesengine
• st2resultstracker (we don’t need it any more in st2 2.7)

8. INTERNET MULTIFEED CO.Copyright ©
Setup summary
• create k8s cluster with `eksctl`
• edit Launch Configuration for nodes to add `yum -y install nfs-utils` to support efs-
provisioner, and replace all nodes
• configure EBS storage class
• create EFS
• configure efs-provisioner
• setup helm
• install MongoDB with helm chart
• install RabbitMQ with helm chart
• create Aurora PostgreSQL instance
• create ElastiCache Redis instance
• prepare configmap
• kubectl apply -f configmaps.yml
• kubectl apply -f st2.yml
• kubectl apply -f init-st2.yml
• detailed step-by-step guide is here:
• https://qiita.com/shusugmt/items/1cdb382579352c67dc08
• (use Google translate!)
8

9. INTERNET MULTIFEED CO.Copyright ©
Caveats / Considerations
• Pod affinity needs to be set for real HA scenario
• Following components (still) need HA specific care
• st2sensorcontainer
• st2rulesengine
9

10. INTERNET MULTIFEED CO.Copyright ©
Caveats / Considerations
• Building k8s cluster with EKS is much harder than GKE
• `eksctl` solves this
• Running and managing cluster: ?
• GKE provides “1 click upgrade” solution
• RWX PV problem can be solved in AWS with EFS
• GCP/GKE still lacks built-in support for RWX PV
• Need custom built image without RWX PV
https://github.com/shusugmt/st2-docker-pack-prebuild-
example
• EFS = managed NFS
• Performance, Stability: ?
10

11. INTERNET MULTIFEED CO.Copyright ©
Caveats / Considerations
• MongoDB / RabbitMQ
• helm chart seems good choice
• Amazon MQ has no luck since it is ActiveMQ
• st2 can’t support ActiveMQ since underlying library (which
is celery/kombu) can’t talk AMQP1.0
• RDBMS for Mistral
• AWS: Amazon RDS
• GCP: Cloud SQL
• Redis for coordination backend
• AWS: Amazon ElastiCache
• GCP: ?
• helm chart available: stable/redis-ha
11