I talk about Elasticsearch on AWS, and on those slides I cover the topics High Availability (HA) and Security best practices.

1. ES on AWS
Implementing ElasticSearch on AWS
~ High Availability and Best Security practices ~

2. Who Am I
(log nerd AND DevOp AND Infrastructure Manager AND
photographer AND .*) XOR (daddy);
rafael@psafe.com for company and business purposes;
dev@rafalop.es to have a beer and eat something;
@bobeirasa;
bobeirasa on Freenode IRC;
;
You can also find me on Elasticsearch-PT on Google Groups;
Tags: bobeirasa, rafalopes, rafael lopes, psafe, mpran, fotografia cotidiana;

3. About PSafe Tecnologia
• Brazilian StartUp focused on security;
• More than 12 MM Android Users;
• #6 app on Top Free Google Play, Browser, Security Suite for Win/Mac and
developing more products;
• Infrastructure with more than 200 high-end servers, tons of memory and bandwidth;
• 3 offices in Brazil, RJ [headquarters], SP and SC;
• LatAm focus with products localized to portuguese and spanish;
• We’ve been looking for you, join us!
Tags: psafe, qihoo, startup, brazilian, psafe techchrunch, VC investment, series C;

4. What will be covered
• High Availability (HA);
• Security;
Tags: rafael lopes, meetup, elasticsearch, segundo encontro;

5. High Availability
~ topics ~
• AWS Region and Availability Zone (AZ);
• Multi-AZ environment;
• Shard allocation awareness;
• unicast or elasticsearch-cloud-aws plugin;
• AZ label mismatch regarding your accounts;
Tags: rafael lopes slides, meetup Elasticsearch rio, segundo encontro, slides;

6. High Availability
AWS Region and Availability Zone (AZ)
Tags: AWS region, AWS availability zone, difference;

7. High Availability
Don’t run your cluster on the same AZ !!!
Tags: Elasticsearch Single-AZ, failure, ec2 classic, failover, disaster, downtime, AWS AZ fail;

8. High Availability
Run it on separate Availability Zones
Tags: Elasticsearch Multi-AZ, VPC, AWS different Subnet, Multi-AZ advantages, HA, high availability;

9. High Availability
Multi-AZ on Elasticsearch Cluster
Tags: Elasticsearch shard allocation awareness, hadoop rack affinity, replica, safe shards;

10. High Availability
Elasticsearch Shard Allocation Awareness
Awareness is a keepalive concept, used to make sure primary shards and
it’s corresponding replicas don't end up on same place, such as rack,
environment or AWS Availability Zones!
Tags: shard allocation awareness, hadoop rack affinity, replica, safe shards, raid10 comparison;

11. High Availability
Shard Allocation Awareness configuration
node.name = “ES01”
node.rack_id: zona-A
cluster.routing.allocation.awareness.attributes: rack_id
!
node.name = “ES02”
node.rack_id: zona-B
cluster.routing.allocation.awareness.attributes: rack_id
!
node.name = “ES03”
node.rack_id: zona-A
cluster.routing.allocation.awareness.attributes: rack_id
!
node.name = “ES04”
node.rack_id: zona-B
cluster.routing.allocation.awareness.attributes: rack_id
Tags: shard allocation syntax, Elasticsearch shard allocation configuration;

12. High Availability
AWS doesn’t allow multicast/broadcast
Tags: elasticsearch ec2 discovery, ec2 broadcast, ec2 ARP, VPC FAQ, VPC limitations, rackspace multicast;

13. High Availability
AWS doesn’t allow multicast/broadcast
2 solutions for that !
Tags: elasticsearch ec2 discovery, ec2 broadcast, ec2 ARP, VPC FAQ, VPC limitations, rackspace multicast;

14. High Availability
Disable multicast, use unicast to specify nodes
cluster.name: StarTrek
node.name: "Locutus"
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 5s
discovery.zen.ping.unicast.hosts: [“Voyager”]
!
cluster.name: StarTrek
node.name: "Voyager"
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 5s
discovery.zen.ping.unicast.hosts: [“Locutus”]
!
cluster.name: StarTrek
node.name: "Enterprise"
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 5s
discovery.zen.ping.unicast.hosts: [“Locutus”, “Voyager” ]
Tags: Elasticsearch unicast, Elasticsearch zen, Elasticsearch discovery ping, Elasticsearch manual hosts;

15. High Availability
Use elasticsearch-cloud-aws plugin
Tags: elasticsearch-cloud-aws, official plugin, github, Elasticsearch AWS credentials;

16. High Availability
More about the elasticsearch-cloud-aws plugin
• Easy installation (as any other plugin):
./plugin -install elasticsearch/elasticsearch-cloud-aws/2.3.0
• Official plugin provided by Elasticsearch;
• Uses AWS API ec2-describe-instances to know stuff about your AWS ec2 instances. Smart,
huh ?
• Compatible with IAM roles (stay with me to see more details about this on Security session);
• Uses ec2 tagging;
• Check Elasticsearch version and Plugin version for mismatches;
• It’s also used for s3 snapshots (which we won’t cover here);
Tags: elasticsearch-cloud-aws install, elasticsearch-cloud instance metadata, ec2 discovery, fake multicast;

17. High Availability
AZ label mismatch across different AWS accounts
Tags: AWS availability zone mismatch, Elasticsearch on different AWS accounts;

18. High Availability
~ recap ~
AWS Region and Availability Zone (AZ);
Shards are automatically replicated across the cluster, but don’t run your cluster
on the same AZ;
Use shard allocation awareness to tell ES to assign shards to different AZ’s (like
Hadoop rack affinity);
AWS doesn’t allow multicast at all, so use unicast instead of multicast, or
elasticsearch-cloud-aws plugin;
If you want (for any reason) to use the same cluster running different AWS accounts,
open a support ticket to check AZ label mismatch regarding your accounts;
Tags: rafael lopes slides, meetup Elasticsearch rio, segundo encontro, slides;

19. Security
~ topics ~
• built-in security credentials and web proxy;
• IAM roles;
• Kibana;
• Bastion server;
• Multicast protocol;
• External scripting;
Tags: rafael lopes slides, meetup elasticsearch rio, segundo encontro, slides;

20. Security
MiTM (man in the middle)
• Elasticsearch does not provide native connection strings, passwords or any kind of
authentication: traditional DBA’s would say that Elasticsearch is insecure because of this; they
say that because they are traditionalists, not DevOps ;)
• Change Elasticsearch http port and use a local firewall to close this port to everybody
unless localhost;
• It must reside on a private subnet with no communication to the public, as any database
server;
• Use 3 firewalls. In addition to security groups and NACL’s, create local firewall rules to
allow traffic just to the consumers (frontend, middletier or internal elb) on the
‘Elasticsearch port’;
• Install a web server acting as a proxy, such as nginx or mitmproxy to control requests and
passwords;
• Jetty plugin also used to implement authentication and encription;
Tags: Elasticsearch proxy, Elasticsearch nginx, protecting Elasticsearch, Elasticsearch firewall, NACL,AWS
security group, AWS private subnet, iptables, ipfw, linux firewall;

21. Security
nginx example: limit everything but not GET on /_settings,
except if the host is 127.0.0.1
location ~ /_settings {
proxy_pass http://elasticsearch;
limit_except GET {
allow 127.0.0.1;
deny all;
}
}
Tags: nginx Elasticsearch protect

22. Security
nginx - implement basic password authentication, if the host is
127.0.0.1, this auth can be skipped
location ~ /_plugin {
satisfy any;
allow 127.0.0.1;
deny all;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/pwds/file.pwd;
proxy_pass http://elasticsearch;
}
Tags: nginx Elasticsearch protect

23. Security
IAM Roles
Tags: IAM roles, codespaces.com, identity steal, access key

24. Security
IAM Roles
• Mandatory to security best practices;
• Keys expires in 6 hours (James Bond would like to use that);
• No access keys in the code or config files;
• Uses the ec2 metadata (http layer between ec2 and
hypervisor) to get the provisory key;
• It just works magically with the elasticsearch-cloud-aws plugin;
Tags: AWS security best practices, AWS IAM roles, ec2 instance metadata, AWS STS, MFA

25. Security
Kibana is a frontend tool meant to be used in internal network
(VPN). Kibana, like ES should not be exposed to public internet
Tags: kibana, protect, node.js, wrapper, kibana frontend danger, close kibana

26. Security
Use a Bastion Server
Tags: bastion server, security best practice;

27. Security
Disable Multicast in production environment
• Reduces noise on environment;
• Safer due to eavesdroppers/sniffers in shared
environments;
Tags: multicast, production environment, security best practice;

28. Security
~ recap ~
Elasticsearch does not provide built-in security credentials, connection strings or passwords,
so protect yourself using a web proxy such as nginx or mitmproxy to prevent deletes or
updates;
IAM roles with AWS STS (simple token service) on ec2 instances (also works with cloud
plugin), stop copy-pasting access key and secret over there;
Kibana is for insiders: use a wrapper to protect if you want to show to the world, or set it
up on an private subnet with VPN access;
Use a bastion server with 2 factor auth on ssh layer to make the DevOps works;
On a shared environment such as a cloud, disable multicast at all;
Disable external scripting if you don't use it;
Tags: rafael lopes slides, meetup elasticsearch rio, segundo encontro, slides;

29. Questions ?

30. Thank you!
Coffee Break ?
(sorry for make you wait so long)
Tags: free food, free bier, elasticsearch rio second meetup, ideais tecnologia, sponsor, happiness;