Recommended
Recommended
More Related Content
Similar to Up-Armoring The Elephant: Adding Kerberos-based Security to Hadoop
Similar to Up-Armoring The Elephant: Adding Kerberos-based Security to Hadoop (20)
Recently uploaded
Recently uploaded (20)
Up-Armoring The Elephant: Adding Kerberos-based Security to Hadoop
- 1. Up-Armoring the Elephant Secure Hadoop is Here Jakob Homan jhoman@yahoo-inc.com
- 2. Who I am 8/14/10 2
- 3. Using Hadoop at Yahoo! 8/14/10 3
- 4. As of 2009, 72% percent of patches going into the Hadoop source code were coming from Yahoo! Developing Hadoop at Yahoo! 8/14/10 4
- 5. Yahoo! provides extensive QE and QA resources to test Hadoop releases at scale. Developing Hadoop at Yahoo! 8/14/10 5
- 6. Developing Hadoop at Yahoo! 8/14/10 6 The Yahoo! distribution of Hadoop, available on Github, is the same code we run internally on our servers. Patches important to stability and performance and stability are applied here, as well as Apache.
- 7. Developing Hadoop at Yahoo! 8/14/10 7 The rest of the family
- 8. Hadoop at Yahoo! Sunnyvale 8/14/10 8
- 9. Why do we need a secure Hadoop? 8/14/10 9
- 10. Current state of security in Hadoop 8/14/10 10
- 11. Current state of security in Hadoop 8/14/10 11 Bowser copyright Nintendo
- 12. The elephant is too trusting 8/14/10 12
- 13. Which can let bad people do bad things 8/14/10 13
- 14. Why is securing Hadoop hard? 8/14/10 14
- 15. Enter Kerberos! 8/14/10 15
- 16. Kerberos workflow 8/14/10 16
- 17. RPC upgraded to use SASL/GSSAPI 8/14/10 17
- 18. What does a secure Hadoop look like? 8/14/10 18
- 19. Like this 8/14/10 19
- 20. Everyone now authenticated 8/14/10 20
- 21. Additional security throughout system 8/14/10 21
- 22. How do I write a secure MapReduce job? 8/14/10 22
- 23. This is how 8/14/10 23 Nochanges!
- 24. Significant user-facing changes 8/14/10 24
- 25. Secure web access is pluggable 8/14/10 25
- 26. DistCP works… in 3 out of 4 cases 8/14/10 26
- 27. Out of scope 8/14/10 27
- 28. Impact on performance 8/14/10 28
- 29. Take security for a test drive 8/14/10 29
- 30. Or build a secure cluster at home 8/14/10 30
- 31. Other projects and security 8/14/10 31
- 32. Current state 8/14/10 32
- 33. Current state 8/14/10 33
- 34. Security list 8/14/10 34
Editor's Notes
- Discussion of how security was not a huge priority. File system permissions not added until 17.
- Kerberos was chosen because it’s a tested, trusted solution. In use at Yahoo! already. Hadoop actors – users, servers such as NameNode, JobTracker, DNs, TTs all authenticate with Kerberos as principals. This allows Hadoop, for the first time to be able to trust the identity of its various components.
- Kerberos provides single-sign-on serviceKinit, kdestroyCan be configured to automatically initialize via PAMBy default last 10 hours, renewable 7 days
- Overall, the entire ship has been tightened. We believe that we’ve secured each of the data access points that were shown in the big scary picture previously.
- * Secure Distributed Cache
- For the majority of jobs, there will be no changes necessary to run under security. It was important to make the switch to security as painless as possible since there are already thousands of different jobs running on our clusters, hundreds of thousands around the world and those needed to continue to run. Also, user education is very difficult.