Securing Hadoop with Apache Ranger

© Hortonworks Inc. 2011 – 2015. All Rights Reserved
Securing Hadoop With Apache Ranger
Strategies & Best Practices
11 June 2015 – Hadoop Summit, San Jose
Presented by:
Selvamohan Neethiraj, Hortonworks
Velmurugan Periasamy, Hortonworks

Session overview
Secured by:

Speakers Introduction
Selvamohan Neethiraj
Senior Director, Enterprise Security Development, HortonWorks
Velmurugan Periasamy
Senior Technical Manager, HortonWorks

Agenda
Current
Hadoop
Security
What
Ranger
Brings
Ranger
Demo
Best
Practices
Q & A

Universal Security Principles
• Authentication
– Who is the user?
• Authorization
– What can the user do?
• Auditing
– Record user’s activities
• Data Integrity
– It is what it is
• Confidentiality
– Restricted

How Hadoop handles security?
• Authentication [Simple, Kerberos]
• Authorization [Acl on files/folders]
• Auditing [audit log on hosts]
• Data Integrity [SSL communication]
• Confidentiality [TDE]

Security on Hadoop tool sets
• Hdfs
• Hive
• Hbase
• Knox
• Storm
• Oozie
• Spark
• Kafka
• Solr
• KMS

Example Access Scenario
HDFS
HDFS
ACL
HDFS
audit log
Users
Access HDFS file/folder
HDFS
checks its ACL
HDFS
Writes audit
Hive
Hive
Authoriza
tion
Hive
audit log
Hive authz
check
Hive
Writes audit Access Hive table
HBase
HBase
ACL
Hbase
audit log
Hbase authz
check
HBase
Writes audit
Access Hbase table

Example Access Scenario with Ranger
HDFS
HDFS
ACL
Users
Access HDFS file/folder
Plugin
Enforces policies
And writes audit
Hive
Hive
Authoriza
tion
Plugin
Enforces policies
And writes audit
Plugins Write to
Centralized
Audit store
Access Hive table
HBase
HBase
ACL
Plugin
Enforces policies
And writes audit
Access Hbase table
Ranger Policy store Ranger Audit store
Ranger
Plugin
Ranger
Plugin
Ranger
Plugin
Plugins Download
Defined Policies from
Centralized Policy store
Centralized
Security Policy
Management and
Auditing provided by
Ranger

Core Ranger Features:
• Ranger 0.4.0
• Authorization Support for HDFS, Hive, HBase, Knox, Storm
• Audit destination support for HDFS, DB
• Ranger 0.5.0 (Released yesterday!)
• Support for High Availability (HA)
• Support for Transparent Data Encryption with KMS implementation
• Support for more plugins (Yarn, Kafka, Solr)
• Solr as audit destination
• Stack based implementation of Plugins
• Dynamic policies with many other attributes
• Internal permission model within Ranger Admin UI
• Tighter integration with Apache Ambari

Ranger: Centralized Policy Admin

Ranger Architecture
Ranger
Policy Admin Server
Ranger
User Sync
Server
HDFS Ranger
Plugin
Enterprise
Directory Services
Sync Users
Security
Administrators
Enterprise
Users
Define Policies
Audit user activity
Sync Users
Enterprise
Legacy Systems
Policy
Integration
Access
Data in
Hadoop
Systems
Authenticate
Ranger Plugins
Authorize the Access &
Audit the Activity
Hive Ranger
Plugin
HBase
Ranger
Plugin
Knox
Ranger
Plugin
Storm
Ranger
Plugin
Ranger
Audit Store
Sync
Policies
Store
Audit
Activity
Ranger Policy
DB

Apache Ranger Summary
• Comprehensive security for Hadoop ecosystem
• Apache Incubator Project – 100% open source
• Centralized Administration of Security Policies
• Integration with HDFS, Hive, Hbase, Knox, Storm, …
• More components integration coming
• Ensures consistent coverage across entire Hadoop stack

Ranger Implementation - components
 Central interface for security administration.
 Users can create and update policies, view
audit activities, manage users
Ranger policy admin server
 Synchronization utility to pull users and groups
from Unix or LDAP or Active Directory.
 User/group information is stored within Ranger
admin policy DB and used for policy definition.
Ranger user sync server
 Lightweight Java programs within Hadoop component.
 Pull in policies from policy admin server and store them locally in cache.
 Act as Authorization module and evaluate user requests against security policies before granting access.
 Collect data from user request and store this data into the audit store.
Ranger plugins

Ranger Policy Admin Server
• Provides web interface to support ranger activities
• Define repositories (resources to be authorized)
• Define access policies (specify which users/groups can access to which resources)
• Manage users/groups
• Define auditing policies
• View/Analyze audit data
• Runs embedded tomcat server
• Supports LDAP/AD and Unix authentication

Ranger User/Group Sync Server
• Stand-alone Java server
• Retrieves users/groups from enterprise directories
• Creates these users/groups in Ranger DB for..
• supporting policy definition
• allowing access to Ranger policy admin server
• Supports Synchronization of users/groups:
• LDAP
• Active Directory
• Unix

Ranger Plugin – HDFS Integration
• HDFS Permissions
• POSIX like permission model (owner/group for files and folders)
• ACL’s for fine-grained permissions (for specific set of users/groups)
• hdfs dfs -getfacl [-R] <path>
• hdfs dfs -setfacl [-R] [options] <path>
• dfs.permissions.enabled must be set to true

Ranger Plugin – HDFS Integration
• Ranger Plugin acts as an authorizer within Namenode.
• Need to install in all name nodes (in HA environment)
• User can define policies on files and folders
• Use of wildcard to define policies (/finance/audit_*)
• Read, Write, Execute permissions are allowed
• Plugin evaluates HDFS requests and provide access
• If no specific ranger policy exists, HDFS ACLs are used as fallback

Ranger Plugin – Hive Integration
• Hive facilitates querying and managing large datasets in distributed storage
(on top of Hadoop)
• HiveServer2 (HS2) is a server interface to Hive
• HiveServer2 supports Access Control similar to relational database model
• SELECT/UPDATE/DELETE permissions on tables/columns
• Permission defined for USERS/ROLES
• Also provides a pluggable authorizer model

• Ranger Plugin acts as authorization provider for Hive Server2
• User can define policies on databases, tables/view, columns and UDFs
• Select, Update, Create, Drop, Alter, Index, Lock Permissions allowed
• Plugin evaluates Hive requests and grants/denies access based on the
policies and creates necessary audit logs based on audit
• Specific Ranger policy must exist for gaining access

• When the user executes GRANT/REVOKE statements, the hive plugin will
creates/deletes necessary Ranger Policies (provided the user has permission
to create/delete policies)
• GRANT SELECT, UPDATE, ALTER ON TABLE test_data TO USER guest; ← will create a new Ranger policy
• REVOKE UPDATE, ALTER ON TABLE test_data FROM USER guest; ← will update/delete existing Ranger policy
• Also, ROLE specified in the GRANT/REVOKE statements will be mapped to
corresponding GROUPS from your corporate directories
• Optionally, you can disable GRANT/REVOKE commands i.e. to force
authorization policies management solely via Ranger Policy Admin

Ranger Plugin – HBase Integration
• Hbase is a non-relational DB on top of Hadoop/HDFS
• HBase Provides Role Based Access Control/ACLs
• ACLs are implemented as a coprocessor called AccessController
• Ranger implements a similar coprocessor for enforcing access control based
on Ranger Policies

• Ranger Plugin is implemented as a coprocessor of HBase Master/Region
Servers to enforce Ranger Policies
• User can define policies on tables, column families and qualifiers
• Supports wildcard in defining policies (table = fin_*, col_fam = audit*)
• Read, Write, Create, Admin permissions allowed
• Plugin evaluates Hbase requests and grants/denies access based on the
policies and creates necessary audit logs based on audit
• Specific ranger policy must exist for gaining access

• Similar to Hive, Hbase also supports Grant/Revoke commands to manage
access control from within Hbase shell.
• Ranger HBase plug-in creates/updates Ranger policies to reflect permissions
set via Grant/Revoke (from within Hbase)
• grant ‘bob’, ‘RWCA’, ‘test_data’ ← will create a new Ranger policy
• grant ‘bob’, ‘R’, ‘test_data’ ← will update the Ranger policy created by earlier grant;
• Option to disable Grant/revoke commands i.e. to force authorization policies
management solely via Ranger Policy Admin tool

Ranger Plugin – Knox Integration
• Knox provides perimeter security for Hadoop REST API
• Authentication and token verification at the perimeter
• Authentication integration with enterprise and cloud identity management systems
• Service level authorization at the perimeter
• Single URL hierarchy that aggregates REST APIs of a Hadoop cluster
• Hadoop services with built-in support currently!
• WebHDFS, WebHCat, Oozie
• Hbase, Hive, Yarn

Ranger Plugin – Knox Integration
• Knox provides service level authorization based on xml configuration.
• Ranger Knox Plugin will allow service level authorization enforcement via
Ranger Policies by acting as Authz Provider within Knox Gateway
• User can define policies on topologies and services
• Provide access to services based on user/group/ip-address
• E.g. Finance group will have access to WebHDFS from 10.1.1.*
• Plugin evaluates Knox requests and grants/denies access based on the
policies and creates necessary audit logs based on audit policies

Ranger Plugin – Storm Integration
• Storm is a distributed real-time computation system.
• Storm provides general primitives for real-time computation similar to how
Hadoop provides general primitives for batch processing.
• Storm Topologies :: Hadoop MR Jobs
• Nimbus server (runs on master node) and Supervisor (runs on each worker
node). Communication via ZooKeeper
• Storm security is implemented based on Kerberos authentication

Ranger Plugin – Storm Integration
• Ranger Plugin acts as an authorizer within Nimbus server
• Ranger plugin then can authorize all incoming requests based on the
Ranger Policies.
• User can define policies on topologies
• Permissions that can be set
• getClusterInfo, Submit/Get/Kill/Activate/Deactivate Topology
• Plugin evaluates Storm requests and grants/denies access based on the
policies and creates necessary audit logs based on audit policies

Ranger Auditing
• Auditing is configured through policies (like authorization)
• Resource access audit supported with following attributes
• User id, request type, repository, access resource, IP address, timestamp, access
granted/denied
• Admin audit
• Changes to policies, login sessions and plugin-monitoring
• Audit Destination - HDFS, SOLR are supported, along with custom LOG4J
based logging …

Ranger Demo

Strategies & Best practices
• Authentication Process
• Kerberos is MUST
• Centralized Access Control & Auditing
• Use Ranger to define & enforce the security policies
• Enable Wire encryption
• For confidential data, use Hadoop TDE for data encryption (at rest)
• Aware of file permission for keystore & kerberos keytab
• Protect access to log files

Ranger Resources
• Do NOT Miss BoF on Apache Ranger/Knox
• Interested in Security ?
-- Join Ranger Community to strengthen Hadoop Security!!
• Join and Contribute!
-- Apache Ranger website - http://ranger.incubator.apache.org/
-- Ranger Wiki - https://cwiki.apache.org/confluence/display/RANGER/Index

References
• http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-
hdfs/HdfsPermissionsGuide.html
• https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization
• http://hbase.apache.org/book/hbase.accesscontrol.configuration.html
• https://knox.apache.org/books/knox-0-5-0/knox-0-5-0.html
• https://github.com/apache/storm/blob/master/SECURITY.md

Q & A
Page 37

Securing Hadoop with Apache Ranger

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Securing Hadoop with Apache Ranger

Similar to Securing Hadoop with Apache Ranger (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Securing Hadoop with Apache Ranger