Hortonworks technical workshop operations with ambari

© Hortonworks Inc. 2011 – 2015. All Rights Reserved
Operations With Apache Ambari
We Do Hadoop.

Apache Ambari
Apache Ambari is the open source
operational platform to provision, manage
and monitor Hadoop clusters

How Do People Use Ambari?
Health Checks, Alerts Stacks, Views
Lifecycle controls, Rolling
Restarts, Decommission/
Re-commission
Host Groups, Versioning,
Compare, Revert,
Recommendations,
Security Setup
Install Wizard (UI),
Blueprints (API)
Config
Management
ExtensibilityMonitoring
Service
Management
Cluster
Provisioning

Recent Ambari Releases
Ambari 1.7.0
Dec 2014
Ambari 1.6.0
May 2014
Introduced
Ambari Blueprints
Introduced
Ambari Views
Ambari 2.0.0
Apr 2014
HDP
2.2 GA

What’s New in Ambari 2.0
Core Platform
Simplified Kerberos Setup (AMBARI-7204)
Ambari Alerts (AMBARI-6354)
Ambari Metrics (AMBARI-5707)
Automated (Rolling) Upgrade (AMBARI-7804)
Stack Support
HDP 2.2: Ranger, Spark, Phoenix
Hive Metastore HA (AMBARI-6684)
HiveServer2 HA (AMBARI-8906)
Oozie HA (AMBARI-6683)
Ambari Platform
Handle umask 027 setting (AMBARI-7796)
Ambari Agent non-root (AMBARI-1596)
Blueprints API
Add Host (AMBARI-8458)
For a complete list of changes
https://issues.apache.org/jira/browse/AMBARI/fixforversion/12327486

Lab Setup

Ambari 2.0 Security Lab Steps – 4 node cluster
•  Detailed steps available at: http://bit.ly/1J4IbIs
•  Install Ambari server and agents
•  Deploy custom Ambari service folders for OpenLDAP, KDC/kadmin, NSLCD
•  Use blueprint/API to provision a minimal Hadoop cluster with custom services
•  Use Add service wizard to also install Hive
•  Configure Ambari to sync/recognize business users in OpenLDAP
•  Authentication: Run Ambari security wizard to enable kerberos using KDC/kadmin service
•  Install Ranger as Ambari service and configure it to recognize LDAP users
•  Authorization/Audit: Setup HDFS/Hive plugins to allow users to set policies to control access
and audit consumption

Core Platform

Stack Components Support
HDP 2.2 HDP 2.1 HDP 2.0
HDFS, YARN, MapReduce, Hive,
HBase, Pig, ZooKeeper, Oozie,
Sqoop
Tez, Storm, Falcon, Flume
Knox, Slider, Kafka
Ranger, Spark, Phoenix NEW in Ambari 2.0
install/manage/monitor

Admin > Stack and Versions
List of Stack Services
Installed or Add Service

Ambari 2.0.0 High Availability Support
High Availability Mode Ambari 1.6.1 Ambari 1.7.0 Ambari 2.0.0
HDFS: NameNode HDP 2.0+ Active/Standby
YARN: ResourceManager HDP 2.1+ Active/Standby
HBase: HBaseMaster HDP 2.1+ Multi-master
Hive: HiveServer2 HDP 2.1+ Multi-instance
Hive: Hive Metastore HDP 2.1+ Multi-instance
Oozie: Oozie Server* HDP 2.1+ Multi-instance
* Oozie Server needs external load balancer to complete HA solution

Hive HA
Services > Hive > Service Actions
+ Add Hive Metastore
+ Add HiveServer2

Ambari Agent Non-Root

Non-Root Ambari Agent
Agent Runs Commands From the Ambari Server
•  Configuration Change
•  Service Start
•  Service Stop
Some Command require root level access
•  /bin/su
hdfs
-‐l
-‐s
/bin/bash
-‐c
/usr/hdp/current/hadoop-‐client/sbin/hadoop-‐
daemon.sh
-‐-‐config
/etc/hadoop/conf
start
datanode

Sudo Leveraged
•  Configuration for:
–  Customizable Users (su hdfs, yarn, etc.)
–  Non-Customizable Users (su mysql)
–  Commands (yum, mkdir, touch, test, etc.)
Ambari
AgentAmbari
AgentAmbari
Agent
python

Configuring Agent for Non-Root
1.  Create and configure a sudoer account
2.  Manually bootstrap Ambari Agents
3.  Set run_as_user in ambari-agent.ini for the sudoer account
Details
http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.0.0/bk_ambari_reference_guide/
content/ch_amb_ref_configuring_ambari_for_non-root.html

Updated umask Handling

What about umask?
Unix Permissions Basics: (user, group, other)
4 – read
2 – write
1 – execute
rwxr-‐xr-‐x
== 755
Previous Behavior:
•  If (umask > 022); Warning during agent pre-req check
•  Installations would fail if ignored
New Behavior:
•  If (umask > 027); Warning during agent pre-req check
•  Installation will fail if ignored

Ambari Alerts

Ambari Alerts
Summary
•  Migrated away from Nagios as the Ambari alerting system
•  No longer offer option to install or manage a Nagios service
•  Replaced with built-in alerting system
Motivation
•  Avoids Nagios package conflicts in customer environments
•  More flexibility with alerts in Ambari Stacks
•  Platform independence

Ambari Alerts
•  Ambari Alerts are installed and configured by default
•  Ambari Web provides centralized management of Health Alerts

Modifying Alerts
•  Control thresholds, check intervals and response text

Alert Groups
•  Create and manage groups of
alerts
•  Group alerts further controls what alerts
are dispatched which notifications
•  Assign group to notifications
•  Only dispatch to interested parties

Alert Notifications
•  What: Create and manage
multiple notification targets
•  Control who gets notified when
•  Why: Filter by severity
•  Send only certain notifications to certain
targets based on severity
•  How: Control dispatch method
•  Support for EMAIL + SNMP

Ambari Alerting System
1.  User creates or modifies cluster
2.  Ambari reads alert definitions
from Stack
3.  Ambari sends alert definitions to
Agents and Agent schedules
instance checks
4.  Agents reports alert instance
status in the heartbeat
5.  Ambari responds to alert instance
status changes and dispatches
notifications (if applicable)
Ambari
Server
1
2
4
Stack definition
alerts.json
5
Ambari
Agent(s)
3
email
snmp

Notable Alert REST APIs
REST Endpoint Description
/api/v1/clusters/:clusterName/alert_definitions The list of alert definitions for the cluster.
/api/v1/clusters/:clusterName/alerts The list of alert instances for the cluster.
Example: find all alert instances that are CRITITAL
/api/v1/clusters/c1/alerts?Alert/state.in(CRITICAL)
Example: find all alert instances for “ZooKeeper Process” alert def
/api/v1/clusters/c1/alerts?Alert/
definition_name=zookeeper_server_process
/api/v1/clusters/:clusterName/alert_groups The list of alert groups.
/api/v1/clusters/:clusterName/alert_history The list of alert instance status changes.
/api/v1/alert_targets/ The list of configured alert notification targets for Ambari.

Ambari Metrics

Ambari Metrics
Summary
•  Migrated from Ganglia as the Ambari metrics collection system
•  No longer offer option to install or manage a Ganglia service
•  Replaced with built-in metrics system “Ambari Metrics”
Motivation
•  Avoids Ganglia package conflicts in customer environments
•  More flexibility to retain metrics in Hadoop
•  Platform independence

Terminology
Term Definition
Ambari Metrics (“AMS”) The built-in metrics collection system for Ambari (“AMS”).
Metrics Collector The standalone server that collects metrics, aggregates metrics, serves
metrics from the Hadoop service sinks and the Metrics Monitor. Analogous
to gmetad.
Metrics Monitor Installed on each host in the cluster to collect system-level metrics and
forward to the Collector. Analogous to gmond.
Metrics Hadoop Sinks Plugs into the Service sinks to send Hadoop metrics to the Collector.

Ambari Metrics Collection System
1.  Metric Monitors send system-
level metrics to Collector
2.  Sinks send Hadoop-level
metrics to Collector
3.  Metrics Collector service stores
and aggregates metrics
4.  Ambari exposes REST API for
metrics retrieval
Ambari
Server
Metrics
Monitor
Metrics
Collector
Host1
Sink(s)
3
Metrics
Monitor
Host1
Sink(s)Metrics
Monitor
Hosts
Sink(s)
1 2
4

Metrics Collector
Built using Hadoop technologies
Default uses local filesystem for
metrics storage (“embedded”) **
Local Filesystem **
HBase
ATS
Phoenix
** Tech Preview “distributed” storage option to use existing HDFS

Ambari Automated Rolling Upgrade
For HDP Stack

Rolling vs. In Place Upgrades
In Place Upgrades Upgrade Stack with one or more service disruptions. Explicit stop all services.
Rolling Upgrades
Ambari 2.0
Update Stack with minimized service disruption and degradation.

Upgrading the Stack with Ambari 2.0
Source HDP
Version
Target HDP
Versions
Method
HDP 2.0.x
HDP 2.0.x
HDP 2.1.x
HDP 2.2.x
In Place
HDP 2.1.x
HDP 2.1.x
HDP 2.2.x
In Place
HDP 2.2.x HDP 2.2.x Rolling NEW!!!

Rolling Upgrade Process
Pre-
requisites
Prepare
Rolling
Upgrade
Finalize
Rolling
Downgrade
Rollback
NOT Rolling. Shutdown all
services.

Process: Rolling Upgrade
HDP has a certified process
for Rolling Upgrades
Services are switched over to
new version in rolling fashion
ZooKeeper
Ranger
Core Masters
Core Slaves
Hive
Oozie
Falcon
Clients
Kafka
Knox
Storm
Slider
Flume
Finalize
HDFS, YARN, MR,
Tez, HBase, Pig.
Hive
HDFS
YARN
HBase

Process: Rolling Downgrade
ZooKeeper
Ranger
Core Masters
Core Slaves
Hive
Oozie
Falcon
Clients
Kafka
Knox
Storm
Slider
Flume
Downgrade
V
V
V
V
V
V
V
V
V
V
V
V
V
Finalize

Wizard Driven Experience Register Install
Perform
Upgrade
Finalize
With
verification
and validation

Process: Service Disruption by Component
Component Service Disruption
Zookeeper No Service Disruption
Ranger No Service Disruption
HDFS No Service Disruption
YARN No Service Disruption
HBase No Service Disruption
Hive No Service Disruption
Oozie No Service Disruption
Falcon Yes – Requires Stop/Start
Kafka Yes – Requires Stop/Start
Knox Yes – Requires Stop/Start
Storm Yes – Requires Stop/Start
Flume No Service Disruption
Slider applications Yes – Requires Stop/Start
Hue Yes – Requires Stop/Start
Accumulo Yes – Requires Stop/Start

Ambari Extensibility
Stacks, Blueprints and Views

Extensibility Features
•  To add new Services (ISV or otherwise) beyond HDP Stack
•  To customize a Stack for customer specific environments
•  To use Ambari for automating cluster installations
•  To share best practices on layout and cluster configuration
•  To extend and customize the Ambari Web UI
•  Add new capabilities, customize existing capabilities
Stacks
Blueprints
Views
Goal: Extend Ambari without hard-coding in Ambari

New in Ambari 2.0 - Blueprints Add Host
Add hosts to a cluster based on a
host group from a Blueprint
Add one or more hosts with a
single call
POST /api/v1/clusters/MyCluster/hosts
{
"blueprint" : "myblueprint",
"host_group" : "workers",
"host_name" : "c6403.ambari.apache.org"
}
POST /api/v1/clusters/MyCluster/hosts
[
{
},
{
}
]
https://issues.apache.org/jira/browse/AMBARI-8458

More on Ambari Extensibility
http://hortonworks.com/partners/learn/#ambari

Simplified Kerberos Setup

Quick Kerberos Overview
REALM
•  EXAMPLE.COM
Principals (Humans)
•  paul@EXAMPLE.COM
Service Principals (Services)
•  hbase@EXAMPLE.COM
•  hbase/r2u3s1.example.local@EXAMPLE.COM
Tickets
•  “paul@EXAMPLE.COM is authenticated and can access the HBASE service”
KDC – Key Distribution Center
•  Grant’s authenticated users tickets
Client
•  r1u2m1.example.com (.example.com maps to realm EXAMPLE.COM)
•  EXAMPLE.COM’s KDC is hosted on r1u2m3.example.com

KDC Implementation Options
•  Microsoft Active Directory
•  Users
•  Service Principals
•  MIT Kerberos
•  Users
•  Service Principals
•  MIT Kerberos + Microsoft Active Directory (Trust Relationship)
•  Users in Active Directory
•  Service Principals in MIT

What Ambari 2.0 will do
•  Step-by-Step wizard to setup Kerberos
•  Supports existing MIT KDC and Active Directory (AD) infrastructure
•  Deploys and manages Kerberos Clients, and configuration
•  First Time Setup as well as New Service/Host/Component
•  Automated creation of principals
•  Automated generation of keytabs
•  Automated distribution of keytabs
•  Support for regeneration and distribution of keytabs

Prerequisites
Category Requirements
General
•  Ambari Server must be part of cluster
•  Ambari Server and all hosts must have JCE installed
•  Ambari Server and all hosts must have network access to the KDC
KDC Admin
•  KDC admin account credentials are on-hand
•  !!! Ambari does not retain KDC admin credentials !!!
Active Directory
•  Security LDAP (LDAPS) connectivity has been configured
•  User container for principals has been created and is on-hand
•  Admin account has delegated control of “Create, delete and manage
user accounts”

Terminology
Term Definition
Service Principals Principals required for HDP Service Components.
Ambari Principals
Headless principals used by Ambari to perform “smoke tests” and “health
alert checks”.
KDC Admin Account
An administrative account that will be used by Ambari to create principals
and generate keytabs in KDC.

Principal and Keytab Generation and Distribution
1.  User provides KDC Admin
Account credentials to Ambari
2.  Ambari connects to KDC, creates
principals (Service and Ambari)
needed for cluster
3.  Ambari generates keytabs for the
principals
4.  Ambari distributes keytabs to
Ambari Server and cluster hosts
5.  Ambari discards the KDC Admin
Account credentials
Ambari
Server KDC
1 2
4
3
5
HDP
Cluster

Ambari + Service Keytab Files
Ambari
Server
HDP
Cluster
Hosts
Keytabs for
Ambari
Principals
Keytabs for
Service +
Ambari
Principals
KDC
Service Principals
Ambari Principals
Ambari and
Service
Principals

Wizard Driven and Automated

KDC and KDC Admin Information

Customizable Principal Attributes

Kerberos Clients
Ambari installs Kerberos clients
on cluster hosts
Optional to not have Ambari
manage krb5.conf client config
OS Client
RHEL/CentOS/OEL krb5-workstation
SLES 11 krb5-client
Ubuntu 12 krb5-user, krb5-config

Configure Ambari and Service Identities

Post-Kerberos Scenarios
Ambari does not retain KDC admin credentials
User is prompted for KDC Admin credentials:
•  Add/Delete Host
•  Add Service
•  Add/Delete Component
•  Regenerate Keytabs
•  Disable Kerberos

Security Lab

Security today in Hadoop with HDP
Authorization
Restrict access to
explicit data
Audit
Understand who
did what
Data Protection
Encrypt data at
rest & in motion
•  Kerberos in native
Apache Hadoop
•  HTTP/REST API
Secured with
Apache Knox
Gateway
Authentication
Who am I/prove it?
•  Wire encryption
in Hadoop
•  Orchestrated
encryption with
partner tools
•  HDFS, Hive and
Hbase, Storm
and Knox
•  Fine grain
access control
•  Centralized
audit reporting
•  Policy and
access history
HDP2.1
Ranger
Centralized Security Administration
More on Security: http://hortonworks.com/partners/learn/#secure

Ambari 2.0 Security Lab Steps
•  Detailed steps available at: http://bit.ly/1J4IbIs
•  Install Ambari server and agents
•  Deploy custom Ambari service folders for OpenLDAP, KDC/kadmin, NSLCD
•  Use blueprint/API to provision a minimal Hadoop cluster with custom services
•  Use Add service wizard to also install Hive
•  Configure Ambari to sync/recognize business users in OpenLDAP
•  Authentication: Run Ambari security wizard to enable kerberos using KDC/kadmin service
•  Install Ranger as Ambari service and configure it to recognize LDAP users
•  Authorization/Audit: Setup HDFS/Hive plugins to allow users to set policies to control access
and audit consumption

Hortonworks technical workshop operations with ambari

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Hortonworks technical workshop operations with ambari

Similar to Hortonworks technical workshop operations with ambari (20)

More from Hortonworks

More from Hortonworks (20)

Recently uploaded

Recently uploaded (20)

Hortonworks technical workshop operations with ambari