5. Stack Terminologies
Term Definition Examples
STACK Defines a set of Services, where to obtain
the software packages and how to manage
the lifecycle.
HDP-2.3, HDP-2.2
SERVICE Defines the Components that make-up the
service.
HDFS, YARN
COMPONENT The building-blocks of a Service, that adhere
to a certain lifecycle.
NAMENODE, DATANODE,
OOZIE_SERVER
CATEGORY The category of Component. MASTER, SLAVE, CLIENT
REPO Repository metadata where the artifacts
reside
http://public-repo-
1.hortonworks.com/HDP/centos6/2
.x/GA/2.3.0.0
6. Ambari Stacks
• Stacks define Services + Repo
• What is a stack composed of and where to get the bits
• Each service has a definition
• What components are part of the Service
• Each service has defined lifecycle commands
• start, stop, status, install, configure
• Lifecycle is controlled via command scripts
Ambari Server
Stack
Service
Definitions
Command
Scripts
xml python
Ambari Agents
Repos
7. Ambari Stack Framework
• Ambari’s extensible stack framework allows different stack vendors to
create their own custom stacks.
9. Current State
• Extensible Framework
• Provides ability for onboarding new Hadoop distributions
• Key role in avoiding vendor lock-in
• Stack driven
• Service lifecycle management
• Custom service commands
• UI layouts and themes
• Upgrade packs
• Stack advisors
• Metrics
• Kerberos
• Stack Inheritance
• Code reuse between stack versions
• Common Service Definitions
• Code reuse between multiple stacks
• ODPi Operations Spec
10. Limitations – Common Services
• AMBARI-7201 focused on adding notion of common services
• Service definitions scripts were not refactored
• Common service scripts still tightly coupled with HDP distribution
• Hardcoded references for stack name (HDP), stack version checks
(HDP-2.2+), stack tools (hdp-select, conf-select), installation paths
(/usr/hdp), method names (format_hdp_stack_version) etc.
11. Limitations – Custom Services
• Rudimentary approach to add third party custom services Ambari Wiki
• No release vehicle for building, releasing and deploying custom
services
• No service level extension points for stack features
• Role command order
• Stack advisor
• Upgrade packs
• Repositories
• Custom service definitions not self-contained and require hacks inside
stack definitions
12. Limitations – Release Management
• Ambari core and stack definitions released together
• Bug fix in stack definition requires Ambari release
• New stack version requires Ambari release
• Stack releases and Ambari releases have to be coordinated
• Need to decouple stack definition releases from Ambari core release
14. Stack Featurization
• Provides a basic framework for refactoring stack-specific hardcoded logic in
common service definitions.
• Features defined at stack-level based on which execution logic in service
definition is triggered.
• All common service definitions have been stack featurized and HDP-specific
hardcodings have been removed.
• Stacks similar to HDP can now use common service definitions.
• EPIC: AMBARI-13363
Apache JIRA Description Target Release
AMBARI-15420 Generalize resource management library 2.4.0.0
AMBARI-13364 Parameterize stack information used by common services 2.4.0.0
AMBARI-15329 Remove HDP hardcoding 2.4.0.0
18. Service Level Extension Points
Apache JIRA Description Ambari Wiki Target Release
AMBARI- 15388 Upgrade Pack Extensions Ambari Wiki 2.4.0.0
AMBARI-15226 Stack Advisor Extensions Ambari Wiki 2.4.0.0
AMBARI-9363 Role command order Extensions Ambari Wiki 2.4.0.0
AMBARI-11268 Quick Links for custom services Ambari Wiki 2.4.0.0
AMBARI-15538 Service-specific repo definitions 2.X.X.X
• Facilitate integration of third-party custom services to a stack definition
• Self-contained service definitions without requiring changes to stack definition
• Upgrade Pack Extensions
• Service-level upgrade packs that specify how to upgrade the custom service
• Defines how to integrate into the stack upgrade pack
• Stack Advisor Extensions
• Service Advisors (service_advisor.py) define recommendations and validations for the service
• Stack framework extends stack advisors with service advisors
• Role Command Order
• Define role command order at service level
• Stack framework merges all role command orders to create dependencies
• Quick Links
• Define quick links (quicklinks.json) in the service definition that is used by Ambari Web UI
• No hardcoded quick links in the Ambari Web UI
• Repo Definitions
• Service specific repositories so that third party artifacts can reside on their own repositories
• Under development
EPIC: AMBARI-15537
19. Upgrade Pack Service Extension
Without Upgrade Pack Extensions:
- Upgrade logic defined at stack level
- Custom services need to modify the stack's upgrade-packs in order to integrate themselves into the cluster
With Upgrade Pack Extensions:
- Each service can define upgrade-packs XML files placed in the service's upgrades/ folder
ambari-server/src/test/resources/stacks/HDP/2.0.5/services/HDFS/upgrades/HDP/2.2.0/upgrade_test_15388.xml
<target>2.4.*</target>
<target-stack>HDP-2.4.0</target-stack>
<type>ROLLING</type>
<prerequisite-checks>
<check>org.apache.ambari.server.checks.FooCheck</check>
</prerequisite-checks>
<order>
<group xsi:type="cluster" name="PRE_CLUSTER" title="Pre {{direction.text.proper}}">
<add-after-group-entry>HDFS</add-after-group-entry>
<execute-stage service="FOO" component="BAR" title="Backup FOO">
<task xsi:type="manual">
<message>Back FOO up.</message>
</task>
</execute-stage>
</group>
….
20. Service Advisor
Without Service Advisor:
- Stack advisor defined at stack level
- Custom services need to modify stack advisor files in order to recommend/validate dynamically service
configurations and the layout of the service on cluster
With Service Advisor:
- Each service can choose to define its own service advisor and explicitly extend parent's service-advisor script
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
def getServiceComponentLayoutValidations(self, services, hosts):
componentsListList = [service["components"] for service in services["services"]]
componentsList = [item["StackServiceComponents"] for sublist in componentsListList for item in sublist]
hawqMasterHosts = self.getHosts(componentsList, "HAWQMASTER")
hawqStandbyHosts = self.getHosts(componentsList, "HAWQSTANDBY")
hawqSegmentHosts = self.getHosts(componentsList, "HAWQSEGMENT")
datanodeHosts = self.getHosts(componentsList, "DATANODE")
# Generate WARNING if any HAWQSEGMENT is not colocated with a DATANODE
mismatchHosts = sorted(set(hawqSegmentHosts).symmetric_difference(set(datanodeHosts)))
if len(mismatchHosts) > 0:
hostsString = ', '.join(mismatchHosts)
message = "HAWQ Segment must be installed on all DataNodes. "
21. Quick Links
Without Quick Links Extensions:
- Hardcoded quick links in the Ambari Web UI
With Quick Links Extensions :
- A service can add a list of quick links to the Ambari web UI by adding predefined JSON format file to metainfo.xml
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/quicklinks/quicklinks.json
<quickLinksConfigurations>
<quickLinksConfiguration>
<fileName>quicklinks.json</fileName>
<default>true</default>
</quickLinksConfiguration>
</quickLinksConfigurations>
mapQuickLinks: function (finalJson, item){
if(!(item && item.ServiceInfo)) return;
var quickLinks = {
OOZIE: [19],
GANGLIA: [20],
STORM: [31],
FALCON: [32],
RANGER: [33],
SPARK: [34],
MY_CUSTOM_SERVICE: [35]
};
{
"name": "default",
"description": "default quick links configuration",
"configuration": {
"protocol":
{
"type":"http"
},
"links": [ {
"name": "hbase_master_ui",
"label": "HBase Master UI",
….. }
}, {
"name": "hbase_logs",
"label": "HBase Logs",
…
22. Stack Extensions
• A stack extension is a collection of custom services which are packaged
together.
• Provides a REST API for installing extensions.
• Extensions are staged at /var/lib/ambari-server/resources/extensions
• After installing extensions requires explicit linking of extensions with the
supported stack versions.
• Custom services contained in the extension may be added to the cluster like
any other service in the stack.
• EPIC: AMBARI-12885
25. Ambari Management Packs
• Release artifact to bundle stack definitions, service definitions, add-on service
definitions, views etc.
• Decouple stack definition releases from Ambari core release.
• Also provide a release vehicle for add-on services.
• Released as tarballs but contains metadata that describes applicability and
contents of the management pack
• Staging Location: /var/lib/ambari-server/resources/mpacks
• Final stack definition can be an overlay of multiple management packs.
• EPIC: AMBARI-14854
• Ambari Wiki
• Target Release: 2.4.0.0
29. Future Goals
• Service level repos
• Management Pack++
• Short Term Goals (Ambari 2.4.0.0)
o Release vehicle for stacks
o HDP management pack, IOP management pack
o Release vehicle for add-on services (custom services)
o Microsoft-R management pack
o Retrofit in existing stack processing infrastructure
o Command line to update stack and service definitions
• Long Term Goals (Ambari 2.4+)
o Release HDP stacks as mpacks
o Build management pack processing infrastructure
o Dynamic creation of stack definitions by processing mpacks
o Rest API for adding/removing mpacks
• UI wizards stack driven
