1. PDD-4932 : IBM Quick Deployer
An Automated CLM/CE Installation and Configuration Utility
Clare Carty
Senior Manager, DevOps Transformation and Enablement
IBM Watson Internet of Things Continuous Engineering
Thomas Piccoli
Software Architect, DevOps Transformation and Enablement
IBM Watson Internet of Things Continuous Engineering
2. Please Note:
1
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole
discretion.
• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in
making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any
material, code or functionality. Information about potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual
throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
3. Agenda
• CLM Devops Pipeline Overview
• IBM Quick Deployer Overview
• IBM Quick Deployer with UrbanCode Deploy
• Questions
2
5. CLM DevOps Pipeline
4
Develop Build (multiple per day) Manual Test (daily but not
completed in a day)
Function Test
Performance Test
System Test
Build
Unit
Test
Automated
Tests
BVTs
Performance
Functional
System Tests
Develop
Staging (daily)
Production on
Jazz.net
(end of each
sprint)
Staging
Environment
Production
Environment
Rational
Jazz Build
Rational
Collaborative
Lifecycle
Management
JUnit
IBM
UrbanCode
Deploy
CSPF
IBM UrbanCode Deploy
Cloud Env
IBM UrbanCode Deploy
Cloud Env
Continuous Integration and Test Continuous Production
And System Test
IBM Quick
Deployer
6. The DevOps pipeline runs on every CLM build
5
The Continuous Test pipeline
runs as part of each ALM build.
The Quick Deployer automation
deploys the build onto VM images in
the cloud and runs the tests in docker
containers.
Test results are reported out to an RTC
dashboard and issues reported as defects.
These can either be test issues (e.g UI change
broke the test) or product issues (e.g test fails
due to product defect).
• There are approx. 20 end to end scenarios run in the DevOps pipeline, comprising more than 2000 individual functional tests.
• Each end to end scenario is run in a departmental (4-VM) CLM topology.
• All scenarios run in parallel, deploying one topology per scenario, totaling approx. 100 VMs per pipeline run.
7. ALM Build runs the Pipeline
6
The ALM.all build is an orchestrated build
• Builds all the products
• Also has a set of properties files which
include lists of automated tests
– For example: cspf_60x-pipeline-
scenarios.properties
– includes a block for each test which
calls on a build definition called
run.cspf.tests.vl which uses Quick
Deployer
– Deploys a CLM topology for each
test so all tests run in parallel.
8. UCD Run Install Applications Process
7
The Install Applications process can be
watched live as it runs – each stage reports
progress and a result
9. Applications Deployed
8
• Once the process has completed,
the applications are setup and
ready to use.
• In the CLM DevOps pipeline, we
use the newly deployed systems
for Web UI automated testing.
11. IBM Quick Deployer
10
• IBM Quick Deployer automates installing and configuring IBM’s
Collaborative Lifecycle Management (CLM) and IoT Continuous
Engineering (CE) solutions version 6.0.1 and CLM version 6.0 on
Red Hat Linux 6.6+ using IBM UrbanCode Deploy
• Developed and used internally by the ALM DevOps pipeline team
• Deploys CLM and CE on user-supplied VMs, either locally or in
the cloud
• Installs and configures all required middleware, including IBM
WebSphere Application Server, IBM HTTP Server and IBM DB2
database server
• Supports the standard Jazz deployment topologies; however,
because it uses an UrbanCode Deploy tagging mechanism,
applications can be deployed across multiple servers in any valid
distribution
• Supports new installations: upgrade support is being considered
for a future a release
• Now available as an unsupported utility that can be downloaded
from the Jazz.net Deployment wiki under a “Non-warranted
Program” license
• Documentation available on the Jazz.net Deployment wiki
12. Background
• Many client requests for automated installation – proof of concept with Raytheon laid the
groundwork
• We already had automation internally but it was specific to topology and embedded with internal
data and settings
• Refactored our automation to exploit the UCD tagging mechanism and to remove wired-in internal
dependencies
• Result:
– Same code for internal and external use
– Install all required middleware to accommodate different deployment platforms
– Parameter driven user, database, LDAP and application configuration
– Platform agnostic, requires bare RHEL 6.6+ OS with a UCD agent installed
– Expanded error checking and reporting to increase usability
– First test platform was our ALM DevOps pipeline (i.e. used in-house)
– Reduced the number of UCD applications from 4 to 1 (CLM D4, CLM E1, SSE E1, SSE D4 QD 1.1)
– Provided us a flexible mechanism for describing topologies and easily switching between deployment platforms
11
13. Quick Deployer v1.0 Highlights
• Installs and configures CLM version 6.0 on Red Hat Linux 6.6+
• Deploys onto user-supplied VMs, either locally or in the cloud
• Installs and configures all required middleware, including WebSphere Application Server, HTTP
Server, and DB2 database server
• Connects to a user-provided LDAP server
• Supports the standard Jazz deployment topologies
– Some additional manual work needed to support SSO for distributed topologies
• Supports new installations
– upgrade support is being considered for a future release
• Single instance of each Jazz application per topology is currently supported
• Developed and used internally by the ALM DevOps pipeline team
• Available as an unsupported utility that can be downloaded from Jazz.net
• Fully documented on the Jazz.net Deployment wiki
12
14. Quick Deployer v1.1 Highlights
• Supports CLM and CE version 6.0.1 and CLM version 6.0 on Red Hat Linux 6.6+
• Added automated WAS SSO configuration so users will not have to provide credentials more than
once on a distributed topology
• Added support for Rhapsody DM
• Added limited support for CLM iFix installation (DM iFix support is being considered for a future
release)
• Provided a utility that allows the user to permanently install defaults eliminating the requirement to run
the change default processes on each UCD environment
– LDAP parameters
– User parameters
– DB2 parameters
• Resolved a few issues found by IBM technical sales teams
– Use OS hostname instead of UCD agent hostname
– Set OS specific line endings dynamically at run time
13
15. Quick Deployer System Requirements
• Current release of IBM UrbanCode Deploy installed (internal testing done with v6.1.1.6)
• User with UrbanCode Deploy admin privileges to import and configure the Quick Deployer application
• Customer provisioned VMs that meet documented system requirements for content, version and size
– RHEL 6.6+ installed
– Python 2.6.6+ is installed
– UrbanCode Deploy agent installed as root (all scripts run as root) …
• Customer provisioned VMs that support snapshots (QD does not support uninstall or restart)
• Customer created blueprints or manually constructed environments with components mapped and application tags applied
• Connection information for a customer supplied LDAP server
• Connection information for a customer supplied database server if used
• DB2, LDAP, WAS and root user credentials
• Customer is responsible for obtaining all required middleware and application installation media
• Connection information for customer maintained Network File Share that contains the middleware and application installation
media
• The full list of the latest site and system requirements is documented here
https://jazz.net/wiki/bin/view/Deployment/IBMQuickDeployerSystemRequirements
14
16. Quick Deployer Limitations
• Only new installations are supported: upgrade support is being considered for a future release
• Only Red Hat 6.6+ is supported
• Only DB2 10.5.0.3 is supported
• Only WAS 8.5.5.4 is supported
• Use of a reverse proxy and an LDAP server is not optional
• LDAP support is limited to Tivoli Directory Server
• In general, the process cannot be restarted (create a snapshot of the VMs before running any UrbanCode Deploy
process)
• Does not support uninstall (again suggest snapshots be used)
• Only one UCD environment can be connected to a customer provided database server; otherwise, name conflicts
and UrbanCode Deploy process concurrency issues can occur
• Does not support multiple instances of any Jazz application
• The context root of the Jazz application's URL cannot be changed from the default
• All Jazz application tags are expected to match the context route of its URL
• The UrbanCode Deploy agent must run as the root user, therefore, all scripts will also run as the root user
15
17. Quick Deployer Known Issues
• In CLM 6.0 scripted JTS Setup fails intermittently when attempting to register TRS providers with LQE
• DB2 user name restrictions are not enforced by the Change Default User Parameters process
• In QD 1.0 the user must run the change default processes on each UCD environment. QD 1.1 introduced a utility that
can be used to eliminate this requirement
• ${p:OSUserGroup} must exist prior to creating ${p:OSUserName}
• The Rational Requirements Management (RM) Converter application that renders the read-only view of RM graphical
artifacts is installed but not configured. There was an issue discovered when deployed on RHEL 7+ so the war file will
be present on the server but will not be configured in the WebSphere Application Server. Tech note Requirements
Management (RM) Converter application configuration and troubleshooting guide contains detailed configuration
instructions
• dcc files were added to CLM 6.0 iFix006 and CLM 6.0.1 iFix001, the automation does not install them so you’ll need to
follow the ifix readme for instructions
• Under very specific circumstances the DB2 database server can be shutdown after running a UCD process
• If DM is deployed on a server by itself then CLM ifixes will fail to install
• On RHEL 7.x the DB2, WebSphere and IBM Http Server processes will not automatically start after reboot. The system
and service managers have changed in RHEL 7.x. Please follow the Red Hat administration guide to enable these
services to start automatically following a reboot
16
18. Quick Deployer Package Content
• The QD installation package is delivered as a single compressed file and can be downloaded from the wiki at QD v1.1 Package
• The Quick Deployer package contains the following files
– UrbanCode Deploy application JSON file
• Rational_QD_601_vyyyymmdd-hhmm.json
– UrbanCode Deploy component scripts
• Rational_QD_Application_601_yyyymmdd-hhmm_artifacts.zip
• Rational_QD_ApplicationServer_601_yyyymmdd-hhmm_artifacts.zip
• Rational_QD_Database_601_yyyymmdd-hhmm_artifacts.zip
• Rational_QD_InstallationManager_601_yyyymmdd-hhmm_artifacts.zip
• Rational_QD_SystemPre-Requisite_601_yyyymmdd-hhmm_artifacts.zip
– UrbanCode Deploy component version create and update scripts
• versionUnzip.sh
• versionImport.sh
• versionUpdate.sh (new in QD v1.1)
– Urban Code Deploy tag create scripts
• tagsCreate.json
• tagsCreate.sh
– License files
17
19. Quick Deployer Installation Media
• There are two options for managing the installation media, local and NFS
• If local is specified the system expects the installers to be present on the local drive of each target VM
• If NFS is specified the system expects the installers to be located on a shared server accessible over
the network
• The user must configure the settings for each of these options to match their environment
• Quick Deployer v1.1 media folder content
– CLM600
– CLM601
– CE601
– CLMiFix/CLM_60_iFix005.zip
– CLMiFix/CLM_601_iFix001.zip
– db2/10.5.0.3
– IM/1821
– WAS/8.5.5
– WAS/8.5.5.4
18
20. Quick Deployer Local Install
19
• The QD installation package
– Agent tag definitions delivered as JSON with an install utility
– UCD component scripts delivered as compressed files with an install utility
– UCD Application delivered as a JSON file imported by the UCD administrator
• Customer acquired installation media
Scripts
as Zip
UCD
JSON
Import
UCD
JSON
Code
station
UCD
Server
UCD
API
agents
agentsUCD
Agent
Tags
JSON
NFS
Media
Server
21. Quick Deployer Future
Future development being considered, in no particular order:
• Capabilities
– Upgrade
• Systems originally deployed by Quick Deployer
• From what version to what version
• Middleware
• CLM/CE
– Middleware version selection
– iFix automation
– Multiple application instances
• Operating System and Middleware
– Windows
– Oracle
– Liberty
• …
20
23. vLaunch
Template
UCD
Blueprint
UCD Agent
Prototype**
Application
Process
UCD
Component
code
Quick Deployer in the Pipeline Process
22
RTC Pipeline
Build
UCD Server
vLaunch*
UCD
Environment
RHEL VM
RHEL VM
RHEL VMRHEL VM
UCD Agent
Install
Applications
*vLaunch – a VM provider
**Each VM hostname is set to the UCD environment name plus a suffix. The UCD agent
name is set to the VM hostname. The UCD agent prototype will cause the environment to
automatically connect to the agents when they register with the UCD server .
WAS - JTS,
CCM, QM, RM
DB2DB2
IHS
WAS - JTS,
CCM, QM, RM
WAS- LQE
ALM Product
Build
24. UCD Resource Templates - Tagging
23
• CLM E1
• CLM D4
• Pipeline
• Tagging allows control of what’s installed and where
• QD supports any valid topology and enforces rules to ensure a functional system is deployed
25. UCD Resource Templates – Agent Prototypes
24
• Agent Prototypes represent target VMs
• Agent name pattern is the UCD
environment name plus a fixed suffix
• The suffix matches the agent prototype’s
parent resource group
• The group is used to help identify the
agent when viewed in a UCD blueprint
• The pipeline automation makes sure the
agent names match the prototype
patterns
• UCD will match registered agent names
to the pattern specified on the prototypes
of the environment allowing it to be
connected to the correct target VMs
26. UCD Resource Templates - Components
25
• The UCD components are mapped to the agents
• The mapping allows additional control over which
process steps are executed on an agent/VM
• We chose to organize components by capability
– Database – install and configure DB2
– Installation Manager – install and configure IM
– Application – install and configure Jazz applictions
– Application server – Install and configure WAS and IHS
– System pre-reqs - performs common functions and
system validation
• Each component contains a set of scripts
• At runtime the component scripts are downloaded to
the target machine and a component process
describes how they are executed
• The component mapping is based on the role to be
provided by the target machine
• Pipeline
27. UCD Application – Blueprints
26
• An application is a container used
to manage a set of environments,
components, blueprints and
processes
• Blueprints can be derived from
resource templates
• We looked at the template earlier
because the agent prototype
pattern is not visible on a
blueprint so must be defined on
the template
• Blueprints and templates provide
a way for the user to define
reusable patterns
• We support many blueprints, the
tagging mechanism allows us to
support all of them with one set of
scripts and one UCD process
• The pipeline uses blueprint
vlaunch-dept-601 to create
environments
28. UCD Application – Environments
27
• Environments are the
bridge between UCD and
the target system, the
environment allows the
user to define what gets
installed on each target
machine
• Environments can be
constructed manually or
be derived from a
blueprint.
• This environment was
derived from the UCD
pipeline blueprint
• Notice the agent names
are the environment name
plus a suffix because we
defined this pattern on the
resource template from
which this blueprint was
derived
29. UCD Application – Components
28
• Components are
used to organize
the automation
scripts by capability
• Scripts are
managed in RTC
and published to
UCD as component
versions
• Components can be
used by more than
one application
30. UCD Application – Processes
29
• Application
processes define
the sequence of
steps required to
perform the desired
action
• Application process
steps typically
invoke component
processes
31. UCD Install Applications Process Design Part 1
30
• The external Install
Applications process is
made up of five phases
– Preparation
– Middleware
installation: DB, IM,
IHS, WAS
– Jazz application
installation: jts, ccm,
… and optional CLM
iFix
– Continued …
32. UCD Install Applications Process Design Part 2
31
– …
– Configuration
– Optional steps – jts setup, ldap
import, MTM deploy
• Each of these steps is further
described by a component
process which is defined in the
referenced component
• Each component process is only
mentioned once because the
step is run on the targets that
have the owning component
mapped
• The limit to tag process step
property also provides further
control over the steps that get
executed
• The default is to run each step in
parallel so that some steps can
be run at the same time on
different VMs
33. Run the Install Applications Process
• Select a process
• Set the process properties
– Install CLM 6.0.1 and fix
pack iFix001
– Run JTS setup
- Enable CM
- Set NTP clock
• The time it takes to
complete is dependent on
network bandwidth and the
number of VMs in the
topology deployed
32
34. Summary and Points to Ponder
• Where we’ve found value from Quick Deployer
– Rapidly deploy multiple environments to support our DevOps pipeline
– Standup test environments with new builds to validate defects
– Able to more quickly adapt to changing infrastructure
– Easy to test and debug on different topologies
• Where you might find value from Quick Deployer
– Quickly try out a new release of CLM/CE
– Test out a different topology
– Setup a training system
– Rapidly deploy when each project requires a standalone system
– ?
33
35. Recommended Session
DDD-5957: An Architectural Approach to Deploying Software at
Raytheon Using IBM UrbanCode Deploy
Tue, 23-Feb ( 01:15 PM – 02:15 PM )
Mandalay Ballroom K – Mandalay Bay SOUTH
Gray Ross, Raytheon
Sami Joueidi, IBM
34
38. Notices and Disclaimers Con’t.
37
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
39. Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.