IBM Interconnect 2015. Erin Schnabel & Chris Vignola STSM

1. © 2015 IBM Corporation
Production Deployment
Best Practices for
WebSphere Liberty Profile
Erin Schnabel
WebSphere Liberty Profile Lead Developer
schnabel@us.ibm.com
Christopher Vignola
WebSphere System Management Architect
cvignola@us.ibm.com

2. Topics
• Profile Configuration
• Topologies & Practices
• Application Deployment
• Management Configuration
• Numbers and Sizes
2

3. Profile
Configuration
3

4. Configuring Data Sources
• Use top-level libraries
• Allows multiple data sources and applications to share the
same class loader
<library id="DB2JCC4Lib">
<fileset dir="C:/DB2/java"
includes="db2jcc4.jar db2jcc_license_cisuz.jar"/>
</library>
<dataSource id="db2" jndiName="jdbc/sampleDB">
<jdbcDriver libraryRef="DB2JCC4Lib"/>
<properties.db2.jcc databaseName="SAMPLEDB"
serverName="localhost" portNumber="50000"/>
</dataSource>
<application location="myApp.ear">
<classloader commonLibraryRef="DB2JCC4Lib"/>
</application>
4

5. Configuring Data Sources: Connections
• Connection pooling and timeouts
• Reduce connectionTimeout to confirm that max pool size is
being exceeded
• Increase maxPoolSize if you see connection timeouts
• If you don’t need two phase commit:
• Use ConnectionPoolDataSource rather than XADataSource
• Data sources can enlist in a global transaction as a one-phase
resource even if they are not XA-capable.
5

6. Configuring Data Sources: Connection sharing
• Set isolation level property instead of programmatically
• Use isolationLevel property on datasource
• Set in resource reference binding/extension
• Declared isolation level allows for better matching/sharing of
connections
• Use containerAuthData instead of user/password
• Prevents applications using res-auth=Application from
accessing the data source with container credentials
6

7. Threading and Thread pools
• Auto-tuning thread pool
• Optimizes for executor throughput (measured every 1.5 sec)
• Dynamically adjusts between coreThreads and maxThreads
– Default coreThreads = (2*processorCores)
– Default maxThreads = MAX_INT
• Not usually necessary to tune the thread pool
• Some workloads may warrant increasing coreThreads
• Long-running / Outbound-to-self
• Measure performance before and after adjustment
7

8. Separate App and Admin HTTPS traffic
• Configure Virtual Hosts to isolate the application from other
(internal/administrative) traffic.
8
<httpEndpoint id=“appEndpoint” … />
<!– Restrict access to default_host: only accessible via default
endpoint -->
<virtualHost id=“default_host”
allowFromEndpointRef=“defaultHttpEndpoint”/>
<!-- define an application-specific virtual host -->
<virtualHost id=“applicationHost”
allowFromEndpointRef=“appEndpoint”>
<hostAlias>*:${app.http.port}</hostAlias>
<hostAlias>*:${app.https.port}</hostAlias>
</virtualHost>
<!-- configure plugin to route to the app-specific endpoint -->
<pluginConfiguration httpEndpointRef=“appEndpoint” />
server.xml:
<virtual-host name=”applicationHost" />
Ibm-web-bnd.xml
Virtual host
binding required in
the application

9. SSL Certificates and KeyStores
• Use only officially signed SSL certificates
• Use a separate keystore for inbound vs. outbound
• Encode or encrypt passwords in your configuration
<ssl id="defaultSSLConfig"
keyStoreRef="defaultKeyStore"
trustStoreRef="defaultTrustStore" />
<keyStore id="defaultKeyStore"
location="${server.config.dir}/key.jks"
type="JKS" password="{aes}..." />
<keyStore id="defaultKeyStore"
location="${server.config.dir}/trust.jks"
type="JKS" password="{aes}…" />
9

10. Security - General
• Harden your environment – extensive resources:
http://www.ibm.com/developerworks/websphere/techjournal/1210_lansche/1210_lansche.html
http://www.ibm.com/developerworks/websphere/techjournal/1303_lansche/1303_lansche.html
• Avoid vulnerabilities – keep service current.
• Register for support notifications at ibm.com/support .
10

11. Topologies
& Practices
11

12. Topologies
• Standalone
• collective Assisted Lifecycle
• Collective

13. When to Use Which Topology?
• Standalone – small, isolated, or situational deployments
• collective Assisted Lifecycle
• Small/Med deployments
• Existing admin skill
• multi-server QoS
• Collective
• Small/Med/Large deployments
• Lightweight environment
• Multi-server QoS
13

14. Universal Practices
• Server Package Deployment
• High Availability – Rule of 3
• Automated deployment
• Secure environment
14

15. QoS Practices (Optional)
• Dynamic Routing
• Auto-scaling
• z/OS integration
15

16. Gold Standard – HA Collective
16
Collective Controller
Replica Set
CC
CC
CC
• Include AdminCenter
• Use Dynamic Routing
• Use Auto-scaling
Machine Boundary
AppServer
AppServerLiberty
Profile
Per
App
IHS
IHS
IHS
Per
Collective

17. z/OS Integration
• Use read-only mount point for servicing runtime
• Use PROCs for running servers
• Use START/MODIFY/STOP commands for server lifecycle
• Use SAF Registry support
• Exploit platform integration as necessary:
• zosWLM
• zosTransaction
• zosLocalAdapters
17

18. Application
Deployment
18

19. Build & Deploy
• Use server package for deployment – 3 models
• Developers direct deploy
• Developers hand off app, admin packages/deploys
• Developer direct deploy w/packaging automation to ensure
“approved server config”
>> Establish build/deploy pipeline
• Automate – use script, Chef, Urbancode, etc
19

20. Example Build/Deploy Pipeline
20
Jenkins
Liberty

21. Server Package Deployment: Managing Config
21
$WLP_USER_DIR/MyServer/
server.xml
apps/
MyApp.war
deploy (apply target overrides)
$WLP_USER_DIR/MyServer/
server.xml
configDropins/
overrides/
prod.xml
apps/
MyApp.war

22. Server Package Update
• Update through build process
• Blue/Green deploy
• Dual install locations
• Ripple start (stop old/start new, JVM by JVM)
• Delete old instance at your convenience
22

23. Upgrade – 1 of 5
23
host1.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib

24. Upgrade – 2 of 5
24
host1.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host1.com
http/s: 9080/9443
status: STOPPED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STOPPED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib

25. Upgrade – 3 of 5
25
host1.com
http/s: 9080/9443
status: STOPPED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host1.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STOPPED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib

26. Upgrade – 4 of 5
26
host1.com
http/s: 9080/9443
status: STOPPED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STOPPED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host1.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib

27. Upgrade – 5 of 5
27
host1.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib

28. Management
Configuration
28

29. High Availability Services
• Collective scope
• Built-in to collective controller replica set
• Host scope (e.g. scalingMember-1.0 feature)
• Uses local port
29

30. Collective SSL
• Collective root, member root
• First controller establishes “true root”
• Must be copied to subsequent replicas
• Trust between
• Controllers and members
• Members and controllers
• Members on same host
• Certificates
• Signers
• Identity
30

31. Security
• Use external user registry – e.g.
• Ldap
• SAF
• Multiple registries allowed – e.g. as granular as per cluster.
• Collective Controller Replica set must use same registry.
31

32. Highly Available Collective Controller
• Three controllers minimum. Odd numbers only.
• Up to 2000 member per controller
Recommended: Max(Members/2000,3)
+1(for controller failure)
+1(for network partition)
• Use configDropIns directory – it’s replicated!
• Configure members with controller failover addresses.
32
CC
CC
CC

33. Dynamic Routing
• Requires IHS (or Apache).
• Use dynamicRouting-1.0 feature in controllers.
• Double-layer IHS to simplify firewall management.
• Terminate SSL at earliest opportunity.
33
CC
CC
CC
IHS
IHS
IHS

34. Auto-scaling
• Put scalingController-1.0 feature in at least 3 controllers.
• Set hostSingleton port for vertical scaling.
• Set min instances based on average demand.
34
AppServer
AppServer
AppServer
Elastic
Resources
CC
CC
CC

35. Auto-scaling Policy
• Leave headroom on max settings (e.g.
• CPU <= 90%
• Heap <= 90%
• Memory <=90%
• Include scaling policy in server package
• scaling-metadata.xml
35

36. Use Admin-metadata
• Owner
• Contacts
• Note
• Tags
• admin-metadata.xml – part of server package
• assignable to host, server, application, cluster, runtime
• There is also an API
• Used in AdminCenter to search and set views
36

37. Numbers and Sizes
37

38. 38
Liberty Profile Collective Size Design Considerations
• Collectives are design for large scale
• What limits the size of a WebSphere Collective?
• Breadth and currency for shared Information across controller JVMs
• Communication and coordination of shared information
• Product features scale differently with large collective size
• Most affected : collectiveController, scalingController
• It is possible to create a large topology collective
But will it function to your requirements?
• Collective works well with defaults settings. Some environments may
require tuning.

39. 39
When do you need multiple collectives?
• Isolation
• Development vs testing vs production
• Critical applications
• Backup site
• Lines of business
• Each funding area may have different policies for when to apply
fixes, when to upgrade
• Geography:
• Controllers can span data centers with qualifying* network and config
• Members can span data centers with tuning
• Large collectives require planning to avoid “urban sprawl”

40. 40
How Large a collective can I create?
• No hard limit – trust, but verify
• Lab tested:
• Tested 10,000 members
• 5 controllers
• 50 VMs
• ~200 members per VM
• Controller VMs: 20GB memory+6 CPUs
• Member VMs: 64GB+16 CPUs

41. 41
How Many Host OS Instances per collective
• No Design Limit
• Typical large topology up to hundreds of hosts.
• Practical Limits
• Operations may take longer:
– Configurations, server deployment
• Notifications flowing back to collective controllers
• Load on controllers with concurrent operations
• 50 hosts per collective controller guideline

42. 42
How Many Application Servers per Host OS?
• Keep WAS JVMs completely within physical memory
• Allow for overhead
• Process footprint about 1.25 to 1.5X maximum heap size for 32 bit
heaps
• 1.6x to 1.8x for 64-bit heaps
• If App Server gets swapped out, add more memory or
else move to different host OS.
• Ensure sufficient CPU is allocated
• Especially for hypervisor
• Avoid CPU starvation

43. 43
How Many Applications Per Server
• Balance between resource usage and isolation
• With one application per server
• One bad application does not bring down all other applications
• Easier to tune each application in isolation
• With more than one application per server
• Less resources: cost of application server runtime amortized
across multiple applications
• Smaller topology to manage
• Configure as much isolation as you can afford
• If you have 300 applications clustered on 3 nodes
– Complete isolation  900 JVMs
– Complete sharing  3 JVMs

44. 44
System Management Best Practices
• Set and track your performance goals
• Measure performance of your most commonly used operations
• Track changes as you increase collective size
• Track changes over time to identify new issues
• Use Scripting (or your own Java framework)
• Automatable, repeatable, testable
• Don’t overload collective controllers
• Give enough memory + CPU
• Run AdminCenter in multiple controllers to spread load
• Don’t co-locate with resource intensive processes, e.g., application
servers with heavy load

45. Wrap Up

46. Covered
• Topologies & Practices
• Profile Configuration
• Application Deployment
• Management Configuration
• Numbers and Sizes
46

47. Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM
shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,
EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF
THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT
OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the
agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not
necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither
intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law.

48. Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents, copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document
Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,
ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,
PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,
urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of
International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

49. Thank You
Your Feedback is
Important!
Access the InterConnect 2015
Conference CONNECT Attendee
Portal to complete your session
surveys from your smartphone,
laptop or conference kiosk.