Troubleshooting Connections Issues

Social Connections 11 Chicago, June 1-2 2017
Two wrongs don’t make a right –
Troubleshooting Connections
Terri Warren, panagenda
@TerriLLBean
Nico Meisenzahl, panagenda
@nmeisenzahl

PLATINUM SPONSORS
GOLD SPONSORS
SILVER SPONSORS

Nico Meisenzahl
• Consultant at panagenda
• IBM Connections since version 3.0 / 2010
• IBM Notes / Domino since 2008
• Focusing in ICS
• Deployment & consulting
• Optimization and migration
• “panagendian” since 2016
• IBM Champion
@nmeisenzahl
linkedin.com/in/nicomeisenzahl
meisenzahl.org
nico.meisenzahl
+49 170 7355081
nico.meisenzahl@panagenda.com

Terri Warren
@TerriLLBean
linkedin.com/in/terrillbean
live:TerriLLBean
terri.warren@panagenda.com
• Consultant at panagenda (since 2016)
• In my recent past I was a developer on IBM
Connections, IBM Domino
(Directory/Security) and an SRE on Watson
• Frequent presenter at IBM conferences
focusing in ICS
• Deployment & consulting
• Optimization and migration

Agenda
• Howto: Troubleshooting
• Troubleshooting…
• Connections itself
• Backend like Spectrum CfC, DB2, TDI,
WebSphere, Directory, SSO
• Optional add-ons like Orient Me, Docs, FEB,
Cognos, CCM
• Tools

Howto: Troubleshooting

Reproduce the error
• Reproducible and/or periodically?
• A sequence error?
• Client-side or server-side problem?
• Analyze the root cause

Be aware of the big picture
• Client-side problems
• Debug in different Browsers (IE, FF, Chrome)
• Do not use a server IE
• Server-side: IBM Connections is based on
many components
• debug on “high level” first
• get an overview which backend service is causing
the error

Configuration changes
• Changes in…
• Connections configuration
• Backend (WebSphere, Database, HTTP, CfC)
• Firewall or network
• OS, hardware or VM
Tip: Even the smallest configuration change can
have big consequences!

Analyze log files and browser
• Analyse log files
• Atom.io, Notepad++ or less/tail
• Baretail or tail –f
• kubectl logs
• ELK stack
• Tools for client-side problems
• Firebug or Developer-Tools
• BurpSuite or Fiddler

Analyze root cause
• Find a hint inside the log
• Network timeout or DNS
• SQL errors
• LDAP errors
• Syntax errors in configuration files „xxx-config.xml“
• Error stack
Tip: In a clustered environment, start and analyze only one
Node (if possible)

Find support
• Knowledge Center https://goo.gl/up6cxG
Troubleshooting Tips https://goo.gl/IaVinx
• IBM Connections Forum http://goo.gl/CVvQCU
• Community Blogs and/or Chats
• Fix Central
• PMR

Troubleshooting

WebSphere Application Server logs
• SystemOut.log
• SystemErr.log
• trace.log (if tracing is enabled)
• Log path:
• <wasroot>/profiles/<profilename>/logs/<servername>/

Analyze WAS log files
• Time stap: 24h time stamp with milli-seconds
• Thread id: eight character hexadecimal value
• Short name: typically java class name
• Event type: one character only (E, W, I,…)
• Message identifier: String based on component
• Message: Some information

WAS Event types
• F - Fatal message
• E - Error message
• W - Warning message
• A - Audit message
• I - Informational message
• C - Configuration message
• D - Detail message
• O - Messages that are written directly to System.out by an application
• R - Messages that are written directly to System.err by an application
• Z - Place holder to indicate type was not recognized

WAS Message identifier
• Prefix by Application or Server (CLFRW)
• Specific application code (0042)
• Event Type (I)

Read trace stack
• First line displays key information
• “Caused by” displays root cause

Environment information
• First log lines on server startup
• WebSphere version
• OS version, Process id
• Installation path
• Java version

Enable tracing
• Enable tracing using ISC
• Runtime or configuration
only
• Define tracing based on
• App prefix / error stack
• Must gather (PMR)

Search issues
• http(s)://<fqdn>/search/serverStatus
• Display index, seedlists, log information
• Data is displayed for one node only
• Using node fqdn to access different nodes
Tip: This will create persistent files on your
hard disk!

Debug Search
• Search queries (runtime)
• com.ibm.connections.search.index.searching.*=all
• Crawling & seedlists
• com.ibm.connections.search.index.indexing.*=all:
com.ibm.connections.search.seedlist.*=all:
com.ibm.connections.httpClient.*=all

Recreate Search Index
• SearchService.startBackgroundIndex()
• Crawls seedlists
• Extracts the file content
• Create index
Tip: Use „all_configured“ to index all apps

CLFRW0394E: Search indexing of services
...
• Search index not ready
• interruption at index creation
• CLFRW0283E: Search has encountered a problem
while crawling
• CLFRW0027E: Error Indexing component <app>
for search
• INDEX.READY file not present
• Recreate and enable tracing

Database connections
• Check datasouces
• ISC – Resources – JDBC – Data sources
• Check logs for more information
• DB2 server log
• <instanceroot>/sqllib/db2dump/
• db2diag.log
• db2diag.xxx.log (log rotation, you should enable this!)
Tip: Oracle users have password expiration enabled
by default!

HTTP Server (IHS & Plugins) logs
• IBM HTTP Server
• <installroot>/logs/
• error_log
• access_log
• based on configuration
• WebSphere AppServer Plugins
• <installroot>/logs/<webserver>/http_plugin.log

HTTP 404 not found
• Outdated Plugin configuration
• Restart IHS
• WAS Plugin configuration issue
• http_plugin.log

HTTP 404 not found
• AppServer or App down
• Network issue
• http_plugin.log

HTTP 500 Internal Server Error
• Unexpected error
• http_plugin.log
• Configuration issue
• WAS Root certificate not trusted or missing
• SSL certificate expired

Debug wsadmin
• Enable trace within wsadmin session
• AdminControl.trace(‘com.ibm.*=all’)
• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout

Directory/Troubleshooting/ Tuning
• Waltz and Sonata overview
• Enabling Log parameters…
• Cache Tuning
• SSO Config Glitches
• CCM Troubleshooting

Connections Troubleshooting
• Waltz: "common directory services"
• Directory Service eXtension (DSX)
REST API.
• Sonata: common HttpClient
services through a RESTful service
• configurable authenticators that support
• SSO security tokens and cookies for
secure server-to-server communication.

Troubleshooting:
DSOutOfServiceExceptions
• Symptoms:
• Access Connections not possible
• DSX not working

Troubleshooting: Enable Logging
• Track all basic Waltz configuration settings and transactions.
• com.ibm.connections.directory.services.*=all
• Track all LDAP transactions in between Waltz & LDAP server(s).
• WaltzLDAPUsage=all
• Track all LDAP entry to be cached and hit from cache upon DN of LDAP entries.
• WaltzDNEntryCache=all
• Track all LDAP entry to be cached and hit from cache upon ID of LDAP entries.
• WaltzExactIDMatchCache=all
• Track all group membership (a list of groups) for a given user upon ID.
• WaltzGroupMembershipCache=all
• Track all members (a list of users) for a given
group upon ID.
• WaltzMemberExpansionCache=all
TIP: Save runtime changes to make changes persistent

• WAS VMM/WIM:
• com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all
• WAS SPNEGO:
• com.ibm.ws.security.spnego.*=all
• Apache commons HttpClient:
• org.apache.commons.httpclient.*=all
• WAS security:
• com.imb.ws.security.*=all:com.ibm.ws.security.policy.*=off;

• Basic Sonata configuration settings and transactions:
• com.ibm.connections.httpClient.*=all"
• Track the amount of HTTP(S) transactions.
• SonataHttpUsage=all
• Track headers received for all HTTP(S) transactions.
• SonataHttpHeader=all
• Track all bodies received for all HTTP(S) transactions.
• SonataHttpBody=all

EJPVJ9284E: Unable to get the groups
from the directory for the user…
• User was not able to access Connections
anymore
• WAS LDAP bind user had no read access
to one of the groups the user was member

SSO Configuration Glitches:
• Warning indicating SSO config is missing or
incomplete enabling application security:
• Ensure you filled
in your domain
Tip: Connections CRs sometimes
resets SSO domain

SSO Configuration Glitches:
• LtpaToken Cookies:
• More secure- recommended to use version 2 LTPA cookies
In other words- interoperability mode is left as disabled by
default.
Only enable interoperability mode if
some app servers are still relying on
LTPA version 1 and need to interoperate.
Connections Security supports custom
LTPA cookie names since 4.5 CR1
Sets com.ibm.websphere.security.
customSSOCookieName custom
properties (verify 3rd party products
Portal/Domino can handle it)

Waltz Cache: Changing Group
Membership/permissions remains in Waltz Cache
• Issue: Administrator changes access or Group
membership for a user. That user does not get
immediate access.
• Cache: Connection's Directory Services keeps a “list” of
directory objects it has already searched for performance
reasons
● Improves performance of connections services
● Reduces load on remote directory servers
● Cache is based on a timing mechanism
● Cache is "flushed and renewed" on a 12 hour time schedule

Tuning the Cache: Continued
• How does this affect Group Membership?
● Group membership exists in the cache for 12 hours
● Operations such as renaming, deleting, or updating groups remain
in the cache for that time
• Configuration parameter for cache timing
mechanism
● Enables Connections administrators to flush the cache in
configurable time increments
● Set cache parameter via JVM arguments to flush them (we'll get to
that!)
● Default remains 12 hours

Cache Configuration: Setting the cache
flush parameters
• Setting the cache parameters via JVM configured in
WebSphere
● Google Setting generic JVM arguments in WebSphere Application
Server
• Time Values must be the same for all cache settings
• Adjust with care! Flushing the cache has implications!

IBM Connections Content Manager
(CCM): Troubleshooting
• During library creation, retrieving or modifying content:
• Calls are made between Communities, Profiles, and FileNet
 Issues with Filenet / Connections Profile or Community configuration can
adversely affect access to Libraries and CCM
 Logging from Connections components (previous slides) and FileNet essential
• SystemOut.log & trace.log – these are your friend
• FileNet core logging:
• <wasprofile>/<servername>/p8_server_error.log
• <wasprofile>/<servername>/p8_server_trace.log
• <wasprofile>/<servername>/pesvr_system.log
• <wasprofile>/<servername>/pesvr_trace.log

CCM Troubleshooting: Filenet URLS
• Status about FileNet Collaboration Services:
• check the status page at http://<fqdn>/dm
 Navigator version and configuration
• Status about FileNet Content Engine
• check the status page at http://<fqdn>/FileNet/Engine
 Server status, version, sonata/waltz version
• Status about FileNet Content Engine Domain and Object Store
• check the status page at http://<fqdn>/P8CE/Health
 Health checks for authentication, stores and database

Debug CCM Widget
• Widget issues
• com.ibm.quickr.communitylibrary.*=all:
com.ibm.lconn.widgets.service.*=all:
com.ibm.lconn.widgets.actions.*=all
• Authentication issues (Remember those cool tips a few
slides back? )
• com.ibm.connections.directory.services.*=all:
com.ibm.connections.httpClient.*=all

Debug FileNet using ACCE
• ACCE tool: check if FileNet Content Engine can
authenticate using Connections and inspect other data
for errors:
http(s)://<fqdn>/acce

CCM: Enable logging for Connections and FileNet
using JVM Properties
• Download and copy the sample waltz.sonata.log4j. as log4j.xml:
• Linux: /<absolute_path_for_log4j_file>/log4j.xml
• Windows:<absolute_path_for_log4j_file>log4j.xml
• Add generic JVM properties
• -Dlog4j.configuration=file:<path>/log4j.xml
-DskipTLC=true
• Tells the Content Engine (CE) server to skip it’s tracing configuration
and writes to waltz.sonata.trace.log
Windows:
c:Program FilesIBMWebSphereAppServerprofilesAppSrv01
Linux: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01

CCM: Enable logging for Connections and FileNet
using JVM Properties
• Copy & customize sample log4j.xml
• <ContentEngineRoot>/config/samples
• 20 subsystems (db, engine, security, search,…)

TDI logs
• <tdisol>/log/ibmdi.log
• TDI log file
• <tdisol>/employee.*
• Files include all changed users (adds, update, delete, error, skip)
• <tdisol>/syncupdates/*
• Temporary files within the sync
• Including database dump and ldiff
• sync_updates_clean_temp_files=false (default: true)
• profiles_tdi.properties
Tip: Check the lock file

Analyze TDI logs
• Error code prefix
• CLFRN: Profile & User synchronization
• CTGDIS: TDI itself
• Error code suffix
• I, E, W, …

Debug TDI
• Profile & User synchronization
(<tdisol>/etc/profiles_tdi.properties)
• source_ldap_debug=true
• debug_update_profile=true
• debug_collect=true
• TDI issues (<tdisol>/etc/log4j.properties)
• log4j.rootCategory=DEBUG, Default

Cognos BI logs
• Cognos BI
• SystemOut.log & trace.log
• <installroot>/logs/cogserver.log
• Cognos Transformer
• <installroot>/logs/cogserver.log
• PowerCube build
• <installroot>/metricsmodel/trxschelog.log
• <userhome>/Transformer/Logs/*.log (win only)

BMT-MD-6003 No connection to the
data source …
• PowerCubes not created yet
• Check cronjobs or scheduled jobs
• trxschelog.log

Debug Cognos BI & Metrics
• Communication between Cognos BI and
Connections Metrics
• SonataHttpUsage=all:
SonataHttpHeader=all:
SonataHttpBody=all:
com.ibm.connections.httpClient.*=all:
com.ibm.connections.metrics.*=all
• Connections Metrics Servlet
• com.ibm.connections.metrics.cognos.servlet.*=all

Docs/Viewer logs & urls
• SystemOut.log & trace.log
• http(s):<fqdn>/vsanity/check
• http(s):<fqdn>/sanity/check?app=all&querytype=report
• http(s):<fqdn>/*/version

Debug LTPA between Domino & WAS
• Debug on Domino side (notes.ini)
• Debug_SSO_Trace_Level=2
• Webauth_verbose_trace=1
• WebSess_verbose_trace=1
• Debug_outfile=<logfilepath>
• Debug on WebSphere
• com.ibm.ws.security.ltpa.*=all

Debug Kerberos
• Configuration
• com.ibm.ws.security.spnego.*=all:
com.ibm.ws.security.*=all: com.ibm.issw.spnegoTAI.*=all:
com.ibm.security.krb5.*=all
• Runtime
• com.ibm.connections.httpClient.*=all:
com.ibm.connections.directory.services.*=all:
com.ibm.websphere.wim.*=all:
com.ibm.ws.wim.*=all
• Fiddler & BurpSuite

IBM Spectrum CfC stats
• https://fqdn:8443

Docker & Kubernetes status
• kubectl get pods (--all-namespaces)
• kubectl get services
• kubectl get deployment
• kubectl describe pods xxx
• docker ps
• docker info

Docker & Kubernetes status
• Persistent Volume status
• kubectl get pv,pvc

Orient Me logs
• Display application logs
• kubectl logs xxx
• -f
• --tail=20
• --since=1h
• Special tasks for MongoDB and Solr

MongoDB Troubleshooting
• Check persistent volumes
• Access logs
• kubectl logs xxx [mongo|mongo-sidecar]
• Get replica status
• kubectl exec -it mongo-0 -- mongo mongo-0.mongo:27017 --eval
'rs.status()’
• Check “health” status for all 3
members

Solr logs
• kubectl exec -it xxx -- cat /var/solr/logs/solr.log
• kubectl logs xxx
• only display deploment logs

Pods startup issues
• kubectl describe pod xxx

CfC logs
• docker logs -f kubelet
• docker logs -f calico

Docker tracing
• service docker stop
• docker daemon –debug
• journalctl -u docker.service

What to get more insights on Orient Me?

Tools

Analyze logs
• Analyze logs live
• Baretail
• tail –f
• View logs
• Atom.io, Notepad++
• less, tail
• ELK Stack
• Elasticsearch, Logstash, Kibana
• Small Docker deployment
• Kubectl (local client)

Analyze logs
• Kubectl (local client)

Analyze Client-side
• Browser
• Firebug / Developer Tools
• Intercepting proxies
• Fiddler
• Burp Suite
• VMs with different IE versions
• Without GPO
• https://www.modern.ie/en-us/virtualization-tools

Database Clients
• db2 command
• Dbeaver or IBM Datastudio
• Robomongo (MongoDB)
Tip: Database write access is not supported!

LDAP & Search
• LDAP
• ldapsearch command
• Softerra LDAP Browser
• Apache Directory Studio
• Search Index
• Luke (Lucence Index Toolbox)

Analyze Network
• Wireshark
• tcpdump

Q&A

Troubleshooting Connections Issues

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Troubleshooting Connections Issues

Similar to Troubleshooting Connections Issues (20)

More from Nico Meisenzahl

More from Nico Meisenzahl (20)

Recently uploaded

Recently uploaded (20)

Troubleshooting Connections Issues

Editor's Notes