2. Agenda :
➢ Objective of performance monitoring
➢ Traditional methods of performance monitoring
➢ Need of a new performance monitoring tool kit
➢ Introduction to Performance Co-Pilot
➢ Component and Architecture of Performance Co-Pilot
➢ Basic performance monitoring tools
➢ Demo
➢ Knowledge sources
3. Objective of performance monitoring :
➢ Early detection of a ( potential ) problem.
➢ Rapidly drill down & pinpoint issue in specific program.
➢ Reduce downtime of mission critical services/systems.
4. Traditional methods of performance
monitoring :
➢ System log files ( rsyslog/syslog-ng/journald )
➢ Native performance monitoring tools
( top/iostat/vmstat/ps etc. )
➢ Combination of scripting languages
( bash/perl/python )
➢ Specific tools vary per platform
5. Need of a new performance monitoring tool kit :
➢ Centralized monitoring of local or distributed systems in a complex
network.
➢ Automated performance monitoring of system and services.
➢ System-level analysis of performance statistics.
➢ To record and replay performance statistics.
➢ Performance regression detection.
➢ Evaluation of effects of an operating system upgrade etc.
6. Introduction to Performance Co-Pilot :
➢ PCP is a system level performance monitoring and performance
management toolkit.
➢ PCP provides a range of services that may be used for collection,
monitoring and analysis of system metrics.
➢ PCP includes many facilities for creating and replaying archive logs
that capture performance information.
➢ Cross platform support : Linux, Mac OS and Windows.
➢ End-to-end support : Hardware, Core OS, Services and
applications.
7. Introduction to Performance Co-Pilot :
➢ Distributed architecture
Monitoring of local and remote nodes.
➢ Real-time or retrospective
Live system or archive.
➢ Pluggable
New agents system metrics within PCP.
8. Performance Co-Pilot – Components :
Collectors
● Collect and export performance metrics
● Performance Metric Domain Agents (PMDA)
● Performance Metric Collection Daemon (PMCD)
Consumers
● Record, visualise, monitor and analyse performance data
● Consume data either in realtime or replay historical data
from archive logs
Note: Hosts may operate as collectors, consumers or both.
● Multiple consumers may connect with one or more collectors
9. Performance Co-Pilot – Collectors :
Performance Metric Domain Agents (PMDA)
● Extracts & exports metric data from a system component
● Communicates with pmcd on local system
Performance Metric Collection Daemon (PMCD)
● One pmcd process per host
● Coordinates handling of fetch requests between consumer
applications and agents
● Listens for connections from localhost & remote clients
● Authenticated & encrypted connection options
10. Performance Co-Pilot – Consumers :
● pmlogger
● Utility to capture and store metrics exported by PMCD
● Concurrent logging of data from local and remote hosts
● Archive playback by other consumer tools (eg pmchart, pmval)
● Tools for log archival, log rotation etc
● Operates on data in realtime only
● pminfo - display PCP metrics available on a host or in an archive
● pmchart - GUI utility providing graphical display of PCP data
● pmstat - vmstat-like utility
● pmatop - top-like utility
● pmie - inference engine and alerting utility
● pmval - display a metric
13. Installation:
To install pcp and pcp-gui package, type the following
command:
# yum install pcp pcpgui
“pcp” package provides a framework and services to support
system-level performance monitoring and performance
management.
“pcp-gui” package provides a visualization tools for the
Performance Co-Pilot toolkit.
14. Services:
To start pmcd and pmlogger services, type the following command:
# chkconfig pmcd on
# chkconfig pmlogger on
# service pmcd start
# service pmlogger start
To check the status of pcp services, type the following command:
# service pcp status
Note: Bug: 1044682 - pcp should use systemd
16. Configuration files:
Main configuration file for pcp:
/etc/pcp.conf
Main configuration file for pmcd:
/etc/pcp/pmcd/pmcd.conf
Command line options for the pmcd:
/etc/pcp/pmcd/pmcd.options
Default pmlogger config file:
/etc/pcp/pmlogger/config.default
PCP archive logging configuration/control:
/etc/pcp/pmlogger/control
PCP log control mechanisms:
/var/lib/pcp/config/pmlogconf
Note: pmlogger is utility to capture and store metrics exported by PMCD
19. Performance monitoring tools:
➢ For all PCP monitoring tools, metrics values may come from a real-
time feed (i.e. from pmcd on some host), or from an archive log.
➢ Performance monitoring tools available in Performance Co-Pilot
(PCP).
pmstat
pmatop
pmcollectl
pmval
pmchart
pminfo
20. Performance monitoring tools:
pmstat - vmstat-like utility, intended to monitor system
performance at the highest level.
# pmstat
# pmstat t 1 T 3
# pmstat a
/var/log/pcp/pmlogger/localhost.localdomain/20140607.20.20.0
22. Performance monitoring tools:
pmcollectl - System-level performance monitoring utility that
records or displays specific operating system data for one or more
sets of subsystems.
Basic Systemlevel performance monitoring:
# pmcollectl
Subsystem Specific performance monitoring:
# pmcollectl sm
# pmcollectl sm –verbose
# pmcollectl smcdn –verbose
Record Mode:
# pmcollectl f perfdata c 3
Playback Mode:
# pmcollectl p perfdata
23. Performance monitoring tools:
pminfo - Display PCP metrics available on a host or in an archive.
Full list of all available metrics:
# pminfo F
Fetch and print values for all or specific metrics:
# pminfo f
# pminfo f <metric>
# pminfo F a
/var/log/pcp/pmlogger/localhost.localdomain/20140607.20.56.0
Eg:
# pminfo f proc.nprocs disk.dev.read filesys.free
Descriptions of all or specific metrics:
# pminfo T d
# pminfo T d <metric>
Eg:
# pminfo T d mem.util.cached
# pminfo T d proc.nprocs disk.dev.read filesys.free
24. Performance monitoring tools:
pmval - Dumps the current values for the named performance
metrics.
# pmval <metric>
Eg:
# pmval proc.nprocs
# pmval kernel.all.load
# pmval t 2sec s 4 kernel.percpu.cpu.idle
26. Knowledge sources:
● What are all the Performance Co-Pilot (PCP) RPM packages in RHEL?
https://access.redhat.com/articles/1146003
● How do I install Performance Co-Pilot (PCP) on my RHEL server to capture
performance logs
https://access.redhat.com/solutions/1137023
● How do I configure a firewall on a RHEL server to allow remote monitoring with
Performance Co-Pilot (PCP)?
https://access.redhat.com/solutions/1145963
● How can I customize the Performance Co-Pilot logging configuration
https://access.redhat.com/articles/1146283
27. Knowledge sources:
● What are the typical Performance Co-Pilot (PCP) deployment strategies ?
https://access.redhat.com/articles/1147393
● PCP extensibility through custom agents
https://access.redhat.com/solutions/749813
● Index of Performance Co-Pilot (PCP) articles, solutions, tutorials and white papers
https://access.redhat.com/articles/1145953
● How does Performance Co-Pilot (PCP) compare with sysstat
https://access.redhat.com/articles/1148133
● Overview of Additional Performance Tuning Utilities in Red Hat Enterprise Linux 7
https://access.redhat.com/articles/785283
