SlideShare a Scribd company logo
1 of 60
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



                                  A Project Report

                                              On


     Centralized Monitoring Station for IT
    Computing and Network Infrastructure

                        for the award of the degree of


                    BACHELOR OF TECHNOLOGY


                                               In


            COMPUTER SCIENCE & ENGINEERING


                                              By:
                       Anurag Tripathi (0621610029)
                  Abhinav Kumar Chand (0621610003)
                                Aviral (0621610038)
                    Indra Nath Banerjee (0521610055)




                              Under Supervision of:
                           Shri Rakesh Kumar Singh



1
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



                                          INTRODUCTION

We can monitor your system's Web performance quite easily with graphical Linux tools.
Here we will see how to monitor our network which has many technology, including MRTG,
which is based on SNMP and monitors raw network traffic, and Webalizer, which tracks Web
site hits.

We are implementing a network monitoring solution for a small network (approximately 5-10
network devices). We'd like to have all devices on our network send SNMP traps to a single
box (technically this will probably be an HA pair of boxes) and then have that box pass the
SNMP traps on to the real processing boxes. This will allow us to have multiple back-end
boxes handling traps, and to distribute load among those back end boxes.




       2
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



2. SNMP

Most servers, routers and firewalls keep their operational statistics in object identifiers
(OIDs) that you can remotely retrieve via the Simple Network Management Protocol
(SNMP). For ease of use, equipment vendors provide Management Information Base (MIB)
files for their devices that define the functions of the OIDs they contain. That's a lot of new
terms to digest in two sentences, so take a moment to look more closely.


2.1 OIDs and MIBs

OIDs are arranged in a structure of management information (SMI) tree defined by the
SNMP standard. The tree starts from a root node, which then descends through branches and
leaves that each add their own reference value to the path separated by a period.. Figure 1
shows an OID structure in which the path to the enterprises OID branch passes through the
org, dod, internet, and private branches first. The OID path for enterprises is, therefore,
1.3.6.1.4.1.

Management Information Bases (MIBs) are text definitions of each of the OID branches.
Table 1 shows how some commonly used OIDs map to their MIB definitions. For example,
the SMI org MIB defines all the topmost OIDs found at the next layer, which is named dod;
the internet MIB under dod defines the function of the topmost OIDs in the directory, mgmt,
experimental, and private branches. This MIB information is very useful for SNMP
management programs, enabling you to click on an OID and see its value, type, and
description




       3
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



Figure 2.1: SNMP OID Structure




Table 2.1: OIDs And Their Equivalent MIBs


                                        OID           MIB
                                        1.3           org
                                        1.3.6         dod
                                        1.3.6.1       internet
                                        1.3.6.1.1     directory
                                        1.3.6.1.2     mgmt
                                        1.3.6.1.3     experimental
                                        1.3.6.1.4     private
                                        1.3.6.1.4.1   enterprises



We can refer to an OID by substituting the values in a branch with one of these more readable
MIB aliases. For example, you can reference the OID 1.3.6.1.4.1.9.9.109.1.1.1.1.5 as

      4
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



enterprises.9.9.109.1.1.1.1.5.1 by substituting the branch name (enterprises) for its OID
numbers (1.3.6.1.4.1).
Remember, only the OID value at the very tip of a branch, the leaf, actually has a readable
value. Think of OIDs like the directory structure on a hard disk. Each branch is equivalent to
a subdirectory, and the very last value at the tip (the leaf) correlates to a file containing data.
The Linux snmpget command outputs the value of a single leaf, and the snmpwalk
command provides the values of all leaves under a branch. I'll discuss these commands later;
for now, all there is a need to know, that the command output frequently doesn't list the entire
OID, just the MIB file in which it was found and the alias within the MIB. For example

  SNMPv2-MIB::sysUpTime.0

Here the OID value was found in the SNMPv2-MIB file and occupies position zero in the
sysUpTime alias.
Equipment manufacturers are usually assigned their own dedicated OID branch under the
enterprises MIB, and they must also provide information in universally accepted OIDs for
ease of manageability. For example, NIC interface data throughput values must always be
placed in a predefined location in the general tree, but a memory use value on a customized
processor card may be defined in a MIB under the manufacturer's own OID branch.


2.2 SNMP Community Strings
As a security measure, you need to know the SNMP password, or community string, to query
OIDs. There are a number of types of community strings, the most commonly used ones are
the Read Only or "get" community string that only provides access for viewing statistics and
system parameters. In many cases the Read Only community string or password is set to the
word "public;" you should change it from this easy-to-guess value whenever possible. The
Read/Write or "set" community string is for not only viewing statistics and system
parameters but also for updating the parameters.




2.3 SNMP Versions
There are currently three versions of SNMP.



       5
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



    1. SNMP Version 1: The first version of SNMP to be implemented, version 1 was
          designed to be a protocol to provide device statistics and error reporting without
          consuming a lot of system resources. Security was limited to community strings and
          access controls based on the IP address of the querying server. Data communication
          wasn't encrypted.
    2. SNMP Version 2: The second version of SNMP, often referred to as v2c, expanded
          the number of supported error codes, increased the size of counters used to track data,
          and had the ability to do bulk queries that more efficiently loaded response packets
          with data. SNMP v2c is backward compatible with version 1.
    3. SNMP Version 3: This version provides greater security and remote configuration
          capabilities than its predecessors. Access isn't limited to a single community string
          for read-only and read/write access, as usernames and passwords have been
          introduced. Views of OIDs in a MIB can also be limited on a per-user basis. Support
          for encrypted SNMP data transfer and transfer error detection is also provided.
Remember their differences, because We will need to specify the version number when doing
SNMP queries.


2.4 Doing SNMP Queries
Configuring SNMP on a server isn't hard, but it does require a number of detailed steps.
2.4.1 Installing SNMP Utilities on a Linux Server
If you intend to use your Linux box to query your network devices, other servers or even
itself using MRTG or any other tool, you need to have the SNMP utility tools package net-
snmp-utils installed. This package may also require pre-requisite packages, so it is best to
use an automated package updater such yum or apt to do this.
When searching for these packages the filenames will start with the package name followed
by a version number, as in net-snmp-utils-5.1.1-2.i386.rpm. (If you need an installation
refresher, see Chapter 6, "Installing Linux Software").




2.4.2 SNMP Utilities Command Syntax



      6
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



The SNMP utility tools package installs a number of new commands on your system for
doing SNMP queries, most notably snmpget for individual OIDs and snmpwalk for
obtaining the contents of an entire MIB. Both commands require you to specify the
community string with a -c operator. They also require you to specify the version of the
SNMP query to be used with a -v 1, -v 2c, or -v 3 operator for versions 1, 2, and 3,
respectively. The first argument is the name or IP address of the target device and all other
arguments list the MIBs to be queried.
This example gets all the values in the interface MIB of the local server using SNMP version
1 and the community string of craz33guy.

  [root@bigboy tmp]# snmpwalk -v 1 -c craz33guy localhost interface
  ...
  ...
  IF-MIB::ifDescr.1 = STRING: lo
  IF-MIB::ifDescr.2 = STRING: eth0
  IF-MIB::ifDescr.3 = STRING: eth1
  ...
  ...
  IF-MIB::ifPhysAddress.1 = STRING:
  IF-MIB::ifPhysAddress.2 = STRING: 0:9:5b:2f:9e:d5
  IF-MIB::ifPhysAddress.3 = STRING: 0:b0:d0:46:32:71
  ...
  ...
  [root@bigboy tmp]#

Upon inspecting the output of the snmpwalk command, you can see that the second interface
seems to have the name eth0 and the MAC address 0:9:5b:2f:9e:d5. You can now retrieve the
individual MAC address using the snmpget command.

  [root@bigboy tmp]# snmpget -v 1 -c const1payted localhost ifPhysAddress.2
  IF-MIB::ifPhysAddress.2 = STRING: 0:9:5b:2f:9e:d5
  [root@bigboy tmp]#

We can confirm this information using the ifconfig command for interface eth0; the very first
line shows a matching MAC address.

        7
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  [root@bigboy tmp]# ifconfig -a eth0
  eth0         Link encap:Ethernet HWaddr 00:09:5B:2F:9E:D5
             inet addr:216.10.119.244 Bcast:216.10.119.255 Mask:255.255.255.240
  ...
  ...
  [root@bigboy tmp]#

Now we will see how you can configure SNMP on your Linux server to achieve these results.


2.4.3 Configuring Simple SNMP on a Linux Server
By default Fedora, installs the net-snmp package as its SNMP server product. This package
uses a configuration file named /etc/snmp/snmpd.conf in which the community strings and
other parameters may be set. The version of the configuration file that comes with net-snmp
is quite complicated. I suggest archiving it and using a much simpler version with only a
single line containing the keyword rocommunity followed by the community string. Here is
an example.


1) Save the old configuration file

  [root@bigboy tmp]# cd /etc/snmp/
  [root@bigboy snmp]# mv snmpd.conf snmpd.conf.old
  [root@bigboy snmp]# vi snmpd.conf




2) Enter the following line in the new configuration file to set the Read Only community
string to craz33guy.

  rocommunity craz33guy




3) Configure Linux to start SNMP services on each reboot with the chkconfig command:


        8
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  [root@bigboy root]# chkconfig snmpd on
  [root@bigboy root]#



4) Start SNMP to load the current configuration file.

  [root@bigboy root]# service snmpd start
  Starting snmpd: [ OK ]
  [root@bigboy root]#



5) Test whether SNMP can read the system and interface MIBs using the snmpwalk
command.

  [root@bigboy snmp]# snmpwalk -v 1 -c craz33guy localhost system
  SNMPv2-MIB::sysDescr.0 = STRING: Linux bigboy 2.4.18-14 #1 Wed Sep 4 11:57:57
  EDT 2002 i586
  SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
  SNMPv2-MIB::sysUpTime.0 = Timeticks: (425) 0:00:04.25
  SNMPv2-MIB::sysContact.0 = STRING: root@localhost
  SNMPv2-MIB::sysName.0 = STRING: bigboy
  ...
  ...
  ...
  [root@bigboy snmp]# snmpwalk -v 1 -c craz33guy localhost interface
  IF-MIB::ifNumber.0 = INTEGER: 3
  IF-MIB::ifIndex.1 = INTEGER: 1
  IF-MIB::ifIndex.2 = INTEGER: 2
  IF-MIB::ifIndex.3 = INTEGER: 3
  IF-MIB::ifDescr.1 = STRING: lo
  IF-MIB::ifDescr.2 = STRING: wlan0
  IF-MIB::ifDescr.3 = STRING: eth0
  ...
  ...


        9
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  ...
  [root@bigboy snmp]#

Now that we know SNMP is working correctly on your Linux server, we can configure
SNMP statistics gathering software, such as MRTG, to create online graphs of your traffic
flows.


2.4.4 SNMP On Other Devices
In the example, we were polling localhost. We can poll any SNMP-aware network device that
has SNMP enabled. All you need is the IP address and SNMP Read Only string and you'll be
able to get similar results. Here is an example of a query of a device with an IP address of
192.168.1.1.

  [root@bigboy snmp]# snmpwalk -v 1 -c chir1qui 192.168.1.1 interface



Note: When issuing snmpwalk and snmpget commands, remember to use the proper version
switch (-v 1, -v 2c, or -v 3) for the version of SNMP you're using.


2.5Basic SNMP Security
The security precautions that need to be taken with SNMP vary depending on the version you
are using. This section outlines the basic steps for protecting your MIB data.
2.5.1SNMP Versions 1 and 2
The most commonly supported versions of SNMP don't encrypt your community string
password so you shouldn't do queries over insecure networks, such as the Internet. You
should also make sure that you use all reasonable security measures to allow queries only
from trusted IP addresses either via a firewall or the SNMP security features available in the
snmp.conf file. You can also configure your server to use the TCP wrappers feature outlined
in Appendix I, "Miscellaneous Linux Topics," to limit access to specific servers without the
need of a firewall.


In case you need it, the snmpd.conf file can support limiting MIB access to trusted hosts and
networks.

        10
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



The snmpd.conf file has two security sections; a section with very restrictive access sits at the
top of the file and is immediately followed by a less restrictive section. The example that
follows is a modification of the less restrictive section. You will have to comment out the
more restrictive statements at the top of the file for it to work correctly.



  ##         sec.name       source           community
  ##         ========         ======            =========
  com2sec local             localhost         craz33guy
  com2sec network_1 172.16.1.0/24                 craz33guy
  com2sec network_2 192.168.2.0/24 craz33guy


  ##         Access.group.name sec.model                sec.name
  ##         ================= =========                         ========
  group        MyROGroup_1              v1             local
  group        MyROGroup_1              v1             network_1
  group        MyROGroup_2              v2c            network_2


  ## MIB.view.name               incl/excl MIB.subtree mask
  ## ============== ========= =========== ====
  view all-mibs             included .1           80


  ##         MIB
  ##         group.name context sec.model sec.level prefix read              write notif
  ##          ==========          ======= ========= ========= ====== ====                          =====
  =====
  access MyROGroup_1 ""                  v1       noauth       exact all-mibs none none
  access MyROGroup_2 ""                  v2c      noauth       exact all-mibs none none




In our example:



       11
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



    Only three networks (localhost, 172.16.1.0/24, and 192.168.1.0/24) are allowed to access
     the server with the craz33guy community string.
    Each network is matched to a either a group called MyROGroup_1 using SNMP version
     1, or group called MyROGroup_2 using SNMP version 2.
    All the MIBs on the server are defined by the view named all-mibs.
    An access statement ensures that only the defined networks have read only access to all
     the MIBs. MyROGroup_1 only has version 1 access with MyROGroup_2 only having
     version 2 access.
    Modification of the MIBs via SNMP is denied because the word "none" is in the write
     section of the access statement.
These precautions are probably unnecessary in a home environment where access is generally
limited to devices on the home network by a NAT firewall.


2.5.2 SNMP Version 3
SNMP Version 3 SNMPv3 is a much more secure alternative to earlier versions as it encrypts
all its data and uses a username / password combination for client authentication. The
username should be located in the /etc/snmp/snmpd.conf file with a corresponding
automatically generated password located in the /var/net-snmp/snmpd.conf file. Here is
how it's done.


1. Install the net-snmp-devel package as it contains the utility that will allow you to generate
the password. If you need an installation refresher.


2. Stop the snmpd process.

    [root@bigboy tmp]# service snmpd stop
    Stopping snmpd: [ OK ]
    [root@bigboy tmp]#



3. Automatically create the /etc/snmp/snmpd.conf and /var/net-snmp/snmpd.conf
username and password entries using the net-snmp-config command. In this example the
authentication password of "rootsrockreggae" for the read only (-ro) username



      12
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



"username4snmpv3" is encrypted using the MD5 algorithm. The data received will not be
encrypted.

  [root@bigboy tmp]# net-snmp-config --create-snmpv3-user -ro 
  -a MD5 -A rootsrockreggae username4snmpv3


  adding the following line to /var/net-snmp/snmpd.conf:
    createUser username4snmpv3 MD5 "rootsrockreggae" DES
  adding the following line to /etc/snmp/snmpd.conf:
    rouser username4snmpv3
  [root@bigboy tmp]# service snmpd start
  Starting snmpd: [ OK ]
  [root@bigboy tmp]#



4. To encrypt the data received we use the net-snmp-config command again to create a
username just for this purpose. A new read only username "securev3user" and authentication
password "mandeville" are used, but this time the data will be encrypted using the DES
algorithm with the privacy password of "savlamar".

  [root@bigboy tmp]# net-snmp-config --create-snmpv3-user -ro 
  -a MD5 -A mandeville -x DES -X savlamar securev3user


  adding the following line to /var/net-snmp/snmpd.conf:
    createUser securev3user MD5 "mandeville" DES savlamar
  adding the following line to /etc/snmp/snmpd.conf:
    rouser securev3user
  [root@bigboy tmp]#




5. Start the snmpd process.

  [root@bigboy tmp]# service snmpd start
  Starting snmpd: [ OK ]


     13
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  [root@bigboy tmp]#




6. First we'll do a query from remote host smallfry. We specify the authentication password
and authentication encryption method, and we also use the -l flag to indicate that
authentication will be used, but that data privacy will be disabled (the authNoPriv option).

  [root@smallfry ~]# snmpget -v 3 -u username4snmpv3 -l authNoPriv 
  -a MD5 -A rootsrockreggae 192.168.1.100 SNMPv2-MIB::sysORDescr.8


  SNMPv2-MIB::sysORDescr.8 = STRING: The management information definitions for
  the SNMP User-based Security Model.
  [root@smallfry ~]#



The query returns an easy to read string, "The management information definitions for the
SNMP User-based Security Model". This unencrypted string can also be seen in the tshark
packet capture of the server's interface.

  [root@bigboy tmp]# tshark -n -i eth1 -x port 161
  Capturing on eth1
  ...
  ...
  ...


    0.005889 192.168.1.100 -> 192.168.1.50 SNMP get-response


  0000 00 c0 4f 46 0c 2e 00 b0 d0 46 32 71 08 00 45 00 ..OF.....F2q..E.
  0010 00 f0 00 00 40 00 40 11 b3 b2 c0 a8 02 c8 c0 a8 ....@.@.........
  0020 02 32 00 a1 80 0a 00 dc 87 38 30 81 d1 02 01 03 .2.......80.....
  0030 30 11 02 04 45 a2 23 54 02 03 00 ff e3 04 01 01 0...E.#T........
  0040 02 01 03 04 38 30 36 04 0d 80 00 1f 88 80 71 11 ....806.......q.
  0050 68 72 0e b1 e7 45 02 01 12 02 01 39 04 0f 75 73 hr...E.....9..us


        14
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  0060 65 72 6e 61 6d 65 34 73 6e 6d 70 76 33 04 0c 46 ername4snmpv3..F
  0070 6c 74 26 51 4d aa 65 61 59 06 1a 04 00 30 7f 04 lt&QM.eaY....0..
  0080 0d 80 00 1f 88 80 71 11 68 72 0e b1 e7 45 04 00 ......q.hr...E..
  0090 a2 6c 02 04 43 4e da d7 02 01 00 02 01 00 30 5e .l..CN........0^
  00a0 30 5c 06 0a 2b 06 01 02 01 01 09 01 03 08 04 4e 0..+..........N
  00b0 54 68 65 20 6d 61 6e 61 67 65 6d 65 6e 74 20 69 The management i
  00c0 6e 66 6f 72 6d 61 74 69 6f 6e 20 64 65 66 69 6e nformation defin
  00d0 69 74 69 6f 6e 73 20 66 6f 72 20 74 68 65 20 53 itions for the S
  00e0 4e 4d 50 20 55 73 65 72 2d 62 61 73 65 64 20 53 NMP User-based S
  00f0 65 63 75 72 69 74 79 20 4d 6f 64 65 6c 2e                    ecurity Model.


  4 packets captured
  [root@bigboy tmp]#




7. Next we'll do a query that will return a response over an encrypted data channel while
crossing the network (the authPriv option).

  [root@smallfry ~]# snmpget -v 3 -u securev3user -l authPriv 
  -a MD5 -A mandeville                       -x DES -X savlamar              192.168.1.100 SNMPv2-
  MIB::sysORDescr.8


  SNMPv2-MIB::sysORDescr.8 = STRING: The management information definitions for
  the SNMP User-based Security Model.
  [root@smallfry ~]#



The query returns the same string, but the tshark packet capture only sees encrypted data,
with only the username being visible.

  [root@bigboy tmp] # tshark -n -i eth1 -x port 161
  Capturing on eth1
  ...


        15
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  ...
  ...
    0.003675 192.168.1.200 -> 192.168.1.50 SNMP Source port: 161 Destination port:
  32778 [UDP CHECKSUM INCORRECT]


  0000 00 c0 4f 46 0c 2e 00 b0 d0 46 32 71 08 00 45 00 ..OF.....F2q..E.
  0010 01 00 00 00 40 00 40 11 b3 a2 c0 a8 02 c8 c0 a8 ....@.@.........
  0020 02 32 00 a1 80 0a 00 ec 87 48 30 81 e1 02 01 03 .2.......H0.....
  0030 30 11 02 04 17 52 82 96 02 03 00 ff e3 04 01 03 0....R..........
  0040 02 01 03 04 3e 30 3c 04 0d 80 00 1f 88 80 71 11 ....>0<.......q.
  0050 68 72 0e b1 e7 45 02 01 11 02 02 00 8e 04 0c 73 hr...E.........s
  0060 65 63 75 72 65 76 33 75 73 65 72 04 0c 01 b2 00 ecurev3user.....
  0070 6e 23 07 83 dc a2 b6 d6 3d 04 08 00 00 00 11 4e n#......=......N
  0080 df 19 a3 04 81 88 36 dd e0 ce e0 52 19 ff 58 7e ......6....R..X~
  0090 be fa d1 96 20 2b 28 65 59 30 e8 d4 cb 18 9f 8f .... +(eY0......
  00a0 1e 5b a3 d6 ae f7 4a 86 bd ed 2a 4b a8 df 52 fb .[....J...*K..R.
  00b0 00 b4 a8 37 3d 74 9e 6d 1d 56 9a ba f2 13 fa 72 ...7=t.m.V.....r
  00c0 4d 47 fb 88 7b d3 54 e1 9d b3 66 f0 29 ab 8a 55 MG..{.T...f.)..U
  00d0 6f 77 65 40 87 ab 0c 51 d9 0e bf 33 7f 9a cb ea owe@...Q...3....
  00e0 37 50 3c 8e 65 dd 8f 3c 49 71 96 59 f9 d3 a8 23 7P<.e..<Iq.Y...#
  00f0 81 c6 1b b2 c2 d0 57 9b 98 1b 89 1e ca 77 3d 84 ......W......w=.
  0100 6f af b6 9b 86 3a 2f 66 44 1a 41 51 03 bc                    o....:/fD.AQ..


  4 packets captured
  [root@bigboy tmp] #



8. Your password and privilege information are stored in /var/net-snmp/snmpd.conf using
the format displayed when you used the net-snmp-config command. here is an example.

  # File: /var/net-snmp/snmpd.conf before SNMP starts


  createUser securev3user MD5 "mandeville" DES savlamar


        16
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  createUser username4snmpv3 MD5 "rootsrockreggae" DES




The snmpd daemon will completely encrypt the SNMP password data in this file when it
restarts which helps to further increase security. We can see an example of this configuration
here.

  # File: /var/net-snmp/snmpd.conf after SNMP starts


  usmUser 1 3 0x80001f88780711168720eb1e745
  0x73656375626576337573657200
  0x73656375726576337573657200
  NULL .1.3.6.1.6.3.10.1.1.2 0xd951
  ac1d95033f4afgf31243eb6907df .1.3.6.1.6.3.10.1.2.2
  0xf1f4bb00452211d27b50c273c09031ac 0x00
  usmUser 1 3 0x80001f8880711168720eb1e745
  0x757365726e61657534736e6d70763300
  0x757365726e616d6534736e6d70763300
  NULL .1.3.6.1.6.3.10.
  1.1.2 0x5e35c9f5352519aa4f53eded09bbdddd
  .1.3.6.1.6.3.10.1.2.2 0x5e35c9f5122519aa4f53eded09bbdddd ""
  setserialno 1464593474



We here showing the practice of the net-snmp-config command so that we and viewer can
become familiar with the syntax it uses to edit the SNMP configuration files. When in doubt,
we can get a full syntax listing of the command if you use it without any arguments like this:

  [root@bigboy tmp]# net-snmp-config
  ...
  ...
   SNMP Setup commands:




        17
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




       --create-snmpv3-user [-ro] [-A authpass] [-X privpass]
                          [-a MD5|SHA] [-x DES|AES] [username]
    ...
    ...
    [root@bigboy tmp]#




With experience, we should become confident enough to edit the configuration files by
ourself.
As we can see, SNMPv3 is more secure than previous versions and should be your first
SNMP choice whenever possible.


2.6 Simple SNMP Troubleshooting
If your SNMP queries fail, then verify that:
    We restarted your snmp.conf file so the configuration settings become active. Remember,
     the snmpd.conf file is only read by the snmpd daemon when it starts up.
    We are using the correct community string.
    Firewalls aren't preventing SNMP queries from the SNMP client to the SNMP target.
    Our SNMP security policy allows the query from our network.
    Any TCP wrappers configuration on our SNMP target machine allows SNMP queries
     from our SNMP client. Generally speaking in a home environment protected by NAT
     your TCP wrappers files (/etc/hosts.allow) and (/etc/hosts.deny) should be blank.
    Network routing between the client and target devices is correct. A simple ping or
     traceroute test should be sufficient.
    The snmpd daemon is running on the SNMP client.
    We are querying using the correct SNMP version.
    Our /var/log/messages file does not contain errors that may have occurred while starting
     snmpd.
Troubleshooting to get functioning SNMP queries is important as many other supporting
applications, such as MRTG which I'll discuss next, rely on them in order to work correctly.




          18
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




3 MRTG
MRTG (Multi-Router Traffic Grapher) is a public domain package for producing graphs of
various router statistics via a Web page. You can easily create graphs of traffic flow statistics
through your home network's firewall/router or even your Linux box's NIC cards using
MRTG. The product is available from the MRTG Web site (www.mrtg.org) and also on your
distribution CDs. Figure 2 shows a sample MRTG graph.




Figure 3.1: A Typical MRTG Web Page




     19
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




3.2 MRTG Download and Installation
You need to install MRTG before proceeding. Most RedHat and Fedora Linux software
products are available in the RPM format. When searching for the file, remember that the
MRTG RPM's filename usually starts with mrtg and a version number, as in
mrtg-2.10.5-3.i386.rpm.
In addition to MRTG, you need to install the SNMP utility tools as explained earlier and you
need to have a Web server package installed for MRTG to work. RedHat Linux usually
comes with the Apache Web server software preinstalled. The easiest way to tell if Apache is
installed is to run the rpm -q httpd command. If you don't get a positive response, you can
refer to Chapter 20, "The Apache Web Server", for installation details. By default Apache
expects the HTML files for your Web site to be located in /var/www/html. MRTG places its
HTML files in /var/www/mrtg.


3.3 Configuring MRTG
By default, MRTG maps the inbound and outbound data throughput rates on the device it is
polling. Methods for specifying other OIDs, such as CPU and memory usage, are discussed
in Chapter 23, "Advanced MRTG for Linux". For now, I'll stick with the default
configuration.
When the MRTG RPM is installed, it creates a directory called /etc/mrtg in which all future
configuration files are stored. To create a replacement default /etc/mrtg/mrtg.cfg
configuration file for the server, follow these steps.
1) Use MRTG's cfgmaker command to create a configuration file named mrtg.cfg for the
server (bigboy) using a Read Only community string of craz33guy. Place all data files in the
directory /var/www/mrtg.

  [root@bigboy tmp]# cfgmaker --output=/etc/mrtg/mrtg.cfg 
  --global "workdir: /var/www/mrtg" -ifref=ip 
  --global 'options[_]: growright,bits' 
  craz33guy@localhost




     20
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  --base: Get Device Info on craz33guy@localhost:
  --base: Vendor Id:
  --base: Populating confcache
  --snpo: confcache craz33guy@localhost: Descr lo --> 1
  --snpo: confcache craz33guy@localhost: Descr wlan0 --> 2
  ...
  ...
  ...
   --base: Walking ifAdminStatus
  --base: Walking ifOperStatus
  --base: Writing /etc/mrtg/mrtg.cfg
  [root@bigboy tmp]#




SNMPv2: As explained in the SNMP section, there are different versions of SNMP. If your
query doesn't work, check to make sure you are using the required version and then check
other SNMP configuration parameters on the target device. You can specify MRTG's SNMP
query version with the --snmp-options cfgmaker option. Here is an example of cfgmaker
using an SNMP version 2 query of a router with an IP address of 192.168.1.3. The --snmp-
options option's five colons before the 2 are important.

  [root@bigboy tmp]# cfgmaker --output=/etc/mrtg/192.168.1.3.cfg 
  -ifref=ip --global "workdir: /var/www/mrtg" 
  --snmp-options=:::::2 craz33guy@192.168.1.3



SNMPv3: The cfgmaker command can also be used to poll SNMPv3 enabled devices, but
we have to first install the Net::SNMP PERL module.

  [root@bigboy tmp]# yum -y install perl-Net-SNMP


  or




        21
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  [root@bigboy tmp]# perl -MCPAN -e "install Net::SNMP"




If we fail to install the module, we will get an error looking like this:

  Undefined subroutine &main::snmpmapOID called at ./cfgmaker line 1480.



Next we need to know the SNMPv3 ID of the host you intend to poll. In fedora, this is value
is located in the /var/net-snmp/snmpd.conf file. You can use the grep command to obtain it.

  [root@bigboy tmp]# grep oldEngineID /var/net-snmp/snmpd.conf
  oldEngineID 0x80001f8880711168720eb1e745
  [root@bigboy tmp]#



We can then repeat the cfgmaker command with options specific to the privacy and
authentication schemes configured on the SNMP target device. Using the configurations of
our previous SNMPv3 example, our configuration for encrypted authentication only would
look like this:

  [root@bigboy tmp]# cfgmaker --global 'WorkDir: /var/www/mrtg' 
  --global 'Options[_]: growright, bits' 
  --output=/etc/mrtg/192.168.1.100.cfg 
  --enablesnmpv3 --username=username4snmpv3 
  --authpassword=rootsrockreggae --authproto=md5 
  --snmp-options=:::::3 
  --contextengineid=0x80001f8880711168720eb1e745 
  securev3user@192.168.1.100



Our configuration for encrypted authentication and data privacy only would look like this:

  [root@bigboy tmp]# cfgmaker --global 'WorkDir: /var/www/mrtg' 



     22
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  --global 'Options[_]: growright, bits' 
  --output=/etc/mrtg/192.168.1.100-secure.cfg 
  --enablesnmpv3 --username=securev3user --authpassword=mandeville 
  --authproto=md5 --privpassword=savlamar --privprotocol=des 
  --snmp-options=:::::3 
  --contextengineid=0x80001f8880711168720eb1e745 
  securev3user@192.168.1.100



Note: The MRTG cfgmaker command reliably supports SNMPv3 as of MRTG version
2.15. Prior to this version you would commonly see this error when attempting to do
DNMPv3 queries.

  SNMP V3 requires a --username parameter as part of the User Security Model for router
  securev3user@192.168.1.100:::::3 at ./cfgmaker line 121.



2) Edit /etc/mrtg/mrtg.cfg, and remove the sections related to interfaces you don't need to
monitor. A certain candidate would be the virtual loopback interface Lo: (with the IP address
of 127.0.0.1), which doesn't pass any network traffic at all.


3) Run MRTG using /etc/mrtg/mrtg.cfg as your argument three times. You'll get an error the
two times as MRTG tries to move old data files, and naturally, the first time it is run, MRTG
has no data files to move.

  [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
  Rateup WARNING: /usr/bin/rateup could not read the primary log file for
  localhost_192.168.1.100
  Rateup WARNING: /usr/bin/rateup The backup log file for localhost_192.168.1.100
  was invalid as well
  Rateup WARNING: /usr/bin/rateup Can't remove localhost_192.168.1.100.old updating
  log file
  Rateup WARNING: /usr/bin/rateup Can't rename localhost_192.168.1.100.log to
  localhost_192.168.1.100.old updating log file


     23
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
  Rateup WARNING: /usr/bin/rateup Can't remove localhost_192.168.1.100.old updating
  log file
  [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
  [root@bigboy tmp]#



4) Use MRTG's indexmaker command to create a Web index page using your new mrtg.cfg
file as a guide. The MRTG Web GUI expects to find the index file in the default MRTG Web
directory of /var/www/mrtg/, so the format of the command would be.

  [root@bigboy tmp]# indexmaker --output=/var/www/mrtg/index.html 
  /etc/mrtg/mrtg.cfg



5) MRTG is run every five minutes by default, and the file that governs this is
/etc/cron.d/mrtg. For MRTG to work correctly, edit this file, replacing all occurrences of
/usr/bin/mrtg with env LANG=C /usr/bin/mrtg. The explanation for changing the language
character set for MRTG is given in the "Troubleshooting MRTG" section.
This isn't all, you need to view the graphs too. This will be covered later, but first I'll show
you how to poll multiple devices.


3.4 Getting MRTG To Poll Multiple Devices
The Fedora Core MRTG installation process creates a cron file named /etc/cron.d/mrtg.
This file tells the cron daemon to run MRTG using the /etc/mrtg/mrtg.cfg file every five
minutes to poll your network devices. You can configure MRTG to poll multiple devices,
each with a separate configuration file. Here's how:
1) Create a new configuration file using the steps from the previous section; choose a
filename that is not mrtg.cfg.


2) Add a new MRTG line in /etc/cron.d/mrtg for each new configuration file you create.

  0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg



     24
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/device1.cfg
  0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/device2.cfg



3) Run the indexmaker command, and include all of your /etc/mrtg configuration files, to
regenerate your Web index page.

  [root@bigboy tmp]# indexmaker --output=/var/www/mrtg/index.html 
  /etc/mrtg/mrtg.cfg /etc/mrtg/device1.cfg /etc/mrtg/device2.cfg



4) Other versions of Linux keep their MRTG cron entries inside the /etc/crontab file. Edit
this file using the same syntax as the Fedora /etc/cron.d/mrtg file, and then restart the cron
daemon to re-read the configuration:

  [root@bigboy tmp]# service crond restart



You could also create a script with the /usr/bin/mrtg /etc/mrtg/device.cfg entries in it and
make cron run it every five minutes. This way you can just edit the script each time you add a
device without having to restart cron.


3.5 Configuring Apache To Work With MRTG
MRTG is useful because it can provide a graphical representation of your server's
performance statistics via a Web browser.
With        Fedora      Core,     MRTG        creates     an    add-on      configuration      file      named
/etc/httpd/conf.d/mrtg.conf that includes all the necessary Apache commands for MRTG to
work.
Some configuration may need to be done, because by default MRTG accepts Web requests
from the Linux console only. You can add your home network to the file by inserting the
network on the Allow from line, or you can allow universal access by commenting out that
line along with the Deny from line. This example adds access from the 192.168.1.0 network.

  <Location /mrtg>



       25
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




     Order deny,allow
     Deny from all
     Allow from localhost 192.168.1.0/24
  </Location>



If you want to access MRTG from the Internet, then you'll have to comment out the Deny
statement and allow from all IP addresses:

  <Location /mrtg>
     Order deny,allow
     Allow from all
  </Location>



Remember to restart Apache once you have made these modifications in order for these
changes to take effect.
Note: With newer versions of Fedora, Apache automatically reads the add-on files in the /etc/
httpd/conf.d/ directory. With Fedora Core 1, you have to specifically configure the Apache
configuration file /etc/httpd/conf/httpd.conf to find it. You can do this yourself by inserting
this line at the very bottom of the main Apache configuration file before restarting Apache
for the change to take effect.

  include "/etc/httpd/conf.d/mrtg.conf"



3.5.1 Basic Security
If you are accessing MRTG graphs from the Internet, you may want to add password
protection to the directory by using a .htaccess file


3.6 How To View The MRTG Graphs In Your Web Browser
You can now access your MRTG graphs by pointing your browser to the URL:
http://server-ip-address/mrtg/



     26
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



3.7 Using MRTG To Monitor Other Subsystems
MRTG will generate HTML pages with daily, weekly, monthly, and yearly statistics for your
interfaces. By default, MRTG provides only network interface statistics. Chapter 23,
"Advanced MRTG for Linux", has detailed examples and explanations of how to monitor
Linux disk, CPU, memory, and Web connection data. The MRTG Web site, www.mrtg.org,
also has links to other sites that show you how to monitor many other subsystems on a variety
of devices and operating systems.


3.8 Troubleshooting MRTG
There are many simple steps you can use to troubleshoot MRTG. Take a look at some of the
most common ones.


3.8.1 Basic Steps
MRTG won't work if SNMP queries don't work. Make sure you follow the SNMP
troubleshooting steps if you have any difficulties.
3.8.2 Setting The Correct Character Set
MRTG usually works only if your system uses an ASCII-based (Western European) character
set. If it isn't set, then you'll get errors such as this every time you run MRTG from the
command line or as part of a cron job:

  [root@bigboy tmp]# mrtg /etc/mrtg/mrtg.cfg
  -------------------------------------------------------------------
  ERROR: Mrtg will most likely not work propperly when the environment
       variable LANG is set to UTF-8. Please run mrtg in an envir..
       where this is not the case:


       env LANG=C /usr/bin/mrtg ...
  -------------------------------------------------------------------
  [root@bigboy tmp]#



Your system's character set is defined in /etc/sysconfig/i18n, and the current Fedora default
of en_US.UTF-8 won't work, but en_US will after a system reboot. This is not necessarily a


     27
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



good idea, especially if the native language Linux uses on your system is not ASCII based,
other things may fail to work.
A better solution is to always run MRTG using this command instead of using just plain /usr/
bin/mrtg.

  env LANG=C /usr/bin/mrtg

This will modify the character set used by MRTG alone and shouldn't affect anything else.


3.8.3 Incorrect SNMPv3 Engine ID
The added security of SNMPv3 forces each client to create its own serial number or engine
ID. If you use an incorrect identifier you may get noSuchInstance errors like this when
polling with MRTG.

  2008-07-26 19:42:40: WARNING: Expected a number but got 'noSuchInstance'
  2008-07-26 19:42:40: WARNING: Expected a number but got 'noSuchInstance'
  2008-07-26 19:42:40: ERROR: Target[localhost_3][_IN_] ' $target->[1]{$mode} ' did
  not eval into defined data
  2008-07-26 19:42:40: ERROR: Target[localhost_3][_OUT_] ' $target->[1]{$mode} '
  did not eval into defined data

Always make sure you are using the correct ID and try again.


3.8.4 Fedora Core 1 MRTG Errors With Net-SNMP
A bug appears in the MRTG implementation for some Fedora Core 1 MRTG versions when
polling another Fedora Core 1 server.
When using a -ifref=ip statement with the cfgmaker command, every line in the configuration
file that is generated becomes commented out. When it works, this statement is very
convenient, because it makes MRTG provide graphs sorted by the IP addresses of the
interfaces instead of the default, which is the much harder to recognize interface MAC
address. Upgrading to the latest Core 1 version of MRTG will fix the problem.

  ### Interface 6 >> Descr: | Name: | Ip: '192.168.1.100'
  ###



     28
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  ### The following interface is commented out because:
  ### * has a speed of which makes no sense
  ### * got 'Received SNMP response with error code
  ###         error status: noSuchName
  ###         index 1 (OID: 1.3.6.1.2.1.2.2.1.10.6)
  ###       SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
  ###                    community: "craz33guy"
  ###                    request ID: 824482716
  ###                   PDU bufsize: 8000 bytes
  ###                      timeout: 2s
  ###                      retries: 5
  #
  # Target[localhost_192.168.1.100]: /192.168.1.100:craz33guy@localhost:
  #              SetEnv[localhost_192.168.1.100]:                   MRTG_INT_IP="192.168.1.100"
  MRTG_INT_DES
  # MaxBytes[localhost_192.168.1.100]: 0
  # Title[localhost_192.168.1.100]: Traffic Analysis for 192.168.1.100
  # PageTop[localhost_192.168.1.100]: Traffic Analysis for 192.168.1.100



As all the lines in the configuration file are commented out with a # character, indexmaker
fails to create an index.html file and gives errors.

  [root@bigboy             tmp]#        indexmaker         --output=/var/www/mrtg/stats/index.html
  /etc/mrtg/mrtg.cfg
  Use of uninitialized value in hash element at /usr/bin/indexmaker line 307.
  [root@bigboy tmp]#




      29
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




4. Webalizer
Webalizer is a Web server log file analysis tool that comes installed by default on
RedHat/Fedora Linux. Each night, Webalizer reads your Apache log files and creates a set of
Web pages that enable you to view Web surfer statistics for your site. The information
provided includes a list of your Web site's most popular pages sorted by hits along with
traffic graphs showing the times of day when your site is most popular.


How To View Your Webalizer Statistics
Fedora creates an add-on configuration file named /etc/httpd/conf.d/Webalizer.conf that
includes all the necessary Apache commands for Webalizer to work. As in the case of the
MRTG add-on file mentioned above, you have to edit it to allow access to the Webalizer
pages from locations other than the Linux console. You also have to restart Apache to make
the changes take effect.
By default, Webalizer places its index page in the directory /var/www/html/usage and allows
you to view your data by visiting the URL http://server-ip-address/usage.


The Webalizer Configuration File
Webalizer stores its configuration in the file /etc/Webalizer.conf. The default settings should
be sufficient for your Web server, but you may want to adjust the directory in which
Webalizer places your graph statistics. This can be adjusted with the OutputDir directive in
the file. After adjustments, Webalizer functions with few annoyances; however, be aware that
running in quiet mode could hide deeper problems that could occur in future.




     30
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




5. The top Command
You can monitor the amount of memory and CPU resources your system is using the top
command.

  [root@bigboy tmp]# top


   3:04pm up 25 days, 23:23, 2 users, load average: 0.00, 0.02, 0.00
  78 processes: 76 sleeping, 2 running, 0 zombie, 0 stopped
  CPU states: 0.9% user, 0.5% system, 0.0% nice, 0.8% idle
  Mem: 384716K av, 327180K used, 57536K free,                        0K shrd, 101544K buff
  Swap: 779112K av,              0K used, 779112K free                  130776K cached


    PID USER               PRI       NI     SIZE    RSS SHARE STAT %CPU %MEM                       TIME
  COMMAND
  27191 root        15 0 1012 1012 780 R              5.6 0.2 0:00 top
   4545 root        16 0 5892 5888 4956 S            0.9 1.5 169:26 magicdev
    1 root       15 0 476 476 432 S                0.0 0.1 0:05 init
    2 root       15 0      0     0        0 SW 0.0 0.0 0:00 keventd
    5 root       15 0      0     0        0 SW 0.0 0.0 0:41 kswapd
    6 root       25 0      0     0        0 SW 0.0 0.0 0:00 bdflush


  [root@bigboy tmp]#



Here the CPU usage is under 1.0% and 14% of memory (57536K) is free. The amount of free
memory may appear low, but in this case, the server doesn't seem to be swapping idle
processes from memory to the swap disk partition as it isn't being used at all. Excessive
swapping can cause your system to slow down dramatically, the simplest ways to avoid this



     31
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



is to add more RAM or reduce the number of processes or users that are active on your
system.
If your system seems slow but the CPU and memory usage is low, then start looking at
networking problems, such as poor duplex negotiation, bad cables, and network congestion
due to excessive traffic.


6. The vmstat Command
You can also determine memory and swap usage with the vmstat command, which provides a
summary of what top produces. In the example, memory is still 14% free (57,452MB used
from a total of 130,780) and swap isn't being used at all.

  [root@bigboy tmp]# vmstat
    procs                     memory     swap         io       system       cpu
    r b w swpd free buff cache si so                 bi       bo in     cs us sy id
    0 0 0       0 57452 101584 130780 0 0                 0    4 18       1 3 1 1
  [root@bigboy tmp]#

As your memory fills up, your system will temporarily store programs and data on your hard
disk's "swap" partition. Excess swapping of programs and data between disk and memory can
cause your system to slow down significantly and memory usage should be monitored to
allow you to plan ways to either increase RAM or tune the way your system operates. System
tuning is beyond the scope of this book, but there are many reference guides which can show
you how to do this.
7. The free Utility
The free utility can determine the amount of free RAM on your system. The output is easier
to understand than vmstat's. Here's a sample.

  [root@bigboy tmp]# free
             total     used      free    shared     buffers       cached
  Mem:          126060        119096      6964         0        58972      40028
  -/+ buffers/cache:        20096       105964
  Swap:         522072        15496     506576
  [root@bigboy tmp]#



     32
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



You should generally try to make your system run with at least 20% free memory on average,
which should allow it to handle moderate spikes in usage caused by running memory-
intensive cron batch jobs or tape backups. If you cannot achieve this, consider running more
efficient versions of programs, offloading some applications to servers with less load, and, of
course, upgrading the capacity of your RAM.


8. Locating And Viewing The Contents Of Linux MIBs
Residing in memory, MIBs are data structures that are constantly updated via the SNMP
daemon. The MIB configuration text files are located on your hard disk and loaded into
memory each time SNMP restarts.
You can easily find your Fedora Linux MIBs by using the locate command and filtering the
output to include only values with the word "snmp" in them. As you can see in this case, the
MIBs are located in the /usr/share/snmp/mibs directory:

  [root@bigboy tmp]# locate mib | grep snmp
  /usr/share/doc/net-snmp-5.0.6/README.mib2c
  /usr/share/snmp/mibs
  /usr/share/snmp/mibs/DISMAN-SCHEDULE-MIB.txt
  ...
  ...
  [root@bigboy tmp]#



As the MIB configurations are text files you can search for keywords in them using the grep
command. This examples searches for the MIBs that keep track of TCP connections and
returns the RFC1213 and TCP MIBs as the result.

  [root@silent mibs]# grep -i tcp /usr/share/snmp/mibs/*.txt | grep connections
  ...
  RFC1213-MIB.txt: "The limit on the total number of TCP connections
  RFC1213-MIB.txt: "The number of times TCP connections have made a
  ...
  TCP-MIB.txt:            "The number of times TCP connections have made a
  ...


        33
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  ...
  [root@silent mibs]#




You can use the vi editor to look at the MIBs. Don't change them, because doing so could
cause SNMP to fail. MIBs are very complicated, but fortunately the key sections are
commented.
Each value tracked in a MIB is called an object and is often referred to by its object ID or
OID. In this snippet of the RFC1213-MIB.txt file, you can see that querying the
tcpActiveOpens object returns the number of active open TCP connections to the server. The
SYNTAX field shows that this is a counter value.
MIBs usually track two types of values. Counter values are used for items that continuously
increase as time passes, such as the amount of packets passing through a NIC or amount of
time CPU been busy since boot time. Integer values change instant by instant and are useful
for tracking such statistics as the amount of memory currently being used.

  tcpActiveOpens OBJECT-TYPE
        SYNTAX Counter
        ACCESS read-only
        STATUS mandatory
        DESCRIPTION
               "The number of times TCP connections have made a
               direct transition to the SYN-SENT state from the
               CLOSED state."
        ::= { tcp 5 }



You'll explore the differences between SNMP and MRTG terminologies in more detail later.
Understanding them will be important in understanding how to use MRTG to track MIB
values.


9. Testing Your MIB Value


        34
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



Once you have identified an interesting MIB value for your Linux system you can then use
the snmpwalk command to poll it. Many times the text aliases in a MIB only reference the
OID branch and not the OID the data located in a leaf ending in an additional number like a
".0" or ".1". The snmpget command doesn't work with branches giving an error stating that
the MIB variable couldn't be found.
In the example below, the ssCpuRawUser OID alias was found to be interesting, but the
snmpget command fails to get a value. Follow up with the snmpwalk command shows that
the value is located in ssCpuRawUser.0 instead. The snmpget is then successful in retrieving
the "counter32" type data with a current value of 396271.

  [root@bigboy tmp]# snmpget -v1 -c craz33guy localhost ssCpuRawUser
  Error in packet
  Reason: (noSuchName) There is no such variable name in this MIB.
  Failed object: UCD-SNMP-MIB::ssCpuRawUser
  [root@bigboy tmp]#


  [root@bigboy tmp]# snmpwalk -v1 -c craz33guy localhost ssCpuRawUser
  UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 396241
  [root@bigboy tmp]# snmpget -v1 -c craz33guy localhost ssCpuRawUser.0
  UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 396271
  [root@bigboy tmp]#

The MIB values that work successfully with snmpget are the ones you should use with
MRTG.


10. Differences In MIB And MRTG Terminology
Always keep in mind that MRTG refers to MIB counter values as counter values. It refers to
MIB integer and gauge values as gauge. By default, MRTG considers all values to be
counters.
MRTG doesn't plot counter values as a constantly increasing graph, it plots only how much
the value has changed since the last polling cycle. CPU usage is typically tracked by MIBs as
a counter value; fortunately, you can edit your MRTG configuration file to make it graph this
information in a percentage use format (more on this later).



     35
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



The syntax type, the MIB object name, and the description of what it does are the most
important things you need to know when configuring MRTG; I'll come back to these later.




11. The CPU And Memory Monitoring MIB
The UCD-SNMP-MIB MIB keeps track of a number of key performance MIB objects,
including the commonly used ones in Table 3.




11.1 Table 3: Important Objects In The UCD-SNMP-MIB MIB



UCD-SNMP-MIB               MIB         MRTG
                                                    Description
Object Variable            Type        Type
ssCpuRawUser               Counter Counter          Total CPU usage by applications run by
                                                    nonprivileged users since the system booted.
                                                    Adding the user, system, and nice values can
                                                    give a good approximation of total CPU usage..
ssCpuRawSystem             Counter Counter          Total CPU usage by applications run by
                                                    privileged system processes since the system
                                                    booted.
ssCpuRawNice               Counter Counter          Total CPU usage by applications running at a
                                                    nondefault priority level.
ssCpuRawIdle               Counter Counter          The percentage of the time the CPU is running
                                                    idle. Subtracting this value from 100 can give a
                                                    good approximation of total CPU usage.
memAvailReal               Integer     Gauge        Available Physical Memory Space On The Host




12. The TCP/IP Monitoring MIB


     36
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



The TCP-MIB MIB keeps track of data connection information and contains the very useful
tcpActiveOpens and tcpCurrEstab objects. Table 3 details the most important objects in TCP-
MIB.




Table 12.1: Important Objects In The TCP-MIB MIB


UCD-SNMP-MIB Object MIB                          MRTG
                                                               Description
Variable                           Type          Type
tcpActiveOpens                     Counter       Counter       Measures the number of completed TCP
                                                               connections.
tcpCurrEstab                       Gauge         Gauge         Measures        the    number       of    TCP
                                                               connections in the established state
tcpInErrs                          Counter       Counter       Total number of TCP segments with bad
                                                               checksum errors




13. Manually Configuring Your MRTG File
The MRTG cfgmaker program creates configuration files for network interfaces only,
simultaneously tracking two OIDs: the NIC's input and output data statistics. The mrtg
program then uses these configuration files to determine the type of data to record in its data
directory. The indexmaker program also uses this information to create the overview, or
Summary View Web page for the MIB OIDs you're monitoring.
This Summary View page shows daily statistics only. You have to click on the Summary
View graphs to get the Detailed View page behind it with the daily, weekly, monthly, and
annual graphs. Some of the parameters in the configuration file refer to the Detailed View,
others refer to the Summary View.
If you want to monitor any other pairs of OIDs, you have to manually create the
configuration files, because cfgmaker isn't aware of any OIDs other than those related to a



       37
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



NIC. The mrtg and indexmaker program can be fed individual OIDs from a customized
configuration file and will function as expected if you edit the file correctly.




13.1 Parameter Formats
MRTG configuration parameters are always followed by a graph name surrounded by square
brackets and a colon. The format looks like this:

  Parameter[graph name]: value



For ease of editing, the parameters for a particular graph are usually grouped together. Each
graph can track two OIDS listed in the Target parameter, which is usually placed at the very
top of the graph name list. The two OID values are separated by an & symbol; the first one
can be is the input OID, and the second one is the output OID.


13.2 Legend Parameters
On the Detailed View Web page, each graph has a legend that shows the max, average, and
current values of the graph's OID statistics. You can use the legendI parameter for the
description of the input graph (first graph OID) and the legendO for the output graph (second
graph OID).
The space available under each graph's legend is tiny so MRTG also has legend1 and legend2
parameters that are placed at the very bottom of the page to provide more details. Parameter
legend1 is the expansion of legendI, and legend2 is the expansion of legendO.
The Ylegend is the legend for the Y axis, the value you are trying to compare. In the case of a
default MRTG configuration this would be the data flow through the interface in bits or bytes
per second. Here is an example of the legends of a default MRTG configuration:

  YLegend[graph1]: Bits per second
  Legend1[graph1]: Incoming Traffic in Bits per Second
  Legend2[graph1]: Outgoing Traffic in Bits per Second


     38
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




   LegendI[graph1]: In
   LegendO[graph1]: Out

You can prevent MRTG from printing the legend at the bottom of the graph by leaving the
value of the legend blank like this:

   LegendI[graph1]:



13.3 Options Parameters
Options parameters provide MRTG with graph formatting information. The growright option
makes sure the data at the right of the screen is for the most current graph values. This
usually makes the graphs more intuitively easy to read. MRTG defaults to growing from the
left.
The nopercent option prevents MRTG from printing percentage style statistics in the legends
at the bottom of the graph. The gauge option alerts MRTG to the fact that the graphed values
are of the gauge type. If the value you are monitoring is in bytes, then you can convert the
output to bits using the bits option. Likewise, you can convert per second values to per
minute graphs using the perminute option. Here are some examples for two different graphs:

   options[graph1]: growright,nopercent,perminute


   options[graph2]: gauge,bits



If you place this parameter at the top with a label of [_] it gets applied to all the graphs
defined in the file. Here's an example.

   options[_]: growright



13.4 Title Parameters
The title on the Summary Page is provided by the Title parameter, the PageTop parameter
tells the title for the Detailed View page. The PageTop string must start with < H1 > and end
with < H1 >.

        39
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  Title[graph1]: Interface eth0


  PageTop[graph1]: < H1 >Detailed Statistics For Interface eth0 < H1 >




13.4 Scaling Parameters
The MaxBytes parameter is the maximum amount of data MRTG will plot on a graph.
Anything more than this seems to disappear over the edge of the graph.
MRTG also tries to adjust its graphs so that the largest value plotted on the graph is always
close to the top. This is so even if you set the MaxBytes parameter.
When you are plotting a value that has a known maximum and you always want to have this
value at the top of the vertical legend, you may want to turn off MRTG's auto scaling. If you
are plotting percentage CPU usage, and the server reaches a maximum of 60%, with scaling,
MRTG will have a vertical plot of 0% to 60%, so that the vertical peak is near the top of the
graph image.
When scaling is off, and MaxBytes is set to 100, then the peak will be only 60% of the way
up as the graph plots from 0% to 100%. The example removes scaling from the yearly,
monthly, weekly, and daily views on the Detailed View page and gives them a maximum
value of 100.

  Unscaled[graph1]: ymwd
  MaxBytes[graph1]: 100



13.5 Defining The MIB Target Parameters
As stated before, MRTG always tries to compare two MIB OID values that are defined by the
Target parameter. You have to specify the two MIB OID objects, the SNMP password and
the IP address of the device you are querying in this parameter, and separate them with an &
character:

  Target[graph1]: mib-object-1.0&mib-object-2.0:<SNMP-password>@<IP-address>


     40
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




The numeric value, in this case .0, at the end of the MIB is required. The next example uses
the SNMP command to return the user mode CPU utilization of a Linux server. Notice how
the .0 is tagged onto the end of the output.

   [root@silent mibs]# snmpwalk -v 1 -c craz33guy localhost ssCpuRawUser
   UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 926739
   [root@silent mibs]#

The MRTG legends map to the MIBs listed in the target as shown in Table 5.


Table 13.1: Mapping MIBs To The Graph Legends


                                  Legend       Maps To Target MIB
                                  Legend1 #1
                                  Legend2 #2
                                  LegendI      #1
                                  LegendO #2


So in the example below, legend1 and legendI describe mib-object-1.0 and legend2 and
legendO describe mib-object-2.0.

  Target[graph1]: mib-object-1.0&mib-object-2.0:<SNMP-password>@<IP-address>



13.5.2 Plotting Only One MIB Value
If you want to plot only one MIB value, you can just repeat the target MIB in the definition as
in the next example, which plots only mib-object-1. The resulting MRTG graph actually
superimposes the input and output graphs one on top of the other.

  Target[graph1]: mib-object-1.0&mib-object-1.0:<SNMP-password>@<IP-address>




     41
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




13.5.3 Adding MIB Values Together For a Graph
You can use the plus sign between the pairs of MIB object values to add them together. The
next example adds mib-object-1.0 and mib-object-3.0 for one graph and adds mib-object-2.0
and mib-object-4.0 for the other.

  Target[graph1]: mib-object-1.0&mib-object-2.0:<SNMP-password>@<IP-address> +
  mib-object-3.0&mib-object-4.0:<SNMP-password>@<IP-address>

You can use other mathematical operators, such as subtract (-), multiply (*), and divide (%).
Left and right parentheses are also valid. There must be white spaces before and after all
these operators for MRTG to work correctly. If not, you'll get oddly shaded graphs.


13.5.4 Target: Total CPU Usage
Linux CPU usage is occupied by system processes, user mode processes, and a few processes
running in nice mode. This example adds them all together in a single plot.

  Target[graph1]:ssCpuRawUser.0&ssCpuRawUser.0:<SNMP-password>@<IP-address>
  +        ssCpuRawSystem.0&ssCpuRawSystem.0:<SNMP-password>@<IP-address>                               +
  ssCpuRawNice.0&ssCpuRawNice.0:<SNMP-password>@<IP-address>

Be sure to place this command on a single line


13.5.5 Target: Memory Usage
Here is an example for the plotting the amount of free memory versus the total RAM installed
in the server. Notice that this is a gauge type variable.

  Target[graph1]: memAvailReal.0&memTotalReal.0:<SNMP-password>@<IP-address>
  options[graph1]: nopercent,growright,gauge



Next, plot the percentage of available memory. Notice how the mandatory white spaces
separate the mathematical operators from the next target element.


      42
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  Target[graph1]:        (   memAvailReal.0&           memAvailReal.0:<SNMP-password>@<IP-
  Address> ) * 100 / ( memTotalReal.0&memTotalReal.0:<SNMP-password>@<IP-
  Address> )
  options[graph1]: nopercent,growright,gauge




13.5.6 Target: Newly Created Connections
HTTP traffic caused by Web browsing usually consists of many very short lived connections.
The tcpPassiveOpens MIB object tracks newly created connections and is suited for this
type of data transfer. The tcpActiveOpens MIB object monitors new connections originating
from the server. On smaller Web sites you may want to use the perminute option to make the
graphs more meaningful.

  Target[graph1]:        tcpPassiveOpens.0&         tcpPassiveOpens.0:<SNMP-password>@<IP-
  address>
  MaxBytes[graph1]: 1000000
  Options[graph1]: perminute



13.5.7 Target: Total TCP Established Connections
Other protocols such as FTP and SSH create longer established connections while people
download large files or stay logged into the server. The tcpCurrEstab MIB object measures
the total number of connections in the established state and is a gauge value.

  Target[graph1]: tcpCurrEstab.0&tcpCurrEstab.0:<SNMP-password>@<IP-address>
  MaxBytes[graph1]: 1000000
  Options[graph1]: gauge



13.5.8 Target: Disk Partition Usage
In this example, you'll monitor the /var and /home disk partitions on the system.
1) First use the df -k command to get a list of the partitions in use.


     43
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  [root@bigboy tmp]# df -k
  Filesystem            1K-blocks       Used Available Use% Mounted on
  /dev/hda8               505605     128199      351302 27% /
  /dev/hda1               101089      19178      76692 21% /boot
  /dev/hda5              1035660      122864     860188 13% /home
  /dev/hda6               505605      8229     471272 2% /tmp
  /dev/hda3              3921436      890092 2832140 24% /usr
  /dev/hda2              1510060      171832 1261520 73% /var
  [root@bigboy tmp]#



2) Add two entries to your snmpd.conf file.

  disk /home
  disk /var



3) Restart the SNMP daemon to reload the values.

  [root@bigboy tmp]# service snmpd restart



4) Use the snmpwalk command to query the the dskPercent MIB. Object dskPercent.1 refers
to the first disk entry in snmpd.conf (/home), and dskPercent.2 refers to the second (/var).

  [root@bigboy tmp]# snmpwalk -v 1 -c craz33guy localhost dskPercent.1
  UCD-SNMP-MIB::dskPercent.1 = INTEGER: 13
  [root@bigboy tmp]# snmpwalk -v 1 -c craz33guy localhost dskPercent.2
  UCD-SNMP-MIB::dskPercent.2 = INTEGER: 73
  [root@bigboy tmp]#



Your MRTG target for these gauge MIB objects should look like this:

  Target[graph1]: dskPercent.1& dskPercent.1:<SNMP-password>@<IP-address>



     44
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  options[graph1]: growright,gauge




13.5.9 Defining Global Variables
You have to make sure MRTG knows where the MIBs you're using are located. The default
location MRTG uses may not be valid. Specify their locations with the global LoadMIBs
parameter. You must also define where the HTML files will be located; the example specifies
the default Fedora MRTG HTML directory.

  LoadMIBs:          /usr/share/snmp/mibs/UCD-SNMP-MIB.txt,                /usr/share/snmp/mibs/TCP-
  MIB.txt
  workdir: /var/www/mrtg/



14 Implementing Advanced Server Monitoring
You now can combine all you have learned to create a configuration file that monitors all
these variables, and then you can integrate it into the existing MRTG configuration.
14.1 A Complete Sample Configuration
Here is a sample configuration file that is used to query server localhost for CPU, memory,
disk, and TCP connection information.

  #
  # File: /etc/mrtg/server-info.cfg
  #
  # Configuration file for non bandwidth server statistics
  #


  #
  # Define global options
  #




      45
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




LoadMIBs:             /usr/share/snmp/mibs/UCD-SNMP-MIB.txt,/usr/share/snmp/mibs/TCP-
MIB.txt
workdir: /var/www/mrtg/




#
# CPU Monitoring
# (Scaled so that the sum of all three values doesn't exceed 100)
#


Target[server.cpu]:ssCpuRawUser.0&ssCpuRawUser.0:craz33guy@localhost                                  +
ssCpuRawSystem.0&ssCpuRawSystem.0:craz33guy@localhost                                                 +
ssCpuRawNice.0&ssCpuRawNice.0:craz33guy@localhost
Title[server.cpu]: Server CPU Load
PageTop[server.cpu]: < H1 >CPU Load - System, User and Nice Processes< /H1 >
MaxBytes[server.cpu]: 100
ShortLegend[server.cpu]: %
YLegend[server.cpu]: CPU Utilization
Legend1[server.cpu]: Current CPU percentage load
LegendI[server.cpu]: Used
LegendO[server.cpu]:
Options[server.cpu]: growright,nopercent
Unscaled[server.cpu]: ymwd




#
# Memory Monitoring (Total Versus Available Memory)
#


Target[server.memory]: memAvailReal.0&memTotalReal.0:craz33guy@localhost
Title[server.memory]: Free Memory
PageTop[server.memory]: < H1 >Free Memory< /H1 >



    46
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




MaxBytes[server.memory]: 100000000000
ShortLegend[server.memory]: B
YLegend[server.memory]: Bytes
LegendI[server.memory]: Free
LegendO[server.memory]: Total
Legend1[server.memory]: Free memory, not including swap, in bytes
Legend2[server.memory]: Total memory
Options[server.memory]: gauge,growright,nopercent
kMG[server.memory]: k,M,G,T,P,X




#
# Memory Monitoring (Percentage usage)
#
Title[server.mempercent]: Percentage Free Memory
PageTop[server.mempercent]: < H1 >Percentage Free Memory< /H1 >
Target[server.mempercent]: ( memAvailReal.0&memAvailReal.0:craz33guy@localhost
) * 100 / ( memTotalReal.0&memTotalReal.0:craz33guy@localhost )
options[server.mempercent]: growright,gauge,transparent,nopercent
Unscaled[server.mempercent]: ymwd
MaxBytes[server.mempercent]: 100
YLegend[server.mempercent]: Memory %
ShortLegend[server.mempercent]: Percent
LegendI[server.mempercent]: Free
LegendO[server.mempercent]: Free
Legend1[server.mempercent]: Percentage Free Memory
Legend2[server.mempercent]: Percentage Free Memory




#
# New TCP Connection Monitoring (per minute)
#



    47
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




Target[server.newconns]: tcpPassiveOpens.0&tcpActiveOpens.0:craz33guy@localhost
Title[server.newconns]: Newly Created TCP Connections
PageTop[server.newconns]: < H1 >New TCP Connections< /H1 >
MaxBytes[server.newconns]: 10000000000
ShortLegend[server.newconns]: c/s
YLegend[server.newconns]: Conns / Min
LegendI[server.newconns]: In
LegendO[server.newconns]: Out
Legend1[server.newconns]: New inbound connections
Legend2[server.newconns]: New outbound connections
Options[server.newconns]: growright,nopercent,perminute




#
# Established TCP Connections
#


Target[server.estabcons]: tcpCurrEstab.0&tcpCurrEstab.0:craz33guy@localhost
Title[server.estabcons]: Currently Established TCP Connections
PageTop[server.estabcons]: < H1 >Established TCP Connections< /H1 >
MaxBytes[server.estabcons]: 10000000000
ShortLegend[server.estabcons]:
YLegend[server.estabcons]: Connections
LegendI[server.estabcons]: In
LegendO[server.estabcons]:
Legend1[server.estabcons]: Established connections
Legend2[server.estabcons]:
Options[server.estabcons]: growright,nopercent,gauge




#



    48
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  # Disk Usage Monitoring
  #


  Target[server.disk]: dskPercent.1&dskPercent.2:craz33guy@localhost
  Title[server.disk]: Disk Partition Usage
  PageTop[server.disk]: < H1 >Disk Partition Usage /home and /var< /H1 >
  MaxBytes[server.disk]: 100
  ShortLegend[server.disk]: %
  YLegend[server.disk]: Utilization
  LegendI[server.disk]: /home
  LegendO[server.disk]: /var
  Options[server.disk]: gauge,growright,nopercent
  Unscaled[server.disk]: ymwd



14.2 Testing The Configuration
The next step is to test that MRTG can load the configuration file correctly.
Restart SNMP to make sure the disk monitoring commands in the snmpd.conf file are
activated. Run the /usr/bin/mrtg command followed by the name of the configuration file
three times. If all goes well, MRTG will complain only about the fact that certain database
files don't exist. MRTG then creates the files. By the third run, all the files are created and
MRTG should operate smoothly.

  [root@bigboy tmp]# service snmpd restart
  [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/server-stats.cfg




14.3 Creating A New MRTG Index Page To Include This File
Use the indexmaker command and include your original MRTG configuration file from
Chapter 22, " Monitoring Server Performance", (/etc/mrtg/mrtg.cfg) plus the new one you
created (/etc/mrtg/server-stats.cfg).



      49
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  [root@bigboy tmp]# indexmaker --output=/var/www/mrtg/index.html 
  /etc/mrtg/mrtg.cfg /etc/mrtg/server-stats.cfg



14.4 Configuring cron To Use The New MRTG File
The final step is to make sure that MRTG is configured to poll your server every five minutes
using this new configuration file. To do so, add this line to your /etc/cron.d/mrtg file.

  0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/server-stats.cfg

Some versions of Linux require you to edit your /etc/crontab file instead. See Chapter 22,
" Monitoring Server Performance", for more details. You will also have to restart cron with
the service crond restart for it to read its new configuration file that tells it to additionally run
MRTG every five minutes using the new MRTG configuration file.

  [root@bigboy tmp]# service crond restart



15 Monitoring Non Linux MIB Values
All the MIBs mentioned so far are for Linux systems; other types of systems will need
additional MIBs whose correct installation may be unclear in user guides or just not
available. In such cases, you'll need to know the exact value of the OID.
15.1 Scenario
Imagine that your small company has purchased a second-hand Cisco switch to connect its
Web site servers to the Internet. The basic MRTG configuration shown in Chapter 22,
" Monitoring Server Performance", provides the data bandwidth statistics, but you want to
measure the CPU load the traffic is having on the device, as well. Downloading MIBs from
Cisco and using them with the snmpget command was not a success. You do not know what
to do next. Find The OIDs
When MIB values fail, it is best to try to find the exact OID value. Like most network
equipment manufacturers, Cisco has an FTP site from which you can download both MIBs
and OIDs. The SNMP files for Cisco's devices can be found at ftp.cisco.com in the /pub/mibs
directory; OIDs are in the oid directory beneath that.


     50
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



After looking at all the OID files, you decide that the file CISCO-PROCESS-MIB.oid will
contain the necessary values and find these entries inside it.

  "cpmCPUTotalPhysicalIndex" "1.3.6.1.4.1.9.9.109.1.1.1.1.2"
  "cpmCPUTotal5sec"                "1.3.6.1.4.1.9.9.109.1.1.1.1.3"
  "cpmCPUTotal1min"                 "1.3.6.1.4.1.9.9.109.1.1.1.1.4"
  "cpmCPUTotal5min"                 "1.3.6.1.4.1.9.9.109.1.1.1.1.5"
  "cpmCPUTotal5secRev"                "1.3.6.1.4.1.9.9.109.1.1.1.1.6"
  "cpmCPUTotal1minRev"                 "1.3.6.1.4.1.9.9.109.1.1.1.1.7"
  "cpmCPUTotal5minRev"                 "1.3.6.1.4.1.9.9.109.1.1.1.1.8"




15.2 Testing The OIDs
As you can see, all the OIDs are a part of the same tree starting with
1.3.6.1.4.1.9.9.109.1.1.1.1. The OIDs provided may be incomplete, so it is best to use the
snmpwalk command to try to get all the values below this root first.

  [root@bigboy           tmp]#        snmpwalk          -v1      -c      craz33guy         cisco-switch
  1.3.6.1.4.1..9.9.109.1.1.1.1
  SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.2.1 = INTEGER: 0
  SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.3.1 = Gauge32: 32
  SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.4.1 = Gauge32: 32
  SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.5.1 = Gauge32: 32
  [root@bigboy tmp]#



Although listed in the OID file, 1.1.1.1.6, 1.1.1.1.7, and 1.1.1.1.8 are not supported. Notice
also how SNMP has determined that the first part of the OID value (1.3.6.1.4.1) in the
original OID file maps to the word "enterprise".
Next, you can use one the snmpget command to set only one of the OID values returned by
snmpwalk.

  [root@bigboy tmp]# snmpget -v1 -c craz33guy cisco-switch 



     51
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




    enterprises.9.9.109.1.1.1.1.5.1
    SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.5.1 = Gauge32: 33
    [root@bigboy tmp]#

Success! Now you can use this OID value, enterprises.9.9.109.1.1.1.1.5.1, for your MRTG
queries.
16 Speeding up MRTG with RRDtool
MRTG is a very useful program but it has a limitation. All the graphs and web pages are
recreated each time a device is polled. This can potentially overload your MRTG server
especially if you have a large number of monitored devices and the graphs take more than
five minutes to generate. RRDtool is an application written by the creator of MRTG that can
store general purpose data, but generates graphs on demand. Integrating MRTG with
RRDtool can have very noticeable performance benefits. The example that follows will show
you how to quickly implement a general purpose solution.


16.1 Scenario
The use of RRDtool is needed to reduce the load on a monitoring server that has been
experiencing very sluggish performance due to the amount of MRTG graphs it has to
regenerate every polling cycle.
    Due to space constraints, the RRD database needs to be located in the /var partition.
    The server has a default Apache configuration with the CGI files needed for dynamically
     generated content being located in the /var/www/cgi-bin directory.
    A CGI script is required that will read the new MRTG data in RRDtool format.
    The MRTG configuration file is /etc/mrtg/mrtg.cfg.
Here's how to proceed.


16.2Installing RRDtool
The RRDtool and RRDtool PERL module file can be downloaded from its website
at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/, but installation can be tricky as the
installation program may look for certain supporting libraries in the wrong directories.




      52
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



Fortunately the prerequisite rrdtool and rrdtool-perl packages now come as part of most
Linux distributions. For more details on installing packages, see Chapter 6, "Installing Linux
Software").




16.3 Storing the MRTG Data in RRDtool Format
This phase of the integration process can be done in a few minutes, but the steps can be
tricky:
     The first step is to add some new options to your cfgmaker command. The first indicates
      that MRTG should only store rrdtool formatted data, and the second defines the /var/mrtg
      directory in which it should be stored. For added security, the directory should be external
      to your web server's document root.

     --global 'LogFormat: rrdtool' --global "workdir: /var/mrtg" --global 'IconDir: /mrtg'



Finally, you should also specify an icon directory which specifies the location of all
miscellaneous MRTG web page icons. The RRD web interface script we'll install later uses
an incorrect location. The icon directory /mrtg is actually a partial URL location. In this
Fedora scenario we are using the default Apache configuration which locates the MRTG icon
files in the /var/www/mrtg directory. If you are using a non default Apache MRTG
configuration or are using other Linux distributions or versions you may have to copy the
icons to the custom directory in which the MRTG PNG format icon files are located.
".
The next step is to create the data repository directory /var/mrtg and make it be owned by the
apache user and process that runs the default Linux web server application.

     [root@bigboy tmp]# mkdir /var/mrtg
     [root@bigboy tmp]# chown apache /var/mrtg
     [root@bigboy tmp]#




       53
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



Note: If you are using SELinux you'll have to change the context of this directory to match
that of the /var/www/html directory so that the apache process will be able to read the
database files when your CGI script needs them. These commands compare the contexts of
the both directories and apply the correct set to /var/mrtg.

      [root@bigboy tmp]# ls -alZ /var/www | grep html
      drwxr-xr-x root        root     system_u:object_r:httpd_sys_content_t html
      [root@bigboy tmp]# ls -alZ /var | grep mrtg
      drwxr-xr-x apache root           root:object_r:var_t            mrtg
      [root@bigboy tmp]# chcon -R -u system_u -r object_r -t httpd_sys_content_t /var/mrtg
      [root@bigboy tmp]#



We now need to test that the RRD files are being created correctly. Run MRTG using the
/etc/mrtg/mrtg.cfg file as the source configuration file then test to see if the contents of the /
var/mrtg directory have changed. Success!

      [root@bigboy tmp]# ls /var/mrtg/
      localhost_192.168.1.100.rrd
      [root@bigboy tmp]#



The files are being created properly. Now we need to find a script to read the new data format
and present it in a web format. This will be discussed next.


16.4 The MRTG / RRDtool Integration Script
The      MRTG         website       recommends     the    script located      on    the mrtg-rrd          website
(http://www.fi.muni.cz/~kas/mrtg-rrd/) as being a good one to use. Let's go ahead and install
it.
Download the script using wget. The site lists several versions; make sure you get the latest
one.

      [root@bigboy        tmp]#       wget    ftp://ftp.linux.cz/pub/linux/people/jan_kasprzak/mrtg-
      rrd/mrtg-rrd-0.7.tar.gz



        54
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




  --12:42:12--                      ftp://ftp.linux.cz/pub/linux/people/jan_kasprzak/mrtg-rrd/mrtg-
  rrd-0.7.tar.gz
              => `mrtg-rrd-0.7.tar.gz'
  Resolving ftp.linux.cz... 147.251.48.205
  Connecting to ftp.linux.cz|147.251.48.205|:21... connected.
  Logging in as anonymous ... Logged in!
  ...
  ...
  ...
  15:24:50 (53.53 KB/s) - `mrtg-rrd-0.7.tar.gz' saved [20863]
  [root@bigboy tmp]# ls
  mrtg-rrd-0.7.tar.gz
  [root@bigboy tmp]#



Extract the contents of the tar file.

  [root@bigboy tmp]# tar -xzvf mrtg-rrd-0.7.tar.gz
  mrtg-rrd-0.7/
  mrtg-rrd-0.7/COPYING
  mrtg-rrd-0.7/FAQ
  mrtg-rrd-0.7/TODO
  mrtg-rrd-0.7/Makefile
  mrtg-rrd-0.7/mrtg-rrd.cgi
  mrtg-rrd-0.7/ChangeLog
  [root@bigboy tmp]#



Create the /var/www/cgi-bin/mrtg directory and copy the mrtg-rrd.cgi file to it.

  [root@bigboy tmp]# mkdir -p /var/www/cgi-bin/mrtg
  [root@bigboy tmp]# cp mrtg-rrd-0.7/mrtg-rrd.cgi /var/www/cgi-bin/mrtg/
  [root@bigboy tmp]#




        55
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



Edit the mrtg-rrd.cgi file and make it refer to the /etc/mrtg/mrtg.cfg file for its configuration
details, or you can specify all the .cfg files in your /etc/mrtg directory.

  #
  # File: mrtg-rrd.cgi (Single File)
  #


  # EDIT THIS to reflect all your MRTG config files
  BEGIN { @config_files = qw(/etc/mrtg/mrtg.cfg); }
  #
  # File: mrtg-rrd.cgi (multipl .cfg files)
  #


  # EDIT THIS to reflect all your MRTG config files
  BEGIN { @config_files = </etc/mrtg/*.cfg>; }



Once installed, RRDtool operates transparently with MRTG. You'll have to remember to add
the RRD statements to any new MRTG configurations and also add the configuration file to
the CGI script. Our monitoring server can now breathe a little easier.


17 Troubleshooting
The troubleshooting techniques for advanced MRTG ", but because you have done some
customizations you'll have to go the extra mile.
 Verify the IP address and community string of the target device you intend to poll.
 Make sure you can do an SNMP walk of the target device. If not, revise your access
    controls on the target device and any firewall rules that may impede SNMP traffic.
 Ensure you can do an SNMP get of the specific OID value listed in your MRTG
    configuration file.
 Check your MRTG parameters to make sure they are correct. Gauge values defined as
    counter and vice versa will cause your graphs to have continuous zero values. Graph
    results that are eight times what you expect may have the bits parameter set.
 There are a few errors common to initial RRDtool integration.



      56
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure



 Web messages like this where the reference to the MRTG configuration file in the CGI
   13.5.2script was incorrect

  Error: Cannot open config file: No such file or directory



"Permission Denied" web messages are usually caused by incorrect file permissions and / or
SELinux contexts

  Error: RRDs::graph failed, opening '/var/mrtg/localhost_192.168.1.100.rrd': Permission
  denied



Errors in the /var/log/httpd/errorlog file referring to files or directories that don't exist can
be caused by an incorrect IconDir statement in the MRTG configuration file.

  [Wed Jan 04 15:42:13 2006] [error] [client 192.168.1.102] File does not exist:
  /var/www/html/var,
  referer: http://bigboy/cgi-bin/mrtg/mrtg-rrd.cgi/


  [Wed Jan 04 15:45:46 2006] [error] [client 192.168.1.102] script not found or unable to
  stat:
   /var/www/cgi-bin/mrtg/mrtg-l.png, referer: http://bigboy/cgi-bin/mrtg/mrtg-rrd.cgi/



Errors caused by not installing the pre-requisite RRD RPM modules rrdtool, perl-RRD-
Simple and rrdtool-perl.

  ERROR: could not find RRDs.pm. Use LibAdd: in mrtg.cfg to help mrtg find RRDs.pm

These quick steps should be sufficient in most cases and will reward you with a more
manageable network.




     57
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




                                           CONCLUSION


By this Project we are able to graph most SNMP MIB values available on any type of device.
MRTG is an excellent, flexible monitoring tool and should be considered as a part of any
systems administrator's server management plans.
Server monitoring is always a good practice, because it can help you predict when things are
going to go wrong or long term trends in your Web traffic.
MRTG can be expanded not only to monitor traffic on your server's NIC cards, but also to
graph many of the statistics listed in top, free, and vmstat




     58
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




                                           REFERENCES


CMU-SNMP for Linux:
ftp://sunsite.unc.edu/pub/Linux/system/network/admin/cmu-snmp-linux-3.2-
bin.tar.gz
PERL 5 module extension for CMU-SNMP:
ftp:/ftp.wellfleet.com/netman/snmp/perl5/SNMP.tar.gz
Multi Router Traffic Grapher (MRTG):
http://www.ee.ethz.ch/~oetiker/webtools/mrtg/mrtg.html
GD Graphic Library: (required by MRTG)
http://www.boutell.com/gd/
Perl 5.003: (required by MRTG)
http://www.perl.com/perl/info/software.html
MRTG mailing lists:
Discussion list

Router Stats:
http://www.scn.de/~iain/router-stats/
SNMP Network Management Software:
http://wwwsnmp.cs.utwente.nl/software/
Scotty:
http://wwwsnmp.cs.utwente.nl/~schoenw/scotty/
Scotty for Linux:
 ftp://sunsite.unc.edu/pub/Linux/system/network/admin/tkined-1.3.4+scotty-
                                 ELF.tar.gz


     59
Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure




60

More Related Content

What's hot

Memory management in vx works
Memory management in vx worksMemory management in vx works
Memory management in vx worksDhan V Sagar
 
Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...
Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...
Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...CSCJournals
 
NetSim Technology Library - Software defined networks
NetSim Technology Library - Software defined networksNetSim Technology Library - Software defined networks
NetSim Technology Library - Software defined networksVishal Sharma
 
Processes and Thread OS_Tanenbaum_3e
Processes and Thread OS_Tanenbaum_3eProcesses and Thread OS_Tanenbaum_3e
Processes and Thread OS_Tanenbaum_3eLe Gia Hoang
 
dos mutual exclusion algos
dos mutual exclusion algosdos mutual exclusion algos
dos mutual exclusion algosAkhil Sharma
 
Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridad
 Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridad Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridad
Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridadMarketing Donalba
 
Ppt project process migration
Ppt project process migrationPpt project process migration
Ppt project process migrationjaya380
 
Cryptography Processing with 3rd Gen Intel Xeon Scalable Processors
Cryptography Processing with 3rd Gen Intel Xeon Scalable ProcessorsCryptography Processing with 3rd Gen Intel Xeon Scalable Processors
Cryptography Processing with 3rd Gen Intel Xeon Scalable ProcessorsDESMOND YUEN
 
IRJET- Collaborative Network Security in Data Center for Cloud Computing
IRJET-  	  Collaborative Network Security in Data Center for Cloud ComputingIRJET-  	  Collaborative Network Security in Data Center for Cloud Computing
IRJET- Collaborative Network Security in Data Center for Cloud ComputingIRJET Journal
 
IEC 60870-5 101 Protocol Server Simulator User manual
IEC 60870-5 101 Protocol Server Simulator User manualIEC 60870-5 101 Protocol Server Simulator User manual
IEC 60870-5 101 Protocol Server Simulator User manualFreyrSCADA Embedded Solution
 
Clock Synchronization in Distributed Systems
Clock Synchronization in Distributed SystemsClock Synchronization in Distributed Systems
Clock Synchronization in Distributed SystemsZbigniew Jerzak
 
MULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMS
MULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMSMULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMS
MULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMSvtunotesbysree
 
Tools for analysis and evaluation of CPU Performance
Tools for analysis and evaluation of CPU PerformanceTools for analysis and evaluation of CPU Performance
Tools for analysis and evaluation of CPU PerformanceMourad Bouache
 

What's hot (20)

Memory management in vx works
Memory management in vx worksMemory management in vx works
Memory management in vx works
 
Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...
Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...
Analysis of Practicality and Performance Evaluation for Monolithic Kernel and...
 
iPDC User Manual
iPDC User ManualiPDC User Manual
iPDC User Manual
 
NetSim Technology Library - Software defined networks
NetSim Technology Library - Software defined networksNetSim Technology Library - Software defined networks
NetSim Technology Library - Software defined networks
 
Processes and Thread OS_Tanenbaum_3e
Processes and Thread OS_Tanenbaum_3eProcesses and Thread OS_Tanenbaum_3e
Processes and Thread OS_Tanenbaum_3e
 
dos mutual exclusion algos
dos mutual exclusion algosdos mutual exclusion algos
dos mutual exclusion algos
 
Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridad
 Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridad Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridad
Procesamiento multinúcleo óptimo para aplicaciones críticas de seguridad
 
Thesis
ThesisThesis
Thesis
 
Ppt project process migration
Ppt project process migrationPpt project process migration
Ppt project process migration
 
Pc hardware
Pc hardwarePc hardware
Pc hardware
 
Cryptography Processing with 3rd Gen Intel Xeon Scalable Processors
Cryptography Processing with 3rd Gen Intel Xeon Scalable ProcessorsCryptography Processing with 3rd Gen Intel Xeon Scalable Processors
Cryptography Processing with 3rd Gen Intel Xeon Scalable Processors
 
Priority Inversion on Mars
Priority Inversion on MarsPriority Inversion on Mars
Priority Inversion on Mars
 
Hfc
HfcHfc
Hfc
 
419 426
419 426419 426
419 426
 
IRJET- Collaborative Network Security in Data Center for Cloud Computing
IRJET-  	  Collaborative Network Security in Data Center for Cloud ComputingIRJET-  	  Collaborative Network Security in Data Center for Cloud Computing
IRJET- Collaborative Network Security in Data Center for Cloud Computing
 
IEC 60870-5 101 Protocol Server Simulator User manual
IEC 60870-5 101 Protocol Server Simulator User manualIEC 60870-5 101 Protocol Server Simulator User manual
IEC 60870-5 101 Protocol Server Simulator User manual
 
Clock Synchronization in Distributed Systems
Clock Synchronization in Distributed SystemsClock Synchronization in Distributed Systems
Clock Synchronization in Distributed Systems
 
MULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMS
MULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMSMULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMS
MULTIPLE CHOICE QUESTIONS WITH ANSWERS ON NETWORK MANAGEMENT SYSTEMS
 
be_report - report
be_report - reportbe_report - report
be_report - report
 
Tools for analysis and evaluation of CPU Performance
Tools for analysis and evaluation of CPU PerformanceTools for analysis and evaluation of CPU Performance
Tools for analysis and evaluation of CPU Performance
 

Viewers also liked

The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...
The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...
The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...University of Wolverhampton
 
introduction to networking
introduction to networkingintroduction to networking
introduction to networkingRobert Almazan
 
introduction to Networking
introduction to Networkingintroduction to Networking
introduction to Networkingiicecollege
 
Introduction to networking
Introduction to networkingIntroduction to networking
Introduction to networkingZaeem Hassan
 
introduction to networking
introduction to networking introduction to networking
introduction to networking asif078
 
Final project report format
Final project report formatFinal project report format
Final project report formatMasud Sarkar
 
Introduction To Networking
Introduction To NetworkingIntroduction To Networking
Introduction To NetworkingSteven Cahill
 
Introduction To Networking
Introduction To NetworkingIntroduction To Networking
Introduction To NetworkingRaul Rupsingh
 

Viewers also liked (10)

The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...
The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...
The characteristics of a Knowledge Exchange and Enterprise Network (KEEN) pro...
 
Networking fundamental
Networking fundamentalNetworking fundamental
Networking fundamental
 
introduction to networking
introduction to networkingintroduction to networking
introduction to networking
 
introduction to Networking
introduction to Networkingintroduction to Networking
introduction to Networking
 
Introduction to networking
Introduction to networkingIntroduction to networking
Introduction to networking
 
introduction to networking
introduction to networking introduction to networking
introduction to networking
 
Final project report format
Final project report formatFinal project report format
Final project report format
 
Introduction To Networking
Introduction To NetworkingIntroduction To Networking
Introduction To Networking
 
Introduction To Networking
Introduction To NetworkingIntroduction To Networking
Introduction To Networking
 
Computer Network
Computer NetworkComputer Network
Computer Network
 

Similar to Centralized Monitoring Station Project Report

Introduction tosnmp
Introduction tosnmpIntroduction tosnmp
Introduction tosnmpjorlugon
 
Nagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios Plugins
Nagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios PluginsNagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios Plugins
Nagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios PluginsNagios
 
Simple Network Management Protocole
Simple Network Management ProtocoleSimple Network Management Protocole
Simple Network Management ProtocoleAmin Komeili
 
Snmp chapter7
Snmp chapter7Snmp chapter7
Snmp chapter7jignash
 
IoT ( M2M) - Big Data - Analytics: Emulation and Demonstration
IoT ( M2M) - Big Data - Analytics: Emulation and DemonstrationIoT ( M2M) - Big Data - Analytics: Emulation and Demonstration
IoT ( M2M) - Big Data - Analytics: Emulation and DemonstrationCHAKER ALLAOUI
 
CCNA4v5 Chapter 8 - Monitoring the Netwok
CCNA4v5 Chapter 8 - Monitoring the NetwokCCNA4v5 Chapter 8 - Monitoring the Netwok
CCNA4v5 Chapter 8 - Monitoring the NetwokAhmed Gad
 
FALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.ppt
FALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.pptFALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.ppt
FALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.pptuseonlyfortech140
 
Running head network design 1 netwo
Running head network design                             1 netwoRunning head network design                             1 netwo
Running head network design 1 netwoAKHIL969626
 
Data Acquisition and Control System for Real Time Applications
Data Acquisition and Control System for Real Time ApplicationsData Acquisition and Control System for Real Time Applications
Data Acquisition and Control System for Real Time Applicationsijsrd.com
 
BSA 385 Week 3 Individual Assignment Essay
BSA 385 Week 3 Individual Assignment EssayBSA 385 Week 3 Individual Assignment Essay
BSA 385 Week 3 Individual Assignment EssayTara Smith
 
Centralized monitoring station for it computing and network infrastructure
Centralized monitoring station for it computing and network infrastructureCentralized monitoring station for it computing and network infrastructure
Centralized monitoring station for it computing and network infrastructureMOHD ARISH
 
Network Monitoring System ppt.pdf
Network Monitoring System ppt.pdfNetwork Monitoring System ppt.pdf
Network Monitoring System ppt.pdfkristinatemen
 
network monitoring system ppt
network monitoring system pptnetwork monitoring system ppt
network monitoring system pptashutosh rai
 
Chapter-2.pdf
Chapter-2.pdfChapter-2.pdf
Chapter-2.pdfvenui2
 

Similar to Centralized Monitoring Station Project Report (20)

Introduction tosnmp
Introduction tosnmpIntroduction tosnmp
Introduction tosnmp
 
Nagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios Plugins
Nagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios PluginsNagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios Plugins
Nagios Conference 2013 - William Leibzon - SNMP Protocol and Nagios Plugins
 
snmp
snmpsnmp
snmp
 
Simple Network Management Protocole
Simple Network Management ProtocoleSimple Network Management Protocole
Simple Network Management Protocole
 
Net Man
Net ManNet Man
Net Man
 
Snmp chapter7
Snmp chapter7Snmp chapter7
Snmp chapter7
 
Snmp
SnmpSnmp
Snmp
 
IoT ( M2M) - Big Data - Analytics: Emulation and Demonstration
IoT ( M2M) - Big Data - Analytics: Emulation and DemonstrationIoT ( M2M) - Big Data - Analytics: Emulation and Demonstration
IoT ( M2M) - Big Data - Analytics: Emulation and Demonstration
 
CCNA4v5 Chapter 8 - Monitoring the Netwok
CCNA4v5 Chapter 8 - Monitoring the NetwokCCNA4v5 Chapter 8 - Monitoring the Netwok
CCNA4v5 Chapter 8 - Monitoring the Netwok
 
Snmpv3
Snmpv3Snmpv3
Snmpv3
 
FALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.ppt
FALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.pptFALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.ppt
FALLSEM2023-24_BCSE308L_TH_VL2023240100841_2023-07-10_Reference-Material-II.ppt
 
Ip Subnet Design
Ip Subnet DesignIp Subnet Design
Ip Subnet Design
 
Running head network design 1 netwo
Running head network design                             1 netwoRunning head network design                             1 netwo
Running head network design 1 netwo
 
Unit 3 IOT.docx
Unit 3 IOT.docxUnit 3 IOT.docx
Unit 3 IOT.docx
 
Data Acquisition and Control System for Real Time Applications
Data Acquisition and Control System for Real Time ApplicationsData Acquisition and Control System for Real Time Applications
Data Acquisition and Control System for Real Time Applications
 
BSA 385 Week 3 Individual Assignment Essay
BSA 385 Week 3 Individual Assignment EssayBSA 385 Week 3 Individual Assignment Essay
BSA 385 Week 3 Individual Assignment Essay
 
Centralized monitoring station for it computing and network infrastructure
Centralized monitoring station for it computing and network infrastructureCentralized monitoring station for it computing and network infrastructure
Centralized monitoring station for it computing and network infrastructure
 
Network Monitoring System ppt.pdf
Network Monitoring System ppt.pdfNetwork Monitoring System ppt.pdf
Network Monitoring System ppt.pdf
 
network monitoring system ppt
network monitoring system pptnetwork monitoring system ppt
network monitoring system ppt
 
Chapter-2.pdf
Chapter-2.pdfChapter-2.pdf
Chapter-2.pdf
 

More from MOHD ARISH

Businese research project on mobile shops
Businese research project on mobile shopsBusinese research project on mobile shops
Businese research project on mobile shopsMOHD ARISH
 
Customers complaint pretaing to baroda connect & atm cum debit card
Customers complaint pretaing to baroda connect & atm  cum debit cardCustomers complaint pretaing to baroda connect & atm  cum debit card
Customers complaint pretaing to baroda connect & atm cum debit cardMOHD ARISH
 
bank of baroda
bank of barodabank of baroda
bank of barodaMOHD ARISH
 
Kaizen theory
Kaizen theory Kaizen theory
Kaizen theory MOHD ARISH
 
Toyata production system
Toyata production systemToyata production system
Toyata production systemMOHD ARISH
 
Theory of constraints
Theory of constraintsTheory of constraints
Theory of constraintsMOHD ARISH
 
HIndustan Unilever Project(Final)
HIndustan Unilever Project(Final)HIndustan Unilever Project(Final)
HIndustan Unilever Project(Final)MOHD ARISH
 
Capacity Planning
Capacity PlanningCapacity Planning
Capacity PlanningMOHD ARISH
 
Automated Process
Automated ProcessAutomated Process
Automated ProcessMOHD ARISH
 
Supply Chain Management
Supply Chain ManagementSupply Chain Management
Supply Chain ManagementMOHD ARISH
 
Project On Operation Management
Project On Operation ManagementProject On Operation Management
Project On Operation ManagementMOHD ARISH
 
Project On Facility Layout
Project On Facility LayoutProject On Facility Layout
Project On Facility LayoutMOHD ARISH
 
Inventory management
Inventory managementInventory management
Inventory managementMOHD ARISH
 
Inventory Management
Inventory ManagementInventory Management
Inventory ManagementMOHD ARISH
 
Inventory Management Project
Inventory Management ProjectInventory Management Project
Inventory Management ProjectMOHD ARISH
 

More from MOHD ARISH (20)

Businese research project on mobile shops
Businese research project on mobile shopsBusinese research project on mobile shops
Businese research project on mobile shops
 
Customers complaint pretaing to baroda connect & atm cum debit card
Customers complaint pretaing to baroda connect & atm  cum debit cardCustomers complaint pretaing to baroda connect & atm  cum debit card
Customers complaint pretaing to baroda connect & atm cum debit card
 
bank of baroda
bank of barodabank of baroda
bank of baroda
 
Six sigma
Six sigmaSix sigma
Six sigma
 
PPT on DELL
PPT on DELLPPT on DELL
PPT on DELL
 
Kaizen theory
Kaizen theory Kaizen theory
Kaizen theory
 
Toyota
ToyotaToyota
Toyota
 
Toyata production system
Toyata production systemToyata production system
Toyata production system
 
Theory of constraints
Theory of constraintsTheory of constraints
Theory of constraints
 
Supply chain
Supply chainSupply chain
Supply chain
 
HIndustan Unilever Project(Final)
HIndustan Unilever Project(Final)HIndustan Unilever Project(Final)
HIndustan Unilever Project(Final)
 
Capacity Planning
Capacity PlanningCapacity Planning
Capacity Planning
 
Automated Process
Automated ProcessAutomated Process
Automated Process
 
Supply Chain Management
Supply Chain ManagementSupply Chain Management
Supply Chain Management
 
Project On Operation Management
Project On Operation ManagementProject On Operation Management
Project On Operation Management
 
Dell
DellDell
Dell
 
Project On Facility Layout
Project On Facility LayoutProject On Facility Layout
Project On Facility Layout
 
Inventory management
Inventory managementInventory management
Inventory management
 
Inventory Management
Inventory ManagementInventory Management
Inventory Management
 
Inventory Management Project
Inventory Management ProjectInventory Management Project
Inventory Management Project
 

Centralized Monitoring Station Project Report

  • 1. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure A Project Report On Centralized Monitoring Station for IT Computing and Network Infrastructure for the award of the degree of BACHELOR OF TECHNOLOGY In COMPUTER SCIENCE & ENGINEERING By: Anurag Tripathi (0621610029) Abhinav Kumar Chand (0621610003) Aviral (0621610038) Indra Nath Banerjee (0521610055) Under Supervision of: Shri Rakesh Kumar Singh 1
  • 2. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure INTRODUCTION We can monitor your system's Web performance quite easily with graphical Linux tools. Here we will see how to monitor our network which has many technology, including MRTG, which is based on SNMP and monitors raw network traffic, and Webalizer, which tracks Web site hits. We are implementing a network monitoring solution for a small network (approximately 5-10 network devices). We'd like to have all devices on our network send SNMP traps to a single box (technically this will probably be an HA pair of boxes) and then have that box pass the SNMP traps on to the real processing boxes. This will allow us to have multiple back-end boxes handling traps, and to distribute load among those back end boxes. 2
  • 3. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 2. SNMP Most servers, routers and firewalls keep their operational statistics in object identifiers (OIDs) that you can remotely retrieve via the Simple Network Management Protocol (SNMP). For ease of use, equipment vendors provide Management Information Base (MIB) files for their devices that define the functions of the OIDs they contain. That's a lot of new terms to digest in two sentences, so take a moment to look more closely. 2.1 OIDs and MIBs OIDs are arranged in a structure of management information (SMI) tree defined by the SNMP standard. The tree starts from a root node, which then descends through branches and leaves that each add their own reference value to the path separated by a period.. Figure 1 shows an OID structure in which the path to the enterprises OID branch passes through the org, dod, internet, and private branches first. The OID path for enterprises is, therefore, 1.3.6.1.4.1. Management Information Bases (MIBs) are text definitions of each of the OID branches. Table 1 shows how some commonly used OIDs map to their MIB definitions. For example, the SMI org MIB defines all the topmost OIDs found at the next layer, which is named dod; the internet MIB under dod defines the function of the topmost OIDs in the directory, mgmt, experimental, and private branches. This MIB information is very useful for SNMP management programs, enabling you to click on an OID and see its value, type, and description 3
  • 4. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Figure 2.1: SNMP OID Structure Table 2.1: OIDs And Their Equivalent MIBs OID MIB 1.3 org 1.3.6 dod 1.3.6.1 internet 1.3.6.1.1 directory 1.3.6.1.2 mgmt 1.3.6.1.3 experimental 1.3.6.1.4 private 1.3.6.1.4.1 enterprises We can refer to an OID by substituting the values in a branch with one of these more readable MIB aliases. For example, you can reference the OID 1.3.6.1.4.1.9.9.109.1.1.1.1.5 as 4
  • 5. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure enterprises.9.9.109.1.1.1.1.5.1 by substituting the branch name (enterprises) for its OID numbers (1.3.6.1.4.1). Remember, only the OID value at the very tip of a branch, the leaf, actually has a readable value. Think of OIDs like the directory structure on a hard disk. Each branch is equivalent to a subdirectory, and the very last value at the tip (the leaf) correlates to a file containing data. The Linux snmpget command outputs the value of a single leaf, and the snmpwalk command provides the values of all leaves under a branch. I'll discuss these commands later; for now, all there is a need to know, that the command output frequently doesn't list the entire OID, just the MIB file in which it was found and the alias within the MIB. For example SNMPv2-MIB::sysUpTime.0 Here the OID value was found in the SNMPv2-MIB file and occupies position zero in the sysUpTime alias. Equipment manufacturers are usually assigned their own dedicated OID branch under the enterprises MIB, and they must also provide information in universally accepted OIDs for ease of manageability. For example, NIC interface data throughput values must always be placed in a predefined location in the general tree, but a memory use value on a customized processor card may be defined in a MIB under the manufacturer's own OID branch. 2.2 SNMP Community Strings As a security measure, you need to know the SNMP password, or community string, to query OIDs. There are a number of types of community strings, the most commonly used ones are the Read Only or "get" community string that only provides access for viewing statistics and system parameters. In many cases the Read Only community string or password is set to the word "public;" you should change it from this easy-to-guess value whenever possible. The Read/Write or "set" community string is for not only viewing statistics and system parameters but also for updating the parameters. 2.3 SNMP Versions There are currently three versions of SNMP. 5
  • 6. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 1. SNMP Version 1: The first version of SNMP to be implemented, version 1 was designed to be a protocol to provide device statistics and error reporting without consuming a lot of system resources. Security was limited to community strings and access controls based on the IP address of the querying server. Data communication wasn't encrypted. 2. SNMP Version 2: The second version of SNMP, often referred to as v2c, expanded the number of supported error codes, increased the size of counters used to track data, and had the ability to do bulk queries that more efficiently loaded response packets with data. SNMP v2c is backward compatible with version 1. 3. SNMP Version 3: This version provides greater security and remote configuration capabilities than its predecessors. Access isn't limited to a single community string for read-only and read/write access, as usernames and passwords have been introduced. Views of OIDs in a MIB can also be limited on a per-user basis. Support for encrypted SNMP data transfer and transfer error detection is also provided. Remember their differences, because We will need to specify the version number when doing SNMP queries. 2.4 Doing SNMP Queries Configuring SNMP on a server isn't hard, but it does require a number of detailed steps. 2.4.1 Installing SNMP Utilities on a Linux Server If you intend to use your Linux box to query your network devices, other servers or even itself using MRTG or any other tool, you need to have the SNMP utility tools package net- snmp-utils installed. This package may also require pre-requisite packages, so it is best to use an automated package updater such yum or apt to do this. When searching for these packages the filenames will start with the package name followed by a version number, as in net-snmp-utils-5.1.1-2.i386.rpm. (If you need an installation refresher, see Chapter 6, "Installing Linux Software"). 2.4.2 SNMP Utilities Command Syntax 6
  • 7. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure The SNMP utility tools package installs a number of new commands on your system for doing SNMP queries, most notably snmpget for individual OIDs and snmpwalk for obtaining the contents of an entire MIB. Both commands require you to specify the community string with a -c operator. They also require you to specify the version of the SNMP query to be used with a -v 1, -v 2c, or -v 3 operator for versions 1, 2, and 3, respectively. The first argument is the name or IP address of the target device and all other arguments list the MIBs to be queried. This example gets all the values in the interface MIB of the local server using SNMP version 1 and the community string of craz33guy. [root@bigboy tmp]# snmpwalk -v 1 -c craz33guy localhost interface ... ... IF-MIB::ifDescr.1 = STRING: lo IF-MIB::ifDescr.2 = STRING: eth0 IF-MIB::ifDescr.3 = STRING: eth1 ... ... IF-MIB::ifPhysAddress.1 = STRING: IF-MIB::ifPhysAddress.2 = STRING: 0:9:5b:2f:9e:d5 IF-MIB::ifPhysAddress.3 = STRING: 0:b0:d0:46:32:71 ... ... [root@bigboy tmp]# Upon inspecting the output of the snmpwalk command, you can see that the second interface seems to have the name eth0 and the MAC address 0:9:5b:2f:9e:d5. You can now retrieve the individual MAC address using the snmpget command. [root@bigboy tmp]# snmpget -v 1 -c const1payted localhost ifPhysAddress.2 IF-MIB::ifPhysAddress.2 = STRING: 0:9:5b:2f:9e:d5 [root@bigboy tmp]# We can confirm this information using the ifconfig command for interface eth0; the very first line shows a matching MAC address. 7
  • 8. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure [root@bigboy tmp]# ifconfig -a eth0 eth0 Link encap:Ethernet HWaddr 00:09:5B:2F:9E:D5 inet addr:216.10.119.244 Bcast:216.10.119.255 Mask:255.255.255.240 ... ... [root@bigboy tmp]# Now we will see how you can configure SNMP on your Linux server to achieve these results. 2.4.3 Configuring Simple SNMP on a Linux Server By default Fedora, installs the net-snmp package as its SNMP server product. This package uses a configuration file named /etc/snmp/snmpd.conf in which the community strings and other parameters may be set. The version of the configuration file that comes with net-snmp is quite complicated. I suggest archiving it and using a much simpler version with only a single line containing the keyword rocommunity followed by the community string. Here is an example. 1) Save the old configuration file [root@bigboy tmp]# cd /etc/snmp/ [root@bigboy snmp]# mv snmpd.conf snmpd.conf.old [root@bigboy snmp]# vi snmpd.conf 2) Enter the following line in the new configuration file to set the Read Only community string to craz33guy. rocommunity craz33guy 3) Configure Linux to start SNMP services on each reboot with the chkconfig command: 8
  • 9. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure [root@bigboy root]# chkconfig snmpd on [root@bigboy root]# 4) Start SNMP to load the current configuration file. [root@bigboy root]# service snmpd start Starting snmpd: [ OK ] [root@bigboy root]# 5) Test whether SNMP can read the system and interface MIBs using the snmpwalk command. [root@bigboy snmp]# snmpwalk -v 1 -c craz33guy localhost system SNMPv2-MIB::sysDescr.0 = STRING: Linux bigboy 2.4.18-14 #1 Wed Sep 4 11:57:57 EDT 2002 i586 SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10 SNMPv2-MIB::sysUpTime.0 = Timeticks: (425) 0:00:04.25 SNMPv2-MIB::sysContact.0 = STRING: root@localhost SNMPv2-MIB::sysName.0 = STRING: bigboy ... ... ... [root@bigboy snmp]# snmpwalk -v 1 -c craz33guy localhost interface IF-MIB::ifNumber.0 = INTEGER: 3 IF-MIB::ifIndex.1 = INTEGER: 1 IF-MIB::ifIndex.2 = INTEGER: 2 IF-MIB::ifIndex.3 = INTEGER: 3 IF-MIB::ifDescr.1 = STRING: lo IF-MIB::ifDescr.2 = STRING: wlan0 IF-MIB::ifDescr.3 = STRING: eth0 ... ... 9
  • 10. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure ... [root@bigboy snmp]# Now that we know SNMP is working correctly on your Linux server, we can configure SNMP statistics gathering software, such as MRTG, to create online graphs of your traffic flows. 2.4.4 SNMP On Other Devices In the example, we were polling localhost. We can poll any SNMP-aware network device that has SNMP enabled. All you need is the IP address and SNMP Read Only string and you'll be able to get similar results. Here is an example of a query of a device with an IP address of 192.168.1.1. [root@bigboy snmp]# snmpwalk -v 1 -c chir1qui 192.168.1.1 interface Note: When issuing snmpwalk and snmpget commands, remember to use the proper version switch (-v 1, -v 2c, or -v 3) for the version of SNMP you're using. 2.5Basic SNMP Security The security precautions that need to be taken with SNMP vary depending on the version you are using. This section outlines the basic steps for protecting your MIB data. 2.5.1SNMP Versions 1 and 2 The most commonly supported versions of SNMP don't encrypt your community string password so you shouldn't do queries over insecure networks, such as the Internet. You should also make sure that you use all reasonable security measures to allow queries only from trusted IP addresses either via a firewall or the SNMP security features available in the snmp.conf file. You can also configure your server to use the TCP wrappers feature outlined in Appendix I, "Miscellaneous Linux Topics," to limit access to specific servers without the need of a firewall. In case you need it, the snmpd.conf file can support limiting MIB access to trusted hosts and networks. 10
  • 11. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure The snmpd.conf file has two security sections; a section with very restrictive access sits at the top of the file and is immediately followed by a less restrictive section. The example that follows is a modification of the less restrictive section. You will have to comment out the more restrictive statements at the top of the file for it to work correctly. ## sec.name source community ## ======== ====== ========= com2sec local localhost craz33guy com2sec network_1 172.16.1.0/24 craz33guy com2sec network_2 192.168.2.0/24 craz33guy ## Access.group.name sec.model sec.name ## ================= ========= ======== group MyROGroup_1 v1 local group MyROGroup_1 v1 network_1 group MyROGroup_2 v2c network_2 ## MIB.view.name incl/excl MIB.subtree mask ## ============== ========= =========== ==== view all-mibs included .1 80 ## MIB ## group.name context sec.model sec.level prefix read write notif ## ========== ======= ========= ========= ====== ==== ===== ===== access MyROGroup_1 "" v1 noauth exact all-mibs none none access MyROGroup_2 "" v2c noauth exact all-mibs none none In our example: 11
  • 12. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure  Only three networks (localhost, 172.16.1.0/24, and 192.168.1.0/24) are allowed to access the server with the craz33guy community string.  Each network is matched to a either a group called MyROGroup_1 using SNMP version 1, or group called MyROGroup_2 using SNMP version 2.  All the MIBs on the server are defined by the view named all-mibs.  An access statement ensures that only the defined networks have read only access to all the MIBs. MyROGroup_1 only has version 1 access with MyROGroup_2 only having version 2 access.  Modification of the MIBs via SNMP is denied because the word "none" is in the write section of the access statement. These precautions are probably unnecessary in a home environment where access is generally limited to devices on the home network by a NAT firewall. 2.5.2 SNMP Version 3 SNMP Version 3 SNMPv3 is a much more secure alternative to earlier versions as it encrypts all its data and uses a username / password combination for client authentication. The username should be located in the /etc/snmp/snmpd.conf file with a corresponding automatically generated password located in the /var/net-snmp/snmpd.conf file. Here is how it's done. 1. Install the net-snmp-devel package as it contains the utility that will allow you to generate the password. If you need an installation refresher. 2. Stop the snmpd process. [root@bigboy tmp]# service snmpd stop Stopping snmpd: [ OK ] [root@bigboy tmp]# 3. Automatically create the /etc/snmp/snmpd.conf and /var/net-snmp/snmpd.conf username and password entries using the net-snmp-config command. In this example the authentication password of "rootsrockreggae" for the read only (-ro) username 12
  • 13. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure "username4snmpv3" is encrypted using the MD5 algorithm. The data received will not be encrypted. [root@bigboy tmp]# net-snmp-config --create-snmpv3-user -ro -a MD5 -A rootsrockreggae username4snmpv3 adding the following line to /var/net-snmp/snmpd.conf: createUser username4snmpv3 MD5 "rootsrockreggae" DES adding the following line to /etc/snmp/snmpd.conf: rouser username4snmpv3 [root@bigboy tmp]# service snmpd start Starting snmpd: [ OK ] [root@bigboy tmp]# 4. To encrypt the data received we use the net-snmp-config command again to create a username just for this purpose. A new read only username "securev3user" and authentication password "mandeville" are used, but this time the data will be encrypted using the DES algorithm with the privacy password of "savlamar". [root@bigboy tmp]# net-snmp-config --create-snmpv3-user -ro -a MD5 -A mandeville -x DES -X savlamar securev3user adding the following line to /var/net-snmp/snmpd.conf: createUser securev3user MD5 "mandeville" DES savlamar adding the following line to /etc/snmp/snmpd.conf: rouser securev3user [root@bigboy tmp]# 5. Start the snmpd process. [root@bigboy tmp]# service snmpd start Starting snmpd: [ OK ] 13
  • 14. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure [root@bigboy tmp]# 6. First we'll do a query from remote host smallfry. We specify the authentication password and authentication encryption method, and we also use the -l flag to indicate that authentication will be used, but that data privacy will be disabled (the authNoPriv option). [root@smallfry ~]# snmpget -v 3 -u username4snmpv3 -l authNoPriv -a MD5 -A rootsrockreggae 192.168.1.100 SNMPv2-MIB::sysORDescr.8 SNMPv2-MIB::sysORDescr.8 = STRING: The management information definitions for the SNMP User-based Security Model. [root@smallfry ~]# The query returns an easy to read string, "The management information definitions for the SNMP User-based Security Model". This unencrypted string can also be seen in the tshark packet capture of the server's interface. [root@bigboy tmp]# tshark -n -i eth1 -x port 161 Capturing on eth1 ... ... ... 0.005889 192.168.1.100 -> 192.168.1.50 SNMP get-response 0000 00 c0 4f 46 0c 2e 00 b0 d0 46 32 71 08 00 45 00 ..OF.....F2q..E. 0010 00 f0 00 00 40 00 40 11 b3 b2 c0 a8 02 c8 c0 a8 ....@.@......... 0020 02 32 00 a1 80 0a 00 dc 87 38 30 81 d1 02 01 03 .2.......80..... 0030 30 11 02 04 45 a2 23 54 02 03 00 ff e3 04 01 01 0...E.#T........ 0040 02 01 03 04 38 30 36 04 0d 80 00 1f 88 80 71 11 ....806.......q. 0050 68 72 0e b1 e7 45 02 01 12 02 01 39 04 0f 75 73 hr...E.....9..us 14
  • 15. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 0060 65 72 6e 61 6d 65 34 73 6e 6d 70 76 33 04 0c 46 ername4snmpv3..F 0070 6c 74 26 51 4d aa 65 61 59 06 1a 04 00 30 7f 04 lt&QM.eaY....0.. 0080 0d 80 00 1f 88 80 71 11 68 72 0e b1 e7 45 04 00 ......q.hr...E.. 0090 a2 6c 02 04 43 4e da d7 02 01 00 02 01 00 30 5e .l..CN........0^ 00a0 30 5c 06 0a 2b 06 01 02 01 01 09 01 03 08 04 4e 0..+..........N 00b0 54 68 65 20 6d 61 6e 61 67 65 6d 65 6e 74 20 69 The management i 00c0 6e 66 6f 72 6d 61 74 69 6f 6e 20 64 65 66 69 6e nformation defin 00d0 69 74 69 6f 6e 73 20 66 6f 72 20 74 68 65 20 53 itions for the S 00e0 4e 4d 50 20 55 73 65 72 2d 62 61 73 65 64 20 53 NMP User-based S 00f0 65 63 75 72 69 74 79 20 4d 6f 64 65 6c 2e ecurity Model. 4 packets captured [root@bigboy tmp]# 7. Next we'll do a query that will return a response over an encrypted data channel while crossing the network (the authPriv option). [root@smallfry ~]# snmpget -v 3 -u securev3user -l authPriv -a MD5 -A mandeville -x DES -X savlamar 192.168.1.100 SNMPv2- MIB::sysORDescr.8 SNMPv2-MIB::sysORDescr.8 = STRING: The management information definitions for the SNMP User-based Security Model. [root@smallfry ~]# The query returns the same string, but the tshark packet capture only sees encrypted data, with only the username being visible. [root@bigboy tmp] # tshark -n -i eth1 -x port 161 Capturing on eth1 ... 15
  • 16. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure ... ... 0.003675 192.168.1.200 -> 192.168.1.50 SNMP Source port: 161 Destination port: 32778 [UDP CHECKSUM INCORRECT] 0000 00 c0 4f 46 0c 2e 00 b0 d0 46 32 71 08 00 45 00 ..OF.....F2q..E. 0010 01 00 00 00 40 00 40 11 b3 a2 c0 a8 02 c8 c0 a8 ....@.@......... 0020 02 32 00 a1 80 0a 00 ec 87 48 30 81 e1 02 01 03 .2.......H0..... 0030 30 11 02 04 17 52 82 96 02 03 00 ff e3 04 01 03 0....R.......... 0040 02 01 03 04 3e 30 3c 04 0d 80 00 1f 88 80 71 11 ....>0<.......q. 0050 68 72 0e b1 e7 45 02 01 11 02 02 00 8e 04 0c 73 hr...E.........s 0060 65 63 75 72 65 76 33 75 73 65 72 04 0c 01 b2 00 ecurev3user..... 0070 6e 23 07 83 dc a2 b6 d6 3d 04 08 00 00 00 11 4e n#......=......N 0080 df 19 a3 04 81 88 36 dd e0 ce e0 52 19 ff 58 7e ......6....R..X~ 0090 be fa d1 96 20 2b 28 65 59 30 e8 d4 cb 18 9f 8f .... +(eY0...... 00a0 1e 5b a3 d6 ae f7 4a 86 bd ed 2a 4b a8 df 52 fb .[....J...*K..R. 00b0 00 b4 a8 37 3d 74 9e 6d 1d 56 9a ba f2 13 fa 72 ...7=t.m.V.....r 00c0 4d 47 fb 88 7b d3 54 e1 9d b3 66 f0 29 ab 8a 55 MG..{.T...f.)..U 00d0 6f 77 65 40 87 ab 0c 51 d9 0e bf 33 7f 9a cb ea owe@...Q...3.... 00e0 37 50 3c 8e 65 dd 8f 3c 49 71 96 59 f9 d3 a8 23 7P<.e..<Iq.Y...# 00f0 81 c6 1b b2 c2 d0 57 9b 98 1b 89 1e ca 77 3d 84 ......W......w=. 0100 6f af b6 9b 86 3a 2f 66 44 1a 41 51 03 bc o....:/fD.AQ.. 4 packets captured [root@bigboy tmp] # 8. Your password and privilege information are stored in /var/net-snmp/snmpd.conf using the format displayed when you used the net-snmp-config command. here is an example. # File: /var/net-snmp/snmpd.conf before SNMP starts createUser securev3user MD5 "mandeville" DES savlamar 16
  • 17. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure createUser username4snmpv3 MD5 "rootsrockreggae" DES The snmpd daemon will completely encrypt the SNMP password data in this file when it restarts which helps to further increase security. We can see an example of this configuration here. # File: /var/net-snmp/snmpd.conf after SNMP starts usmUser 1 3 0x80001f88780711168720eb1e745 0x73656375626576337573657200 0x73656375726576337573657200 NULL .1.3.6.1.6.3.10.1.1.2 0xd951 ac1d95033f4afgf31243eb6907df .1.3.6.1.6.3.10.1.2.2 0xf1f4bb00452211d27b50c273c09031ac 0x00 usmUser 1 3 0x80001f8880711168720eb1e745 0x757365726e61657534736e6d70763300 0x757365726e616d6534736e6d70763300 NULL .1.3.6.1.6.3.10. 1.1.2 0x5e35c9f5352519aa4f53eded09bbdddd .1.3.6.1.6.3.10.1.2.2 0x5e35c9f5122519aa4f53eded09bbdddd "" setserialno 1464593474 We here showing the practice of the net-snmp-config command so that we and viewer can become familiar with the syntax it uses to edit the SNMP configuration files. When in doubt, we can get a full syntax listing of the command if you use it without any arguments like this: [root@bigboy tmp]# net-snmp-config ... ... SNMP Setup commands: 17
  • 18. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure --create-snmpv3-user [-ro] [-A authpass] [-X privpass] [-a MD5|SHA] [-x DES|AES] [username] ... ... [root@bigboy tmp]# With experience, we should become confident enough to edit the configuration files by ourself. As we can see, SNMPv3 is more secure than previous versions and should be your first SNMP choice whenever possible. 2.6 Simple SNMP Troubleshooting If your SNMP queries fail, then verify that:  We restarted your snmp.conf file so the configuration settings become active. Remember, the snmpd.conf file is only read by the snmpd daemon when it starts up.  We are using the correct community string.  Firewalls aren't preventing SNMP queries from the SNMP client to the SNMP target.  Our SNMP security policy allows the query from our network.  Any TCP wrappers configuration on our SNMP target machine allows SNMP queries from our SNMP client. Generally speaking in a home environment protected by NAT your TCP wrappers files (/etc/hosts.allow) and (/etc/hosts.deny) should be blank.  Network routing between the client and target devices is correct. A simple ping or traceroute test should be sufficient.  The snmpd daemon is running on the SNMP client.  We are querying using the correct SNMP version.  Our /var/log/messages file does not contain errors that may have occurred while starting snmpd. Troubleshooting to get functioning SNMP queries is important as many other supporting applications, such as MRTG which I'll discuss next, rely on them in order to work correctly. 18
  • 19. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 3 MRTG MRTG (Multi-Router Traffic Grapher) is a public domain package for producing graphs of various router statistics via a Web page. You can easily create graphs of traffic flow statistics through your home network's firewall/router or even your Linux box's NIC cards using MRTG. The product is available from the MRTG Web site (www.mrtg.org) and also on your distribution CDs. Figure 2 shows a sample MRTG graph. Figure 3.1: A Typical MRTG Web Page 19
  • 20. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 3.2 MRTG Download and Installation You need to install MRTG before proceeding. Most RedHat and Fedora Linux software products are available in the RPM format. When searching for the file, remember that the MRTG RPM's filename usually starts with mrtg and a version number, as in mrtg-2.10.5-3.i386.rpm. In addition to MRTG, you need to install the SNMP utility tools as explained earlier and you need to have a Web server package installed for MRTG to work. RedHat Linux usually comes with the Apache Web server software preinstalled. The easiest way to tell if Apache is installed is to run the rpm -q httpd command. If you don't get a positive response, you can refer to Chapter 20, "The Apache Web Server", for installation details. By default Apache expects the HTML files for your Web site to be located in /var/www/html. MRTG places its HTML files in /var/www/mrtg. 3.3 Configuring MRTG By default, MRTG maps the inbound and outbound data throughput rates on the device it is polling. Methods for specifying other OIDs, such as CPU and memory usage, are discussed in Chapter 23, "Advanced MRTG for Linux". For now, I'll stick with the default configuration. When the MRTG RPM is installed, it creates a directory called /etc/mrtg in which all future configuration files are stored. To create a replacement default /etc/mrtg/mrtg.cfg configuration file for the server, follow these steps. 1) Use MRTG's cfgmaker command to create a configuration file named mrtg.cfg for the server (bigboy) using a Read Only community string of craz33guy. Place all data files in the directory /var/www/mrtg. [root@bigboy tmp]# cfgmaker --output=/etc/mrtg/mrtg.cfg --global "workdir: /var/www/mrtg" -ifref=ip --global 'options[_]: growright,bits' craz33guy@localhost 20
  • 21. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure --base: Get Device Info on craz33guy@localhost: --base: Vendor Id: --base: Populating confcache --snpo: confcache craz33guy@localhost: Descr lo --> 1 --snpo: confcache craz33guy@localhost: Descr wlan0 --> 2 ... ... ... --base: Walking ifAdminStatus --base: Walking ifOperStatus --base: Writing /etc/mrtg/mrtg.cfg [root@bigboy tmp]# SNMPv2: As explained in the SNMP section, there are different versions of SNMP. If your query doesn't work, check to make sure you are using the required version and then check other SNMP configuration parameters on the target device. You can specify MRTG's SNMP query version with the --snmp-options cfgmaker option. Here is an example of cfgmaker using an SNMP version 2 query of a router with an IP address of 192.168.1.3. The --snmp- options option's five colons before the 2 are important. [root@bigboy tmp]# cfgmaker --output=/etc/mrtg/192.168.1.3.cfg -ifref=ip --global "workdir: /var/www/mrtg" --snmp-options=:::::2 craz33guy@192.168.1.3 SNMPv3: The cfgmaker command can also be used to poll SNMPv3 enabled devices, but we have to first install the Net::SNMP PERL module. [root@bigboy tmp]# yum -y install perl-Net-SNMP or 21
  • 22. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure [root@bigboy tmp]# perl -MCPAN -e "install Net::SNMP" If we fail to install the module, we will get an error looking like this: Undefined subroutine &main::snmpmapOID called at ./cfgmaker line 1480. Next we need to know the SNMPv3 ID of the host you intend to poll. In fedora, this is value is located in the /var/net-snmp/snmpd.conf file. You can use the grep command to obtain it. [root@bigboy tmp]# grep oldEngineID /var/net-snmp/snmpd.conf oldEngineID 0x80001f8880711168720eb1e745 [root@bigboy tmp]# We can then repeat the cfgmaker command with options specific to the privacy and authentication schemes configured on the SNMP target device. Using the configurations of our previous SNMPv3 example, our configuration for encrypted authentication only would look like this: [root@bigboy tmp]# cfgmaker --global 'WorkDir: /var/www/mrtg' --global 'Options[_]: growright, bits' --output=/etc/mrtg/192.168.1.100.cfg --enablesnmpv3 --username=username4snmpv3 --authpassword=rootsrockreggae --authproto=md5 --snmp-options=:::::3 --contextengineid=0x80001f8880711168720eb1e745 securev3user@192.168.1.100 Our configuration for encrypted authentication and data privacy only would look like this: [root@bigboy tmp]# cfgmaker --global 'WorkDir: /var/www/mrtg' 22
  • 23. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure --global 'Options[_]: growright, bits' --output=/etc/mrtg/192.168.1.100-secure.cfg --enablesnmpv3 --username=securev3user --authpassword=mandeville --authproto=md5 --privpassword=savlamar --privprotocol=des --snmp-options=:::::3 --contextengineid=0x80001f8880711168720eb1e745 securev3user@192.168.1.100 Note: The MRTG cfgmaker command reliably supports SNMPv3 as of MRTG version 2.15. Prior to this version you would commonly see this error when attempting to do DNMPv3 queries. SNMP V3 requires a --username parameter as part of the User Security Model for router securev3user@192.168.1.100:::::3 at ./cfgmaker line 121. 2) Edit /etc/mrtg/mrtg.cfg, and remove the sections related to interfaces you don't need to monitor. A certain candidate would be the virtual loopback interface Lo: (with the IP address of 127.0.0.1), which doesn't pass any network traffic at all. 3) Run MRTG using /etc/mrtg/mrtg.cfg as your argument three times. You'll get an error the two times as MRTG tries to move old data files, and naturally, the first time it is run, MRTG has no data files to move. [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg Rateup WARNING: /usr/bin/rateup could not read the primary log file for localhost_192.168.1.100 Rateup WARNING: /usr/bin/rateup The backup log file for localhost_192.168.1.100 was invalid as well Rateup WARNING: /usr/bin/rateup Can't remove localhost_192.168.1.100.old updating log file Rateup WARNING: /usr/bin/rateup Can't rename localhost_192.168.1.100.log to localhost_192.168.1.100.old updating log file 23
  • 24. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg Rateup WARNING: /usr/bin/rateup Can't remove localhost_192.168.1.100.old updating log file [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg [root@bigboy tmp]# 4) Use MRTG's indexmaker command to create a Web index page using your new mrtg.cfg file as a guide. The MRTG Web GUI expects to find the index file in the default MRTG Web directory of /var/www/mrtg/, so the format of the command would be. [root@bigboy tmp]# indexmaker --output=/var/www/mrtg/index.html /etc/mrtg/mrtg.cfg 5) MRTG is run every five minutes by default, and the file that governs this is /etc/cron.d/mrtg. For MRTG to work correctly, edit this file, replacing all occurrences of /usr/bin/mrtg with env LANG=C /usr/bin/mrtg. The explanation for changing the language character set for MRTG is given in the "Troubleshooting MRTG" section. This isn't all, you need to view the graphs too. This will be covered later, but first I'll show you how to poll multiple devices. 3.4 Getting MRTG To Poll Multiple Devices The Fedora Core MRTG installation process creates a cron file named /etc/cron.d/mrtg. This file tells the cron daemon to run MRTG using the /etc/mrtg/mrtg.cfg file every five minutes to poll your network devices. You can configure MRTG to poll multiple devices, each with a separate configuration file. Here's how: 1) Create a new configuration file using the steps from the previous section; choose a filename that is not mrtg.cfg. 2) Add a new MRTG line in /etc/cron.d/mrtg for each new configuration file you create. 0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg 24
  • 25. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/device1.cfg 0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/device2.cfg 3) Run the indexmaker command, and include all of your /etc/mrtg configuration files, to regenerate your Web index page. [root@bigboy tmp]# indexmaker --output=/var/www/mrtg/index.html /etc/mrtg/mrtg.cfg /etc/mrtg/device1.cfg /etc/mrtg/device2.cfg 4) Other versions of Linux keep their MRTG cron entries inside the /etc/crontab file. Edit this file using the same syntax as the Fedora /etc/cron.d/mrtg file, and then restart the cron daemon to re-read the configuration: [root@bigboy tmp]# service crond restart You could also create a script with the /usr/bin/mrtg /etc/mrtg/device.cfg entries in it and make cron run it every five minutes. This way you can just edit the script each time you add a device without having to restart cron. 3.5 Configuring Apache To Work With MRTG MRTG is useful because it can provide a graphical representation of your server's performance statistics via a Web browser. With Fedora Core, MRTG creates an add-on configuration file named /etc/httpd/conf.d/mrtg.conf that includes all the necessary Apache commands for MRTG to work. Some configuration may need to be done, because by default MRTG accepts Web requests from the Linux console only. You can add your home network to the file by inserting the network on the Allow from line, or you can allow universal access by commenting out that line along with the Deny from line. This example adds access from the 192.168.1.0 network. <Location /mrtg> 25
  • 26. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Order deny,allow Deny from all Allow from localhost 192.168.1.0/24 </Location> If you want to access MRTG from the Internet, then you'll have to comment out the Deny statement and allow from all IP addresses: <Location /mrtg> Order deny,allow Allow from all </Location> Remember to restart Apache once you have made these modifications in order for these changes to take effect. Note: With newer versions of Fedora, Apache automatically reads the add-on files in the /etc/ httpd/conf.d/ directory. With Fedora Core 1, you have to specifically configure the Apache configuration file /etc/httpd/conf/httpd.conf to find it. You can do this yourself by inserting this line at the very bottom of the main Apache configuration file before restarting Apache for the change to take effect. include "/etc/httpd/conf.d/mrtg.conf" 3.5.1 Basic Security If you are accessing MRTG graphs from the Internet, you may want to add password protection to the directory by using a .htaccess file 3.6 How To View The MRTG Graphs In Your Web Browser You can now access your MRTG graphs by pointing your browser to the URL: http://server-ip-address/mrtg/ 26
  • 27. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 3.7 Using MRTG To Monitor Other Subsystems MRTG will generate HTML pages with daily, weekly, monthly, and yearly statistics for your interfaces. By default, MRTG provides only network interface statistics. Chapter 23, "Advanced MRTG for Linux", has detailed examples and explanations of how to monitor Linux disk, CPU, memory, and Web connection data. The MRTG Web site, www.mrtg.org, also has links to other sites that show you how to monitor many other subsystems on a variety of devices and operating systems. 3.8 Troubleshooting MRTG There are many simple steps you can use to troubleshoot MRTG. Take a look at some of the most common ones. 3.8.1 Basic Steps MRTG won't work if SNMP queries don't work. Make sure you follow the SNMP troubleshooting steps if you have any difficulties. 3.8.2 Setting The Correct Character Set MRTG usually works only if your system uses an ASCII-based (Western European) character set. If it isn't set, then you'll get errors such as this every time you run MRTG from the command line or as part of a cron job: [root@bigboy tmp]# mrtg /etc/mrtg/mrtg.cfg ------------------------------------------------------------------- ERROR: Mrtg will most likely not work propperly when the environment variable LANG is set to UTF-8. Please run mrtg in an envir.. where this is not the case: env LANG=C /usr/bin/mrtg ... ------------------------------------------------------------------- [root@bigboy tmp]# Your system's character set is defined in /etc/sysconfig/i18n, and the current Fedora default of en_US.UTF-8 won't work, but en_US will after a system reboot. This is not necessarily a 27
  • 28. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure good idea, especially if the native language Linux uses on your system is not ASCII based, other things may fail to work. A better solution is to always run MRTG using this command instead of using just plain /usr/ bin/mrtg. env LANG=C /usr/bin/mrtg This will modify the character set used by MRTG alone and shouldn't affect anything else. 3.8.3 Incorrect SNMPv3 Engine ID The added security of SNMPv3 forces each client to create its own serial number or engine ID. If you use an incorrect identifier you may get noSuchInstance errors like this when polling with MRTG. 2008-07-26 19:42:40: WARNING: Expected a number but got 'noSuchInstance' 2008-07-26 19:42:40: WARNING: Expected a number but got 'noSuchInstance' 2008-07-26 19:42:40: ERROR: Target[localhost_3][_IN_] ' $target->[1]{$mode} ' did not eval into defined data 2008-07-26 19:42:40: ERROR: Target[localhost_3][_OUT_] ' $target->[1]{$mode} ' did not eval into defined data Always make sure you are using the correct ID and try again. 3.8.4 Fedora Core 1 MRTG Errors With Net-SNMP A bug appears in the MRTG implementation for some Fedora Core 1 MRTG versions when polling another Fedora Core 1 server. When using a -ifref=ip statement with the cfgmaker command, every line in the configuration file that is generated becomes commented out. When it works, this statement is very convenient, because it makes MRTG provide graphs sorted by the IP addresses of the interfaces instead of the default, which is the much harder to recognize interface MAC address. Upgrading to the latest Core 1 version of MRTG will fix the problem. ### Interface 6 >> Descr: | Name: | Ip: '192.168.1.100' ### 28
  • 29. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure ### The following interface is commented out because: ### * has a speed of which makes no sense ### * got 'Received SNMP response with error code ### error status: noSuchName ### index 1 (OID: 1.3.6.1.2.1.2.2.1.10.6) ### SNMPv1_Session (remote host: "localhost" [127.0.0.1].161) ### community: "craz33guy" ### request ID: 824482716 ### PDU bufsize: 8000 bytes ### timeout: 2s ### retries: 5 # # Target[localhost_192.168.1.100]: /192.168.1.100:craz33guy@localhost: # SetEnv[localhost_192.168.1.100]: MRTG_INT_IP="192.168.1.100" MRTG_INT_DES # MaxBytes[localhost_192.168.1.100]: 0 # Title[localhost_192.168.1.100]: Traffic Analysis for 192.168.1.100 # PageTop[localhost_192.168.1.100]: Traffic Analysis for 192.168.1.100 As all the lines in the configuration file are commented out with a # character, indexmaker fails to create an index.html file and gives errors. [root@bigboy tmp]# indexmaker --output=/var/www/mrtg/stats/index.html /etc/mrtg/mrtg.cfg Use of uninitialized value in hash element at /usr/bin/indexmaker line 307. [root@bigboy tmp]# 29
  • 30. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 4. Webalizer Webalizer is a Web server log file analysis tool that comes installed by default on RedHat/Fedora Linux. Each night, Webalizer reads your Apache log files and creates a set of Web pages that enable you to view Web surfer statistics for your site. The information provided includes a list of your Web site's most popular pages sorted by hits along with traffic graphs showing the times of day when your site is most popular. How To View Your Webalizer Statistics Fedora creates an add-on configuration file named /etc/httpd/conf.d/Webalizer.conf that includes all the necessary Apache commands for Webalizer to work. As in the case of the MRTG add-on file mentioned above, you have to edit it to allow access to the Webalizer pages from locations other than the Linux console. You also have to restart Apache to make the changes take effect. By default, Webalizer places its index page in the directory /var/www/html/usage and allows you to view your data by visiting the URL http://server-ip-address/usage. The Webalizer Configuration File Webalizer stores its configuration in the file /etc/Webalizer.conf. The default settings should be sufficient for your Web server, but you may want to adjust the directory in which Webalizer places your graph statistics. This can be adjusted with the OutputDir directive in the file. After adjustments, Webalizer functions with few annoyances; however, be aware that running in quiet mode could hide deeper problems that could occur in future. 30
  • 31. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 5. The top Command You can monitor the amount of memory and CPU resources your system is using the top command. [root@bigboy tmp]# top 3:04pm up 25 days, 23:23, 2 users, load average: 0.00, 0.02, 0.00 78 processes: 76 sleeping, 2 running, 0 zombie, 0 stopped CPU states: 0.9% user, 0.5% system, 0.0% nice, 0.8% idle Mem: 384716K av, 327180K used, 57536K free, 0K shrd, 101544K buff Swap: 779112K av, 0K used, 779112K free 130776K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 27191 root 15 0 1012 1012 780 R 5.6 0.2 0:00 top 4545 root 16 0 5892 5888 4956 S 0.9 1.5 169:26 magicdev 1 root 15 0 476 476 432 S 0.0 0.1 0:05 init 2 root 15 0 0 0 0 SW 0.0 0.0 0:00 keventd 5 root 15 0 0 0 0 SW 0.0 0.0 0:41 kswapd 6 root 25 0 0 0 0 SW 0.0 0.0 0:00 bdflush [root@bigboy tmp]# Here the CPU usage is under 1.0% and 14% of memory (57536K) is free. The amount of free memory may appear low, but in this case, the server doesn't seem to be swapping idle processes from memory to the swap disk partition as it isn't being used at all. Excessive swapping can cause your system to slow down dramatically, the simplest ways to avoid this 31
  • 32. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure is to add more RAM or reduce the number of processes or users that are active on your system. If your system seems slow but the CPU and memory usage is low, then start looking at networking problems, such as poor duplex negotiation, bad cables, and network congestion due to excessive traffic. 6. The vmstat Command You can also determine memory and swap usage with the vmstat command, which provides a summary of what top produces. In the example, memory is still 14% free (57,452MB used from a total of 130,780) and swap isn't being used at all. [root@bigboy tmp]# vmstat procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 57452 101584 130780 0 0 0 4 18 1 3 1 1 [root@bigboy tmp]# As your memory fills up, your system will temporarily store programs and data on your hard disk's "swap" partition. Excess swapping of programs and data between disk and memory can cause your system to slow down significantly and memory usage should be monitored to allow you to plan ways to either increase RAM or tune the way your system operates. System tuning is beyond the scope of this book, but there are many reference guides which can show you how to do this. 7. The free Utility The free utility can determine the amount of free RAM on your system. The output is easier to understand than vmstat's. Here's a sample. [root@bigboy tmp]# free total used free shared buffers cached Mem: 126060 119096 6964 0 58972 40028 -/+ buffers/cache: 20096 105964 Swap: 522072 15496 506576 [root@bigboy tmp]# 32
  • 33. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure You should generally try to make your system run with at least 20% free memory on average, which should allow it to handle moderate spikes in usage caused by running memory- intensive cron batch jobs or tape backups. If you cannot achieve this, consider running more efficient versions of programs, offloading some applications to servers with less load, and, of course, upgrading the capacity of your RAM. 8. Locating And Viewing The Contents Of Linux MIBs Residing in memory, MIBs are data structures that are constantly updated via the SNMP daemon. The MIB configuration text files are located on your hard disk and loaded into memory each time SNMP restarts. You can easily find your Fedora Linux MIBs by using the locate command and filtering the output to include only values with the word "snmp" in them. As you can see in this case, the MIBs are located in the /usr/share/snmp/mibs directory: [root@bigboy tmp]# locate mib | grep snmp /usr/share/doc/net-snmp-5.0.6/README.mib2c /usr/share/snmp/mibs /usr/share/snmp/mibs/DISMAN-SCHEDULE-MIB.txt ... ... [root@bigboy tmp]# As the MIB configurations are text files you can search for keywords in them using the grep command. This examples searches for the MIBs that keep track of TCP connections and returns the RFC1213 and TCP MIBs as the result. [root@silent mibs]# grep -i tcp /usr/share/snmp/mibs/*.txt | grep connections ... RFC1213-MIB.txt: "The limit on the total number of TCP connections RFC1213-MIB.txt: "The number of times TCP connections have made a ... TCP-MIB.txt: "The number of times TCP connections have made a ... 33
  • 34. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure ... [root@silent mibs]# You can use the vi editor to look at the MIBs. Don't change them, because doing so could cause SNMP to fail. MIBs are very complicated, but fortunately the key sections are commented. Each value tracked in a MIB is called an object and is often referred to by its object ID or OID. In this snippet of the RFC1213-MIB.txt file, you can see that querying the tcpActiveOpens object returns the number of active open TCP connections to the server. The SYNTAX field shows that this is a counter value. MIBs usually track two types of values. Counter values are used for items that continuously increase as time passes, such as the amount of packets passing through a NIC or amount of time CPU been busy since boot time. Integer values change instant by instant and are useful for tracking such statistics as the amount of memory currently being used. tcpActiveOpens OBJECT-TYPE SYNTAX Counter ACCESS read-only STATUS mandatory DESCRIPTION "The number of times TCP connections have made a direct transition to the SYN-SENT state from the CLOSED state." ::= { tcp 5 } You'll explore the differences between SNMP and MRTG terminologies in more detail later. Understanding them will be important in understanding how to use MRTG to track MIB values. 9. Testing Your MIB Value 34
  • 35. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Once you have identified an interesting MIB value for your Linux system you can then use the snmpwalk command to poll it. Many times the text aliases in a MIB only reference the OID branch and not the OID the data located in a leaf ending in an additional number like a ".0" or ".1". The snmpget command doesn't work with branches giving an error stating that the MIB variable couldn't be found. In the example below, the ssCpuRawUser OID alias was found to be interesting, but the snmpget command fails to get a value. Follow up with the snmpwalk command shows that the value is located in ssCpuRawUser.0 instead. The snmpget is then successful in retrieving the "counter32" type data with a current value of 396271. [root@bigboy tmp]# snmpget -v1 -c craz33guy localhost ssCpuRawUser Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: UCD-SNMP-MIB::ssCpuRawUser [root@bigboy tmp]# [root@bigboy tmp]# snmpwalk -v1 -c craz33guy localhost ssCpuRawUser UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 396241 [root@bigboy tmp]# snmpget -v1 -c craz33guy localhost ssCpuRawUser.0 UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 396271 [root@bigboy tmp]# The MIB values that work successfully with snmpget are the ones you should use with MRTG. 10. Differences In MIB And MRTG Terminology Always keep in mind that MRTG refers to MIB counter values as counter values. It refers to MIB integer and gauge values as gauge. By default, MRTG considers all values to be counters. MRTG doesn't plot counter values as a constantly increasing graph, it plots only how much the value has changed since the last polling cycle. CPU usage is typically tracked by MIBs as a counter value; fortunately, you can edit your MRTG configuration file to make it graph this information in a percentage use format (more on this later). 35
  • 36. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure The syntax type, the MIB object name, and the description of what it does are the most important things you need to know when configuring MRTG; I'll come back to these later. 11. The CPU And Memory Monitoring MIB The UCD-SNMP-MIB MIB keeps track of a number of key performance MIB objects, including the commonly used ones in Table 3. 11.1 Table 3: Important Objects In The UCD-SNMP-MIB MIB UCD-SNMP-MIB MIB MRTG Description Object Variable Type Type ssCpuRawUser Counter Counter Total CPU usage by applications run by nonprivileged users since the system booted. Adding the user, system, and nice values can give a good approximation of total CPU usage.. ssCpuRawSystem Counter Counter Total CPU usage by applications run by privileged system processes since the system booted. ssCpuRawNice Counter Counter Total CPU usage by applications running at a nondefault priority level. ssCpuRawIdle Counter Counter The percentage of the time the CPU is running idle. Subtracting this value from 100 can give a good approximation of total CPU usage. memAvailReal Integer Gauge Available Physical Memory Space On The Host 12. The TCP/IP Monitoring MIB 36
  • 37. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure The TCP-MIB MIB keeps track of data connection information and contains the very useful tcpActiveOpens and tcpCurrEstab objects. Table 3 details the most important objects in TCP- MIB. Table 12.1: Important Objects In The TCP-MIB MIB UCD-SNMP-MIB Object MIB MRTG Description Variable Type Type tcpActiveOpens Counter Counter Measures the number of completed TCP connections. tcpCurrEstab Gauge Gauge Measures the number of TCP connections in the established state tcpInErrs Counter Counter Total number of TCP segments with bad checksum errors 13. Manually Configuring Your MRTG File The MRTG cfgmaker program creates configuration files for network interfaces only, simultaneously tracking two OIDs: the NIC's input and output data statistics. The mrtg program then uses these configuration files to determine the type of data to record in its data directory. The indexmaker program also uses this information to create the overview, or Summary View Web page for the MIB OIDs you're monitoring. This Summary View page shows daily statistics only. You have to click on the Summary View graphs to get the Detailed View page behind it with the daily, weekly, monthly, and annual graphs. Some of the parameters in the configuration file refer to the Detailed View, others refer to the Summary View. If you want to monitor any other pairs of OIDs, you have to manually create the configuration files, because cfgmaker isn't aware of any OIDs other than those related to a 37
  • 38. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure NIC. The mrtg and indexmaker program can be fed individual OIDs from a customized configuration file and will function as expected if you edit the file correctly. 13.1 Parameter Formats MRTG configuration parameters are always followed by a graph name surrounded by square brackets and a colon. The format looks like this: Parameter[graph name]: value For ease of editing, the parameters for a particular graph are usually grouped together. Each graph can track two OIDS listed in the Target parameter, which is usually placed at the very top of the graph name list. The two OID values are separated by an & symbol; the first one can be is the input OID, and the second one is the output OID. 13.2 Legend Parameters On the Detailed View Web page, each graph has a legend that shows the max, average, and current values of the graph's OID statistics. You can use the legendI parameter for the description of the input graph (first graph OID) and the legendO for the output graph (second graph OID). The space available under each graph's legend is tiny so MRTG also has legend1 and legend2 parameters that are placed at the very bottom of the page to provide more details. Parameter legend1 is the expansion of legendI, and legend2 is the expansion of legendO. The Ylegend is the legend for the Y axis, the value you are trying to compare. In the case of a default MRTG configuration this would be the data flow through the interface in bits or bytes per second. Here is an example of the legends of a default MRTG configuration: YLegend[graph1]: Bits per second Legend1[graph1]: Incoming Traffic in Bits per Second Legend2[graph1]: Outgoing Traffic in Bits per Second 38
  • 39. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure LegendI[graph1]: In LegendO[graph1]: Out You can prevent MRTG from printing the legend at the bottom of the graph by leaving the value of the legend blank like this: LegendI[graph1]: 13.3 Options Parameters Options parameters provide MRTG with graph formatting information. The growright option makes sure the data at the right of the screen is for the most current graph values. This usually makes the graphs more intuitively easy to read. MRTG defaults to growing from the left. The nopercent option prevents MRTG from printing percentage style statistics in the legends at the bottom of the graph. The gauge option alerts MRTG to the fact that the graphed values are of the gauge type. If the value you are monitoring is in bytes, then you can convert the output to bits using the bits option. Likewise, you can convert per second values to per minute graphs using the perminute option. Here are some examples for two different graphs: options[graph1]: growright,nopercent,perminute options[graph2]: gauge,bits If you place this parameter at the top with a label of [_] it gets applied to all the graphs defined in the file. Here's an example. options[_]: growright 13.4 Title Parameters The title on the Summary Page is provided by the Title parameter, the PageTop parameter tells the title for the Detailed View page. The PageTop string must start with < H1 > and end with < H1 >. 39
  • 40. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Title[graph1]: Interface eth0 PageTop[graph1]: < H1 >Detailed Statistics For Interface eth0 < H1 > 13.4 Scaling Parameters The MaxBytes parameter is the maximum amount of data MRTG will plot on a graph. Anything more than this seems to disappear over the edge of the graph. MRTG also tries to adjust its graphs so that the largest value plotted on the graph is always close to the top. This is so even if you set the MaxBytes parameter. When you are plotting a value that has a known maximum and you always want to have this value at the top of the vertical legend, you may want to turn off MRTG's auto scaling. If you are plotting percentage CPU usage, and the server reaches a maximum of 60%, with scaling, MRTG will have a vertical plot of 0% to 60%, so that the vertical peak is near the top of the graph image. When scaling is off, and MaxBytes is set to 100, then the peak will be only 60% of the way up as the graph plots from 0% to 100%. The example removes scaling from the yearly, monthly, weekly, and daily views on the Detailed View page and gives them a maximum value of 100. Unscaled[graph1]: ymwd MaxBytes[graph1]: 100 13.5 Defining The MIB Target Parameters As stated before, MRTG always tries to compare two MIB OID values that are defined by the Target parameter. You have to specify the two MIB OID objects, the SNMP password and the IP address of the device you are querying in this parameter, and separate them with an & character: Target[graph1]: mib-object-1.0&mib-object-2.0:<SNMP-password>@<IP-address> 40
  • 41. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure The numeric value, in this case .0, at the end of the MIB is required. The next example uses the SNMP command to return the user mode CPU utilization of a Linux server. Notice how the .0 is tagged onto the end of the output. [root@silent mibs]# snmpwalk -v 1 -c craz33guy localhost ssCpuRawUser UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 926739 [root@silent mibs]# The MRTG legends map to the MIBs listed in the target as shown in Table 5. Table 13.1: Mapping MIBs To The Graph Legends Legend Maps To Target MIB Legend1 #1 Legend2 #2 LegendI #1 LegendO #2 So in the example below, legend1 and legendI describe mib-object-1.0 and legend2 and legendO describe mib-object-2.0. Target[graph1]: mib-object-1.0&mib-object-2.0:<SNMP-password>@<IP-address> 13.5.2 Plotting Only One MIB Value If you want to plot only one MIB value, you can just repeat the target MIB in the definition as in the next example, which plots only mib-object-1. The resulting MRTG graph actually superimposes the input and output graphs one on top of the other. Target[graph1]: mib-object-1.0&mib-object-1.0:<SNMP-password>@<IP-address> 41
  • 42. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 13.5.3 Adding MIB Values Together For a Graph You can use the plus sign between the pairs of MIB object values to add them together. The next example adds mib-object-1.0 and mib-object-3.0 for one graph and adds mib-object-2.0 and mib-object-4.0 for the other. Target[graph1]: mib-object-1.0&mib-object-2.0:<SNMP-password>@<IP-address> + mib-object-3.0&mib-object-4.0:<SNMP-password>@<IP-address> You can use other mathematical operators, such as subtract (-), multiply (*), and divide (%). Left and right parentheses are also valid. There must be white spaces before and after all these operators for MRTG to work correctly. If not, you'll get oddly shaded graphs. 13.5.4 Target: Total CPU Usage Linux CPU usage is occupied by system processes, user mode processes, and a few processes running in nice mode. This example adds them all together in a single plot. Target[graph1]:ssCpuRawUser.0&ssCpuRawUser.0:<SNMP-password>@<IP-address> + ssCpuRawSystem.0&ssCpuRawSystem.0:<SNMP-password>@<IP-address> + ssCpuRawNice.0&ssCpuRawNice.0:<SNMP-password>@<IP-address> Be sure to place this command on a single line 13.5.5 Target: Memory Usage Here is an example for the plotting the amount of free memory versus the total RAM installed in the server. Notice that this is a gauge type variable. Target[graph1]: memAvailReal.0&memTotalReal.0:<SNMP-password>@<IP-address> options[graph1]: nopercent,growright,gauge Next, plot the percentage of available memory. Notice how the mandatory white spaces separate the mathematical operators from the next target element. 42
  • 43. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Target[graph1]: ( memAvailReal.0& memAvailReal.0:<SNMP-password>@<IP- Address> ) * 100 / ( memTotalReal.0&memTotalReal.0:<SNMP-password>@<IP- Address> ) options[graph1]: nopercent,growright,gauge 13.5.6 Target: Newly Created Connections HTTP traffic caused by Web browsing usually consists of many very short lived connections. The tcpPassiveOpens MIB object tracks newly created connections and is suited for this type of data transfer. The tcpActiveOpens MIB object monitors new connections originating from the server. On smaller Web sites you may want to use the perminute option to make the graphs more meaningful. Target[graph1]: tcpPassiveOpens.0& tcpPassiveOpens.0:<SNMP-password>@<IP- address> MaxBytes[graph1]: 1000000 Options[graph1]: perminute 13.5.7 Target: Total TCP Established Connections Other protocols such as FTP and SSH create longer established connections while people download large files or stay logged into the server. The tcpCurrEstab MIB object measures the total number of connections in the established state and is a gauge value. Target[graph1]: tcpCurrEstab.0&tcpCurrEstab.0:<SNMP-password>@<IP-address> MaxBytes[graph1]: 1000000 Options[graph1]: gauge 13.5.8 Target: Disk Partition Usage In this example, you'll monitor the /var and /home disk partitions on the system. 1) First use the df -k command to get a list of the partitions in use. 43
  • 44. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure [root@bigboy tmp]# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda8 505605 128199 351302 27% / /dev/hda1 101089 19178 76692 21% /boot /dev/hda5 1035660 122864 860188 13% /home /dev/hda6 505605 8229 471272 2% /tmp /dev/hda3 3921436 890092 2832140 24% /usr /dev/hda2 1510060 171832 1261520 73% /var [root@bigboy tmp]# 2) Add two entries to your snmpd.conf file. disk /home disk /var 3) Restart the SNMP daemon to reload the values. [root@bigboy tmp]# service snmpd restart 4) Use the snmpwalk command to query the the dskPercent MIB. Object dskPercent.1 refers to the first disk entry in snmpd.conf (/home), and dskPercent.2 refers to the second (/var). [root@bigboy tmp]# snmpwalk -v 1 -c craz33guy localhost dskPercent.1 UCD-SNMP-MIB::dskPercent.1 = INTEGER: 13 [root@bigboy tmp]# snmpwalk -v 1 -c craz33guy localhost dskPercent.2 UCD-SNMP-MIB::dskPercent.2 = INTEGER: 73 [root@bigboy tmp]# Your MRTG target for these gauge MIB objects should look like this: Target[graph1]: dskPercent.1& dskPercent.1:<SNMP-password>@<IP-address> 44
  • 45. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure options[graph1]: growright,gauge 13.5.9 Defining Global Variables You have to make sure MRTG knows where the MIBs you're using are located. The default location MRTG uses may not be valid. Specify their locations with the global LoadMIBs parameter. You must also define where the HTML files will be located; the example specifies the default Fedora MRTG HTML directory. LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt, /usr/share/snmp/mibs/TCP- MIB.txt workdir: /var/www/mrtg/ 14 Implementing Advanced Server Monitoring You now can combine all you have learned to create a configuration file that monitors all these variables, and then you can integrate it into the existing MRTG configuration. 14.1 A Complete Sample Configuration Here is a sample configuration file that is used to query server localhost for CPU, memory, disk, and TCP connection information. # # File: /etc/mrtg/server-info.cfg # # Configuration file for non bandwidth server statistics # # # Define global options # 45
  • 46. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt,/usr/share/snmp/mibs/TCP- MIB.txt workdir: /var/www/mrtg/ # # CPU Monitoring # (Scaled so that the sum of all three values doesn't exceed 100) # Target[server.cpu]:ssCpuRawUser.0&ssCpuRawUser.0:craz33guy@localhost + ssCpuRawSystem.0&ssCpuRawSystem.0:craz33guy@localhost + ssCpuRawNice.0&ssCpuRawNice.0:craz33guy@localhost Title[server.cpu]: Server CPU Load PageTop[server.cpu]: < H1 >CPU Load - System, User and Nice Processes< /H1 > MaxBytes[server.cpu]: 100 ShortLegend[server.cpu]: % YLegend[server.cpu]: CPU Utilization Legend1[server.cpu]: Current CPU percentage load LegendI[server.cpu]: Used LegendO[server.cpu]: Options[server.cpu]: growright,nopercent Unscaled[server.cpu]: ymwd # # Memory Monitoring (Total Versus Available Memory) # Target[server.memory]: memAvailReal.0&memTotalReal.0:craz33guy@localhost Title[server.memory]: Free Memory PageTop[server.memory]: < H1 >Free Memory< /H1 > 46
  • 47. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure MaxBytes[server.memory]: 100000000000 ShortLegend[server.memory]: B YLegend[server.memory]: Bytes LegendI[server.memory]: Free LegendO[server.memory]: Total Legend1[server.memory]: Free memory, not including swap, in bytes Legend2[server.memory]: Total memory Options[server.memory]: gauge,growright,nopercent kMG[server.memory]: k,M,G,T,P,X # # Memory Monitoring (Percentage usage) # Title[server.mempercent]: Percentage Free Memory PageTop[server.mempercent]: < H1 >Percentage Free Memory< /H1 > Target[server.mempercent]: ( memAvailReal.0&memAvailReal.0:craz33guy@localhost ) * 100 / ( memTotalReal.0&memTotalReal.0:craz33guy@localhost ) options[server.mempercent]: growright,gauge,transparent,nopercent Unscaled[server.mempercent]: ymwd MaxBytes[server.mempercent]: 100 YLegend[server.mempercent]: Memory % ShortLegend[server.mempercent]: Percent LegendI[server.mempercent]: Free LegendO[server.mempercent]: Free Legend1[server.mempercent]: Percentage Free Memory Legend2[server.mempercent]: Percentage Free Memory # # New TCP Connection Monitoring (per minute) # 47
  • 48. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Target[server.newconns]: tcpPassiveOpens.0&tcpActiveOpens.0:craz33guy@localhost Title[server.newconns]: Newly Created TCP Connections PageTop[server.newconns]: < H1 >New TCP Connections< /H1 > MaxBytes[server.newconns]: 10000000000 ShortLegend[server.newconns]: c/s YLegend[server.newconns]: Conns / Min LegendI[server.newconns]: In LegendO[server.newconns]: Out Legend1[server.newconns]: New inbound connections Legend2[server.newconns]: New outbound connections Options[server.newconns]: growright,nopercent,perminute # # Established TCP Connections # Target[server.estabcons]: tcpCurrEstab.0&tcpCurrEstab.0:craz33guy@localhost Title[server.estabcons]: Currently Established TCP Connections PageTop[server.estabcons]: < H1 >Established TCP Connections< /H1 > MaxBytes[server.estabcons]: 10000000000 ShortLegend[server.estabcons]: YLegend[server.estabcons]: Connections LegendI[server.estabcons]: In LegendO[server.estabcons]: Legend1[server.estabcons]: Established connections Legend2[server.estabcons]: Options[server.estabcons]: growright,nopercent,gauge # 48
  • 49. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure # Disk Usage Monitoring # Target[server.disk]: dskPercent.1&dskPercent.2:craz33guy@localhost Title[server.disk]: Disk Partition Usage PageTop[server.disk]: < H1 >Disk Partition Usage /home and /var< /H1 > MaxBytes[server.disk]: 100 ShortLegend[server.disk]: % YLegend[server.disk]: Utilization LegendI[server.disk]: /home LegendO[server.disk]: /var Options[server.disk]: gauge,growright,nopercent Unscaled[server.disk]: ymwd 14.2 Testing The Configuration The next step is to test that MRTG can load the configuration file correctly. Restart SNMP to make sure the disk monitoring commands in the snmpd.conf file are activated. Run the /usr/bin/mrtg command followed by the name of the configuration file three times. If all goes well, MRTG will complain only about the fact that certain database files don't exist. MRTG then creates the files. By the third run, all the files are created and MRTG should operate smoothly. [root@bigboy tmp]# service snmpd restart [root@bigboy tmp]# env LANG=C /usr/bin/mrtg /etc/mrtg/server-stats.cfg 14.3 Creating A New MRTG Index Page To Include This File Use the indexmaker command and include your original MRTG configuration file from Chapter 22, " Monitoring Server Performance", (/etc/mrtg/mrtg.cfg) plus the new one you created (/etc/mrtg/server-stats.cfg). 49
  • 50. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure [root@bigboy tmp]# indexmaker --output=/var/www/mrtg/index.html /etc/mrtg/mrtg.cfg /etc/mrtg/server-stats.cfg 14.4 Configuring cron To Use The New MRTG File The final step is to make sure that MRTG is configured to poll your server every five minutes using this new configuration file. To do so, add this line to your /etc/cron.d/mrtg file. 0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/server-stats.cfg Some versions of Linux require you to edit your /etc/crontab file instead. See Chapter 22, " Monitoring Server Performance", for more details. You will also have to restart cron with the service crond restart for it to read its new configuration file that tells it to additionally run MRTG every five minutes using the new MRTG configuration file. [root@bigboy tmp]# service crond restart 15 Monitoring Non Linux MIB Values All the MIBs mentioned so far are for Linux systems; other types of systems will need additional MIBs whose correct installation may be unclear in user guides or just not available. In such cases, you'll need to know the exact value of the OID. 15.1 Scenario Imagine that your small company has purchased a second-hand Cisco switch to connect its Web site servers to the Internet. The basic MRTG configuration shown in Chapter 22, " Monitoring Server Performance", provides the data bandwidth statistics, but you want to measure the CPU load the traffic is having on the device, as well. Downloading MIBs from Cisco and using them with the snmpget command was not a success. You do not know what to do next. Find The OIDs When MIB values fail, it is best to try to find the exact OID value. Like most network equipment manufacturers, Cisco has an FTP site from which you can download both MIBs and OIDs. The SNMP files for Cisco's devices can be found at ftp.cisco.com in the /pub/mibs directory; OIDs are in the oid directory beneath that. 50
  • 51. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure After looking at all the OID files, you decide that the file CISCO-PROCESS-MIB.oid will contain the necessary values and find these entries inside it. "cpmCPUTotalPhysicalIndex" "1.3.6.1.4.1.9.9.109.1.1.1.1.2" "cpmCPUTotal5sec" "1.3.6.1.4.1.9.9.109.1.1.1.1.3" "cpmCPUTotal1min" "1.3.6.1.4.1.9.9.109.1.1.1.1.4" "cpmCPUTotal5min" "1.3.6.1.4.1.9.9.109.1.1.1.1.5" "cpmCPUTotal5secRev" "1.3.6.1.4.1.9.9.109.1.1.1.1.6" "cpmCPUTotal1minRev" "1.3.6.1.4.1.9.9.109.1.1.1.1.7" "cpmCPUTotal5minRev" "1.3.6.1.4.1.9.9.109.1.1.1.1.8" 15.2 Testing The OIDs As you can see, all the OIDs are a part of the same tree starting with 1.3.6.1.4.1.9.9.109.1.1.1.1. The OIDs provided may be incomplete, so it is best to use the snmpwalk command to try to get all the values below this root first. [root@bigboy tmp]# snmpwalk -v1 -c craz33guy cisco-switch 1.3.6.1.4.1..9.9.109.1.1.1.1 SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.2.1 = INTEGER: 0 SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.3.1 = Gauge32: 32 SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.4.1 = Gauge32: 32 SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.5.1 = Gauge32: 32 [root@bigboy tmp]# Although listed in the OID file, 1.1.1.1.6, 1.1.1.1.7, and 1.1.1.1.8 are not supported. Notice also how SNMP has determined that the first part of the OID value (1.3.6.1.4.1) in the original OID file maps to the word "enterprise". Next, you can use one the snmpget command to set only one of the OID values returned by snmpwalk. [root@bigboy tmp]# snmpget -v1 -c craz33guy cisco-switch 51
  • 52. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure enterprises.9.9.109.1.1.1.1.5.1 SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.5.1 = Gauge32: 33 [root@bigboy tmp]# Success! Now you can use this OID value, enterprises.9.9.109.1.1.1.1.5.1, for your MRTG queries. 16 Speeding up MRTG with RRDtool MRTG is a very useful program but it has a limitation. All the graphs and web pages are recreated each time a device is polled. This can potentially overload your MRTG server especially if you have a large number of monitored devices and the graphs take more than five minutes to generate. RRDtool is an application written by the creator of MRTG that can store general purpose data, but generates graphs on demand. Integrating MRTG with RRDtool can have very noticeable performance benefits. The example that follows will show you how to quickly implement a general purpose solution. 16.1 Scenario The use of RRDtool is needed to reduce the load on a monitoring server that has been experiencing very sluggish performance due to the amount of MRTG graphs it has to regenerate every polling cycle.  Due to space constraints, the RRD database needs to be located in the /var partition.  The server has a default Apache configuration with the CGI files needed for dynamically generated content being located in the /var/www/cgi-bin directory.  A CGI script is required that will read the new MRTG data in RRDtool format.  The MRTG configuration file is /etc/mrtg/mrtg.cfg. Here's how to proceed. 16.2Installing RRDtool The RRDtool and RRDtool PERL module file can be downloaded from its website at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/, but installation can be tricky as the installation program may look for certain supporting libraries in the wrong directories. 52
  • 53. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Fortunately the prerequisite rrdtool and rrdtool-perl packages now come as part of most Linux distributions. For more details on installing packages, see Chapter 6, "Installing Linux Software"). 16.3 Storing the MRTG Data in RRDtool Format This phase of the integration process can be done in a few minutes, but the steps can be tricky:  The first step is to add some new options to your cfgmaker command. The first indicates that MRTG should only store rrdtool formatted data, and the second defines the /var/mrtg directory in which it should be stored. For added security, the directory should be external to your web server's document root. --global 'LogFormat: rrdtool' --global "workdir: /var/mrtg" --global 'IconDir: /mrtg' Finally, you should also specify an icon directory which specifies the location of all miscellaneous MRTG web page icons. The RRD web interface script we'll install later uses an incorrect location. The icon directory /mrtg is actually a partial URL location. In this Fedora scenario we are using the default Apache configuration which locates the MRTG icon files in the /var/www/mrtg directory. If you are using a non default Apache MRTG configuration or are using other Linux distributions or versions you may have to copy the icons to the custom directory in which the MRTG PNG format icon files are located. ". The next step is to create the data repository directory /var/mrtg and make it be owned by the apache user and process that runs the default Linux web server application. [root@bigboy tmp]# mkdir /var/mrtg [root@bigboy tmp]# chown apache /var/mrtg [root@bigboy tmp]# 53
  • 54. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Note: If you are using SELinux you'll have to change the context of this directory to match that of the /var/www/html directory so that the apache process will be able to read the database files when your CGI script needs them. These commands compare the contexts of the both directories and apply the correct set to /var/mrtg. [root@bigboy tmp]# ls -alZ /var/www | grep html drwxr-xr-x root root system_u:object_r:httpd_sys_content_t html [root@bigboy tmp]# ls -alZ /var | grep mrtg drwxr-xr-x apache root root:object_r:var_t mrtg [root@bigboy tmp]# chcon -R -u system_u -r object_r -t httpd_sys_content_t /var/mrtg [root@bigboy tmp]# We now need to test that the RRD files are being created correctly. Run MRTG using the /etc/mrtg/mrtg.cfg file as the source configuration file then test to see if the contents of the / var/mrtg directory have changed. Success! [root@bigboy tmp]# ls /var/mrtg/ localhost_192.168.1.100.rrd [root@bigboy tmp]# The files are being created properly. Now we need to find a script to read the new data format and present it in a web format. This will be discussed next. 16.4 The MRTG / RRDtool Integration Script The MRTG website recommends the script located on the mrtg-rrd website (http://www.fi.muni.cz/~kas/mrtg-rrd/) as being a good one to use. Let's go ahead and install it. Download the script using wget. The site lists several versions; make sure you get the latest one. [root@bigboy tmp]# wget ftp://ftp.linux.cz/pub/linux/people/jan_kasprzak/mrtg- rrd/mrtg-rrd-0.7.tar.gz 54
  • 55. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure --12:42:12-- ftp://ftp.linux.cz/pub/linux/people/jan_kasprzak/mrtg-rrd/mrtg- rrd-0.7.tar.gz => `mrtg-rrd-0.7.tar.gz' Resolving ftp.linux.cz... 147.251.48.205 Connecting to ftp.linux.cz|147.251.48.205|:21... connected. Logging in as anonymous ... Logged in! ... ... ... 15:24:50 (53.53 KB/s) - `mrtg-rrd-0.7.tar.gz' saved [20863] [root@bigboy tmp]# ls mrtg-rrd-0.7.tar.gz [root@bigboy tmp]# Extract the contents of the tar file. [root@bigboy tmp]# tar -xzvf mrtg-rrd-0.7.tar.gz mrtg-rrd-0.7/ mrtg-rrd-0.7/COPYING mrtg-rrd-0.7/FAQ mrtg-rrd-0.7/TODO mrtg-rrd-0.7/Makefile mrtg-rrd-0.7/mrtg-rrd.cgi mrtg-rrd-0.7/ChangeLog [root@bigboy tmp]# Create the /var/www/cgi-bin/mrtg directory and copy the mrtg-rrd.cgi file to it. [root@bigboy tmp]# mkdir -p /var/www/cgi-bin/mrtg [root@bigboy tmp]# cp mrtg-rrd-0.7/mrtg-rrd.cgi /var/www/cgi-bin/mrtg/ [root@bigboy tmp]# 55
  • 56. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure Edit the mrtg-rrd.cgi file and make it refer to the /etc/mrtg/mrtg.cfg file for its configuration details, or you can specify all the .cfg files in your /etc/mrtg directory. # # File: mrtg-rrd.cgi (Single File) # # EDIT THIS to reflect all your MRTG config files BEGIN { @config_files = qw(/etc/mrtg/mrtg.cfg); } # # File: mrtg-rrd.cgi (multipl .cfg files) # # EDIT THIS to reflect all your MRTG config files BEGIN { @config_files = </etc/mrtg/*.cfg>; } Once installed, RRDtool operates transparently with MRTG. You'll have to remember to add the RRD statements to any new MRTG configurations and also add the configuration file to the CGI script. Our monitoring server can now breathe a little easier. 17 Troubleshooting The troubleshooting techniques for advanced MRTG ", but because you have done some customizations you'll have to go the extra mile.  Verify the IP address and community string of the target device you intend to poll.  Make sure you can do an SNMP walk of the target device. If not, revise your access controls on the target device and any firewall rules that may impede SNMP traffic.  Ensure you can do an SNMP get of the specific OID value listed in your MRTG configuration file.  Check your MRTG parameters to make sure they are correct. Gauge values defined as counter and vice versa will cause your graphs to have continuous zero values. Graph results that are eight times what you expect may have the bits parameter set.  There are a few errors common to initial RRDtool integration. 56
  • 57. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure  Web messages like this where the reference to the MRTG configuration file in the CGI 13.5.2script was incorrect Error: Cannot open config file: No such file or directory "Permission Denied" web messages are usually caused by incorrect file permissions and / or SELinux contexts Error: RRDs::graph failed, opening '/var/mrtg/localhost_192.168.1.100.rrd': Permission denied Errors in the /var/log/httpd/errorlog file referring to files or directories that don't exist can be caused by an incorrect IconDir statement in the MRTG configuration file. [Wed Jan 04 15:42:13 2006] [error] [client 192.168.1.102] File does not exist: /var/www/html/var, referer: http://bigboy/cgi-bin/mrtg/mrtg-rrd.cgi/ [Wed Jan 04 15:45:46 2006] [error] [client 192.168.1.102] script not found or unable to stat: /var/www/cgi-bin/mrtg/mrtg-l.png, referer: http://bigboy/cgi-bin/mrtg/mrtg-rrd.cgi/ Errors caused by not installing the pre-requisite RRD RPM modules rrdtool, perl-RRD- Simple and rrdtool-perl. ERROR: could not find RRDs.pm. Use LibAdd: in mrtg.cfg to help mrtg find RRDs.pm These quick steps should be sufficient in most cases and will reward you with a more manageable network. 57
  • 58. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure CONCLUSION By this Project we are able to graph most SNMP MIB values available on any type of device. MRTG is an excellent, flexible monitoring tool and should be considered as a part of any systems administrator's server management plans. Server monitoring is always a good practice, because it can help you predict when things are going to go wrong or long term trends in your Web traffic. MRTG can be expanded not only to monitor traffic on your server's NIC cards, but also to graph many of the statistics listed in top, free, and vmstat 58
  • 59. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure REFERENCES CMU-SNMP for Linux: ftp://sunsite.unc.edu/pub/Linux/system/network/admin/cmu-snmp-linux-3.2- bin.tar.gz PERL 5 module extension for CMU-SNMP: ftp:/ftp.wellfleet.com/netman/snmp/perl5/SNMP.tar.gz Multi Router Traffic Grapher (MRTG): http://www.ee.ethz.ch/~oetiker/webtools/mrtg/mrtg.html GD Graphic Library: (required by MRTG) http://www.boutell.com/gd/ Perl 5.003: (required by MRTG) http://www.perl.com/perl/info/software.html MRTG mailing lists: Discussion list Router Stats: http://www.scn.de/~iain/router-stats/ SNMP Network Management Software: http://wwwsnmp.cs.utwente.nl/software/ Scotty: http://wwwsnmp.cs.utwente.nl/~schoenw/scotty/ Scotty for Linux: ftp://sunsite.unc.edu/pub/Linux/system/network/admin/tkined-1.3.4+scotty- ELF.tar.gz 59
  • 60. Project Report: Centralized Monitoring Station for IT Computing and Network Infrastructure 60