![]() |
Owl Monitoring SystemFull Installation Manual |
The Owl Monitoring System was designed such that the Owl sensor hosts need not be under the same administrative control as the Owl manager host. In fact, each sensor in a set of Owl sensors may be under administrative control of different organizations and still report to a single Owl manager.
An Owl sensor may provide its data to more than one Owl manager. In such situations, the managers operate independently of each other and do not know of each other's existence. Throughout the installation manuals, it will be assumed that an installation will have a single manager. Special instructions for multiple-manager environments will be given where needed.
Contact between sensors and their manager may be initiated and performed by the sensor or the manager. This is a configuration decision that must be made on a case by case basis. The push or pull model may cover all of an Owl manager's sensors. For example, a particular manager may initiate sensor data retrieval from all of its sensors. Also, the push/pull model may be specific to each sensor, so a particular manager could retrieve data from one sensor but wait for another sensor to provide its data.
This document provides an operational overview of the Owl system, installation instructions for the Owl and supporting third-party software. Configuration instructions for the Owl software are also provided, along with configuration information for the required third-party software.
There are two installation manuals, one for the manager and one for the sensor. Sensor administrators only need to read the Owl Sensor Installation Manual. Manager administrators only need to read the Owl Manager Installation Manual As you might expect, those who are administrators for both manager and sensor hosts must read both installation guides.
Owl Manager Installation Manual | Owl Sensor Installation Manual |
An Owl sensor is configured to send DNS queries to a set of DNS nameservers. The queries may be for a variety of DNS resource record types, and they are sent to a specific nameserver to request data about a specific target host. For example, one query might request that the D root nameserver return NS records for the "www.example.com" host, while another query might request that a local (non-root) nameserver return AAAA records for the "mail.example.com" host. Data will be collected for each configured query and given to the Owl manager to whom the sensor reports. The different sensors in the sensor set may perform different queries or the same queries.
A configuration file controls the behavior of the Owl sensor. This file defines the queries that the sensor will perform, the Owl manager (or managers) to whom the sensor reports, locations of data and log files, frequency of queries, and other important behaviors.
An Owl sensor may provide its data to more than one Owl manager. In such situations, the managers operate independently of each other and do not know of each other's existence. Throughout the installation manuals, it will be assumed that an installation will have a single manager. Special instructions for multiple-manager environments will be given where needed.
This document provides an operational overview of the Owl system, installation instructions for the Owl sensor and the third-party software that supports it. Instructions for configuring the Owl sensor are also given.
Sections:
0. | Setting Up Sensors for the Owl Monitoring System | ||
1. | Operational Overview of the Owl Monitoring System | ||
2. | Owl Sensor Installation | ||
3. | An Interlude on Sensor Queries | ||
4. | Adding Owl Sensors to an Existing Owl Installation | ||
5. | Adding New Queries to Existing Owl Sensors | ||
6. | Owl Sensor Commands | ||
The Owl Monitoring System has a conceptually simple method of operations. There is a manager host and a set of sensor hosts. The sensors periodically make a set of DNS queries and time how long it takes to get a response. The response times are saved to data files. After a certain amount of time, the data files are transferred from the sensor to the manager host. For any particular sensor, this data transfer may be configured to be initiated by either the manager or by the sensor. The manager transfers the data into a monitoring package, which allows easy display of the data.
Below is a diagram of the flow of Owl sensor data through two Owl monitoring installations.
Take note of the following:
The remainder of this section provides a little more information on the components that make up the sensor and manager software.
An Owl sensor makes timed DNS queries and records the time required for a response. The response time data are transferred to the Owl manager after a certain period of time. The transfer may occur at the instigation of either the manager or the sensor. The Owl sensor's three primary programs that handle these tasks are described below.
The Owl sensor's actions are controlled by a configuration file. This file defines the queries that must be performed, how often the queries must happen, how frequently the response data files are transferred to the manager, data and logging information, and other required information.
The owl-dnstimer program performs periodic DNS lookups, as specified by the configuration file. These lookups are timed so that the sensor can record how long it took to make a particular request. The configuration parameters used by owl-dnstimer include the nameserver to query, the target zone to query about, the type of DNS record, and the frequency of queries.
If the Owl system is configured so that the sensor transfers its data to the manager, then the transfer is performed by the owl-transfer program. The transfers are performed using rsync over an ssh connection. The sensor host must be able to do a password-less ssh into the manager host. The frequency of transfers is defined in the configuration file. Choosing the transfer frequency is a trade-off between lower impact on the sensor, manager, and network (less frequent) and lower latency of data display on the manager (more frequent.)
The owl-sensord program is an optional controller for owl-dnstimer and owl-transfer. If used, it will run these daemons and restart them if they stop. If one of the daemons tries to restart too often in too short a time, then owl-sensord will warn an administrator (via email) and temporarily stop trying to execute the program. owl-sensord is not required in order to run owl-dnstimer and owl-transfer; the Owl sensor can run fine without it. However, it provides a convenient way to keep them running.
As part of the installation process, you must decide whether you will run the Owl sensor using the owl-sensord daemon to control execution or if you will run owl-dnstimer and owl-transfer directly.
The Owl sensor has a number of programs for administrative support. Most of these are not required, but they assist in sensor administration. In particular, the owl-dataarch and owl-heartbeat programs are intended to run as cron jobs. owl-dataarch moves "old" sensor data into an archive directory, in order to keep the sensor-manager data transfer from bogging down (within rsync) as the data collection grows. The owl-heartbeat program periodically touches a webpage hosted on the Owl manager, in order to let the manager know that the sensor host is still running. owl-heartbeat is not critical and can be left off, but owl-dataarch is essential. There are a few other administrative commands available; see section 6 for a brief description of all the Owl sensor commands.
The Owl manager's purpose is to receive sensor data and make it available for display. The majority of the Owl manager is actually third-party packages, and the Owl manager software provides the "glue" that allows the Owl sensor data to work with those packages. In particular, Owl provides data to the Nagios monitoring system, which then displays the data.
The Owl sensors may report to managers that are under separate administrative control. A particular installation of the Owl Monitoring System may use multiple manager hosts, but this document assumes that there is only one.
The manager can have either an active or passive role in the process of moving Owl sensor data from the sensor to the manager. If the manager is active, it will use the owl-transfer-mgr program to pull sensor data from the Owl sensor host. If the manager is passive, then the sensor will transfer the data when it sees fit.
The are two groups of "glue" programs that Owl uses to interface with Nagios. The owl-dnswatch and owl-perfdata inject Owl sensor DNS response data into Nagios for display and graphing. The owl-sensor-heartbeat.cgi and owl-stethoscope programs insert Owl "heartbeat" data into Nagios so it can track availability of the Owl sensors.
The Owl manager also has a number of programs for administrative support. Most of these are not required, but they assist in manager administration. In particular, the owl-archdata and owl-monthly programs are intended to run as cron jobs. owl-archdata moves "old" sensor data into an archive directory, in order to keep data transfer from the sensor from bogging down as the data collection grows. The owl-monthly program archives the previous month's sensor data into a compressed tarfile. owl-newsensor and owl-initsensor assist in setting up the manager for a new sensor. There are a few others available; see section 7 for a brief description of all the Owl manager commands.
The following third-party software packages are used in with this distribution of the Owl Monitoring System. The packages are not included with the Owl distribution, but must be retrieved as required for your operating system. These packages must be installed on the manager:
Monitoring system.
| | Monitoring system.
| | Provides graphing for Nagios.
| | Database tool for use by nagiosgraph.
| | Draws graphs for nagiosgraph of data stored by rrdtool.
| |
Instructions are given in this document for integrating these packages with the Owl software. It would be for the best to read the installation instructions below for each package prior to installing them.
Different management, graphing, and database tools may be used instead of those listed above, if so desired. Integrating the Owl manager software with other packages is beyond the scope of this manual.
David Josephsen's Building a Monitoring Infrastructure with Nagios is very helpful for understanding Nagios and how the various pieces work together. If you're going to be running a Nagios monitor, you would be doing yourself a favor to read this book.
Owl sensors generate a large amount of DNS response data. For example, a sensor running five queries, once a minute each, generated roughly 517KB of data per day. (Due to the way the data are stored, this will be affected by the amount length of target and nameserver hostnames used.) You may or may not want to retain any of this data. The data files can be useful for rebuilding the Owl manager's rrdtool databases (or building with different parameters.)
On both the Owl manager and the Owl sensors, sensor data are stored in a data directory. After a few days, the data files are moved to a data archive directory. A full month's data files will be collapsed into a compressed format.
The Owl Monitoring System provides tools that assist with managing Owl sensor data. After the sensor data have been moved from the sensor to the manager, the sensor has no further need of the data. On the manager, after the data have been moved into the data archive, the Owl Monitoring System has no further need of the old data; it may be retained or deleted. (The data will have been moved into the manager's rrdtool databases by then, so the data is not lost.) Each installation must decide their own data retention policy.
This section gives installation procedures for installing the Owl Monitoring software on sensor hosts. Installation of the Owl sensor is relatively simple. This requires the creation of a sensor account, SSH initialization, software installation, and a few other set-up activities. The most complicated aspect is likely to be coordinating SSH access with the Owl manager host. The required actions for installing an Owl sensor are described in this section.
The installation instructions discuss software installation of Owl software and supporting third-party software packages. There are a number of Owl-specific configuration actions that must be performed, and these are detailed in section 4.
The Owl sensors require Perl. You must install it on your system if it isn't already available. If the sensor host's Perl or operating system does not support threading, then each query will run in its own process, rather than all queries running in their own threads within a single process.
Below is a diagram of the flow of Owl sensor data on the Owl sensor. The details may not make sense at this stage, but this will show how the various pieces fit together.
The Owl sensor will run several daemons and cron jobs. It is a good idea, but not an imperative one, for these to run as their own user. It is easier to segregate the Owl sensor's actions, processes, and files if they are performed by a dedicated user.
However, it is certainly acceptable to run them as yourself or some other existing user. If you choose to do this, you may disregard the rest of this section.
There are no requirements on the Owl username; it may be anything that makes sense to you. Some possibilities are: owl, owl-sensor, owl-user, bob, and tricorder.
To create a new user, use your operating system's user-creation command (e.g., adduser or useradd) to create a new unprivileged user.
Examples of commands to create a new user:
Operating System | Example Command | Notes | ||
Centos | useradd hoots-mon | remember to add a password | ||
FreeBSD | adduser | you'll be prompted for all user information | ||
Mac OS X | "Users & Groups" in System Preferences | you'll be prompted for all user information | ||
Owl sensor data are transferred to the Owl manager using the rsync and ssh commands. You must initialize the Owl sensor user's SSH environment by generating keys for the ssh command to use. This may be dependent on your operating system or the version of SSH installed.
The OpenSSH package is a widely available SSH package, and its ssh-keygen command is used to create SSH keys. This manual will assume that OpenSSH is being used.
There are no specific key parameters required by Owl. The generated key must be sufficient to allow the sensor and the manager to transfer files without having to manually enter a password.
Example of a command to generate a new SSH key:
$ ssh-keygen -b 1024 -t dsa
If the Owl sensor will be pushing data to the manager, then the newly generated public key must be transferred to the Owl manager host. This key must be added to the list of authorized keys for the Owl manager's account. For OpenSSH, the public key will be in a file called id_rsa.pub and it will be added to the authorized_keys file on the manager host. Once the sensor's public key has been added to the Owl manager, you must ensure that the Owl sensor user may successfully transfer files to the manager without need of a password.
If the Owl manager will be pulling data from the sensor, then you must do the following things. First, install the rrsync utility. This is a front-end to rsync that restricts the locations to which rsync may copy data. rrsync is available here: http://ftp.samba.org/pub/unpacked/rsync/support/rrsync. Next, the manager's administrator must provide you with a public SSH key. This public key must be added to the authorized_keys file belonging to the Owl sensor account.
The authorized_keys entry should look something like this:
command="/usr/local/bin/rrsync /owl/data" ssh-rsa AAAAB3Nza...A3Q== owl-manager@owl.example.com
You must select a place to install the Owl sensor's files. The files and directories may live in the Owl sensor user's home directory, a subdirectory beneath that home directory, or somewhere else entirely. For ease of finding these files later, it may be helpful to keep the Owl sensor's files all beneath a single directory.
There are a set of directories that will be used by the Owl sensor: bin, conf, and perllib. These names shouldn't be changed.
Additional modifications to the Owl sensor environment will be discussed in section 4.1.
Several installation examples are given below. For each, it is assumed the Owl sensor user is named owl-sensor.
Installation example 1: In this case, the software will be installed in the sensor user's home directory by unpacking a tar file located in /tmp/owl-sensor.tar.gz.
$ cd ~owl-sensor $ tar xzf /tmp/owl-sensor.tar.gz
Installation example 2: In this case, the software will be installed in a subdirectory within the sensor user's home directory by unpacking a tar file located in /tmp/owl-sensor.tar.gz.
$ mkdir ~owl-sensor/owl-dir $ cd ~owl-sensor/owl-dir $ tar xzf /tmp/owl-sensor.tar.gz
There is a set of Perl modules required by the Owl sensor. These should be available through CPAN.
Modules required by the Owl sensor:
|
|
If the two threads-related modules are unavailable for your system, or if your version of Perl doesn't support threading, then the Owl sensor will run each query in its own process. In this case, these two modules are not needed.
Before running your Owl Monitoring System, you must consider a number of questions. How many Owl sensors will be providing data to the Owl manager? How many queries will the sensors be performing? What queries will the sensors be performing? Will each sensor be performing the same queries? Will the sensor be providing "heartbeat" information to the manager? Who will transfer the sensor data to the manager? These questions are important for the sensor administrator, but they are critical for the manager administrator.
A query consists of three parts: a nameserver, a target host, and a query type. Responses to each query will be kept in their own datafile. The amount of data generated by each sensor will depend on the number of queries and the frequency with which the queries are performed.
The sensors will have to gather the data and more queries will result in more data that will be collected. The data will all have to be transferred from the sensor to the manager. If the manager is configured for graphing, then data from each query will be stored in its own database. These databases can be quite large, so the manager host must have a good amount of available disk space.
The administrators of an Owl sensor and the Owl manager (if they are different people) must coordinate the queries that the sensor will be performing. The manager's administrator can request that a set of queries be performed, but it is up to the sensor's administrator to actually configure the sensor for the queries. The manager is effectively at the mercy of the sensor.
Sensor data filenames contain metadata about the sensor data contained therein. Therefore, the filenames will reflect the queries that generated the files. The fields are separated by commas and have this format:
121129.1701,seattle-sensor,example.com,a.root-servers.net,A.dns 121129.1701,seattle-sensor,example.com,m.root-servers.net,A.dns 121129.1701,seattle-sensor,example.com,m.root-servers.net,NS.dns 121129.1701,portland-sensor,example.com,a.root-servers.net,A.dns
Breaking out the fields of the last example above gives this table:
Field | Purpose | Example | |||
Timestamp | File creation timestamp; (YYMMDD.hhmm) | 121129.1701 | |||
Sensor Name | Name of sensor that recorded data | portland-sensor | |||
Target Name | Host whose name was target of DNS lookups | example.com | |||
Nameserver Name | Name of queried nameserver | a.root-servers.net | |||
Query Type | Type of DNS query | A | |||
File suffix | File suffix | .dns |
The Owl Monitoring System provides utilities to transfer sensor data from the sensor to the manager. One utility, owl-transfer runs on the sensor and transfers the data to the manager. The other utility, owl-transfer-mgr runs on the manager and transfers the data from the sensor. Both utilities are front-ends for the rsync program and run using an ssh-initiated connection.
In both cases, the transferring host must be able to ssh into the remote host without a password. The FOSS rrsync ("restricted rsync", written by Joe Smith, modified by Wayne Davison) is used to restrict the transferring host's access to the remote host.
The administrators of the sensor and the manager must agree on which host will be transferring data to the manager. The appropriate SSH public keys from the transferring host must be placed on the other host, so that the password-less ssh (as described above) may be performed.
An Owl system with multiple sensors may have a mixed transfer configuration. In such a system, some sensors will transfer their data to the manager, while the manager will transfer data from other sensors.
The heartbeat facility is not necessary for Owl operation and Owl runs perfectly well without it. However, it can provide a useful assurance to the manager administrator that the sensor is still working. Enabling this facility on the sensor is at the discretion of the sensor administrator. Since this requires a webserver to be running on the manager host, that aspect is at the discretion of the manager administrator.
Whenever a new Owl sensor is added to those handled by an Owl manager, there are a number of actions that must take place on both the sensor and the manager. These actions should be followed consecutively within each section. However, there is some amount of necessary interleaving of sensor actions and manager actions. For example, a particular sensor action may be required before a particular manager action.
It is acceptable for the Owl sensor and Owl manager to be under completely different administrative control. Each host may even be owned by different commercial, educational, governmental, or other such entity. All that is required is that there be some initial coordination between administrators when the sensor is first configured, along with (probably) aperiodic support from time to time later.
The discussions on adding Owl sensors assumes that the sensor and manager are under different administrative control. Required actions will not change if they are under unified administrative control, but they may be easier to accomplish.
This section describes the configuration and testing actions that must be undertaken in order to have a working Owl sensor. It assumes the installation procedures detailed in section 2.1 have been completed.
The Owl sensor and the Owl manager communicate by way of SSH and HTTP. If the Owl sensor is protected by a firewall (either on the sensor host itself or by an enterprise-level firewall) then the firewall must be configured to allow data to be transferred between the sensor and the manager. The direction of this transfer (initiated by sensor or initiated by manager) depends on the Owl configuration. These firewall modifications are far beyond the scope of this document.
You must generate a new SSH key for the sensor's Owl user.
If the sensor will be transferring data to the manager, then you must provide the public key to the administrator of the Owl manager. This key will allow owl-transfer to pass Owl sensor data to the manager. You must generate keys with key characteristics (e.g., length and type) that are acceptable to the manager's administrator.
If the manager will be pulling data from the sensor, then the manager's administrator must provide you with the manager host's public key. You must add this key to your authorized_keys file. This key will allow owl-transfer-mgr to retrieve Owl sensor data. If you have particular key requirements (e.g., length and type), then you must give them to the manager's administrator.
There are several pieces of information that must be provided by the administrator of the Owl manager. These include values for the sensor's configuration file and the sensor's name.
The following data will allow the sensor to work with the Owl manager as expected. These data are used, in conjunction with the manager keyword, in the sensor's configuration file.
Configuration Field | Purpose | Example | ||
ssh-user | User on Owl manager with which owl-transfer will connect
via ssh. This is only required if the sensor will be transferring data to the manager. | sensor-ottawa@owl-manager.example.com | ||
heartbeat | URL to provide "heartbeat" data to manager. This is only required if the sensor will be providing heartbeat data. | http://owl.example.com/cgi-bin/owl-sensor-heartbeat.cgi |
In addition to these data, it would be a good idea for you and the manager administrator to agree on a name for the sensor. This is not a hostname, but is a name by which your sensor will be known to the Owl Monitoring System. Name agreement between sensor and manager isn't required, but it will probably make things easier for you both in the long run to refer to the sensor by the same name.
The sensor name can be very generic, such as sensor42, sensor-d, or owl-us-east. It can also be very specific, such as washdc-1600-penn-ave-nw or cheltenham-bldg4. The intent is to provide distinguishing information to the intended audience of the DNS monitoring data. You should use names that easily distinguish sensors and are acceptable to the manager and sensor administrators.
The Owl configuration file must be initialized. The data fields include sensor information, query definitions, file locations, transfer intervals, and information about contacting the manager.
The following is a sample owl.conf file:
host name rome-sensor host sensor-args -config /home/owl/conf/owl.conf host transfer-args -conf /home/owl/conf/owl.conf host admin owl-mgmt bob@example.com query example.com d ns query example.com d a query . h A query . m data dir data data interval 60 log dir log remote ssh-user owl-mgr@owlmonitor.example.com remote heartbeat http://owlmonitor.example.com/cgi-bin/owl-sensor-heartbeat.cgi
The full definition of the configuration file may be found in the owl-config.pod manpage.
The sensor configuration file must include a set of query definitions. These are best specified by the administrator of the Owl manager, but you must be willing to generate and transfer the volume of data required by those queries. The impact won't be as large for the sensor as it will be for the manager since the manager is likely to have multiple sensors for which it will store data.
In addition, the manager-contact fields must be set with data provided by the administrator of the Owl manager. The ssh-user field is required for transfer of data files. The heartbeat field is optional, but must be specified if the sensor will provide "heartbeat" information to the manager.
When translated to the configuration file format, the example lines from section 4.3 will look like the following:
manager ssh-user sensor-ottawa@owl-manager.example.com manager heartbeat http://owl.example.com/cgi-bin/owl-sensor-heartbeat.cgi
At this point, the manager should be ready to accept data from the Owl sensor. Test that this will succeed with the following steps:
If the file was successfully transferred, then the Owl sensor is ready to start generating data and providing it to the manager.
If not, you must determine why the file wasn't transferred. owl-transfer uses the rsync and ssh commands to perform the transfer, so problem diagnosis should start with them.
As discussed in section 1.1, you must decide whether you will run the Owl sensor using the owl-sensord daemon or if you will run owl-dnstimer and owl-transfer directly. Both methods are acceptable; the first is more robust in the case of problems.
In any event, you must add entries to your operating system's boot process to start the appropriate Owl daemons. This is highly dependent on the operating system, and may involve such things as adding some lines to /etc/rc.local or adding a start-up file to /etc/rc.d/rc2.d.
These lines executing the Owl sensor daemons may look something like this:
/home/owl/bin/owl-sensord -config /home/owl/bin/conf/owl.confor
/home/owl/bin/owl-dnstimer -config /home/owl/bin/conf/owl.conf /home/owl/bin/owl-transfer -config /home/owl/bin/conf/owl.conf
It is incumbent upon the installer to determine the proper method of Owl start-up and the proper place to set up execution.
The owl-dataarch, owl-archold, and owl-heartbeat programs provide additional services for the Owl Monitoring System. These programs are not required, but they can ease some of the burden of administering an Owl sensor. However, due to the volume of data generated by an Owl sensor, it is strongly advised that owl-dataarch be run.
These programs may be set up as cron jobs. This will remove the necessity for the administrator to run these programs manually, and it will provide regularity to their execution.
owl-dataarch periodically moves sensor data from the data directory to an archive directory. It should be run on a daily basis.
owl-archold will create a compressed tarfile of a month's data that has been previously stored in the archive directory. It should be run on a monthly basis -- but not on the first or second day of the month. If the sensor host is used for much other than as an Owl sensor, then it would be best to run owl-archold whenever the system is normally at "off hours."
owl-heartbeat touches a webpage (specified in the configuration file) and is intended to inform the Owl manager that the sensor is up and running. This may be run as often as desired. The more frequently it's run, the more up-to-date is the information on the Owl manager. The less frequently it's run, the lower the granularity of the sensor-availability information on the manager. Once per minute or hour is probably sufficient for most purposes.
Program | Frequency of Execution |
owl-dataarch | once per day |
owl-archold | once per month, on the 3rd of the month or later |
owl-heartbeat | as desired, but once per minute or hour is reasonable |
Installation and configuration are complete for the Owl sensor. The Owl sensor is now ready to gather timing data on DNS queries. If the sensor will be transferring data to the Owl manager, it is ready for that task as well. To start the Owl sensor daemons, you can either start them manually (either by running just owl-sensord or by running both owl-dnswatch and owl-transfer) or by rebooting your system.
From time to time, you might decide to change the queries being performed by the Owl sensors. This could be adding new queries to some or all of the sensors, stopping some of the queries from being performed, or modifying existing queries. Regardless of the changes, there are actions that must be taken on the Owl manager and all affected Owl sensors.
It is relatively painless to modify queries for a sensor. The following actions must be performed:
If you have correctly set up your configuration file, then owl-dnstimer will start collecting data for the new queries.
Stopping collection of certain queries only requires deleting (or commenting out) the appropriate entries from the configuration file. owl-dnstimer, of course, must be restarted.
The Owl sensor has a number of Owl-specific commands required for its operation. This section provides a brief overview of the available commands. These are all Perl scripts and have self-contained man pages, which contain greater details about their use. These scripts are intended to run as an unprivileged user.
This section briefly describes the software used on the Owl sensor hosts.
An Owl sensor is configured to send DNS queries to a set of DNS nameservers. The queries may be for a variety of DNS resource record types, and they are sent to a specific nameserver to request data about a specific target host. For example, one query might request that the D root nameserver return NS records for the "www.example.com" host, while another query might request that a local (non-root) nameserver return AAAA records for the "mail.example.com" host. Data will be collected for each configured query and given to the Owl manager to whom the sensor reports. The different sensors in the sensor set may perform different queries or the same queries.
A configuration file controls the behavior of the Owl sensor. This file defines the queries that the sensor will perform, the Owl manager (or managers) to whom the sensor reports, locations of data and log files, frequency of queries, and other important behaviors.
An Owl sensor may provide its data to more than one Owl manager. In such situations, the managers operate independently of each other and do not know of each other's existence. Throughout the installation manuals, it will be assumed that an installation will have a single manager. Special instructions for multiple-manager environments will be given where needed.
This document provides an operational overview of the Owl system, installation instructions for the Owl manager and the third-party software that supports it. Instructions for configuring the Owl manager are also given.
Organization of Installation ManualSections:
1. Operational Overview of the Owl Monitoring SystemThe Owl Monitoring System has a conceptually simple method of operations. There is a manager host and a set of sensor hosts. The sensors periodically make a set of DNS queries and time how long it takes to get a response. The response times are saved to data files. After a certain amount of time, the data files are transferred from the sensor to the manager host. For any particular sensor, this data transfer may be configured to be initiated by either the manager or by the sensor. The manager transfers the data into a monitoring package, which allows easy display of the data. Below is a diagram of the flow of Owl sensor data through two Owl monitoring installations.
![]() Take note of the following:
The remainder of this section provides a little more information on the components that make up the sensor and manager software. 1.1. Owl Sensor OverviewAn Owl sensor makes timed DNS queries and records the time required for a response. The response time data are transferred to the Owl manager after a certain period of time. The transfer may occur at the instigation of either the manager or the sensor. The Owl sensor's three primary programs that handle these tasks are described below. The Owl sensor's actions are controlled by a configuration file. This file defines the queries that must be performed, how often the queries must happen, how frequently the response data files are transferred to the manager, data and logging information, and other required information. The owl-dnstimer program performs periodic DNS lookups, as specified by the configuration file. These lookups are timed so that the sensor can record how long it took to make a particular request. The configuration parameters used by owl-dnstimer include the nameserver to query, the target zone to query about, the type of DNS record, and the frequency of queries. If the Owl system is configured so that the sensor transfers its data to the manager, then the transfer is performed by the owl-transfer program. The transfers are performed using rsync over an ssh connection. The sensor host must be able to do a password-less ssh into the manager host. The frequency of transfers is defined in the configuration file. Choosing the transfer frequency is a trade-off between lower impact on the sensor, manager, and network (less frequent) and lower latency of data display on the manager (more frequent.) The owl-sensord program is an optional controller for owl-dnstimer and owl-transfer. If used, it will run these daemons and restart them if they stop. If one of the daemons tries to restart too often in too short a time, then owl-sensord will warn an administrator (via email) and temporarily stop trying to execute the program. owl-sensord is not required in order to run owl-dnstimer and owl-transfer; the Owl sensor can run fine without it. However, it provides a convenient way to keep them running. As part of the installation process, you must decide whether you will run the Owl sensor using the owl-sensord daemon to control execution or if you will run owl-dnstimer and owl-transfer directly. The Owl sensor has a number of programs for administrative support. Most of these are not required, but they assist in sensor administration. In particular, the owl-dataarch and owl-heartbeat programs are intended to run as cron jobs. owl-dataarch moves "old" sensor data into an archive directory, in order to keep the sensor-manager data transfer from bogging down (within rsync) as the data collection grows. The owl-heartbeat program periodically touches a webpage hosted on the Owl manager, in order to let the manager know that the sensor host is still running. owl-heartbeat is not critical and can be left off, but owl-dataarch is essential. There are a few other administrative commands available; see section 6 for a brief description of all the Owl sensor commands. 1.2. Owl Manager OverviewThe Owl manager's purpose is to receive sensor data and make it available for display. The majority of the Owl manager is actually third-party packages, and the Owl manager software provides the "glue" that allows the Owl sensor data to work with those packages. In particular, Owl provides data to the Nagios monitoring system, which then displays the data. The Owl sensors may report to managers that are under separate administrative control. A particular installation of the Owl Monitoring System may use multiple manager hosts, but this document assumes that there is only one. The manager can have either an active or passive role in the process of moving Owl sensor data from the sensor to the manager. If the manager is active, it will use the owl-transfer-mgr program to pull sensor data from the Owl sensor host. If the manager is passive, then the sensor will transfer the data when it sees fit. The are two groups of "glue" programs that Owl uses to interface with Nagios. The owl-dnswatch and owl-perfdata inject Owl sensor DNS response data into Nagios for display and graphing. The owl-sensor-heartbeat.cgi and owl-stethoscope programs insert Owl "heartbeat" data into Nagios so it can track availability of the Owl sensors. The Owl manager also has a number of programs for administrative support. Most of these are not required, but they assist in manager administration. In particular, the owl-archdata and owl-monthly programs are intended to run as cron jobs. owl-archdata moves "old" sensor data into an archive directory, in order to keep data transfer from the sensor from bogging down as the data collection grows. The owl-monthly program archives the previous month's sensor data into a compressed tarfile. owl-newsensor and owl-initsensor assist in setting up the manager for a new sensor. There are a few others available; see section 7 for a brief description of all the Owl manager commands. The following third-party software packages are used in with this distribution of the Owl Monitoring System. The packages are not included with the Owl distribution, but must be retrieved as required for your operating system. These packages must be installed on the manager:
Instructions are given in this document for integrating these packages with the Owl software. It would be for the best to read the installation instructions below for each package prior to installing them. Different management, graphing, and database tools may be used instead of those listed above, if so desired. Integrating the Owl manager software with other packages is beyond the scope of this manual. David Josephsen's Building a Monitoring Infrastructure with Nagios is very helpful for understanding Nagios and how the various pieces work together. If you're going to be running a Nagios monitor, you would be doing yourself a favor to read this book. 1.3. Data Retention on Owl HostsOwl sensors generate a large amount of DNS response data. For example, a sensor running five queries, once a minute each, generated roughly 517KB of data per day. (Due to the way the data are stored, this will be affected by the amount length of target and nameserver hostnames used.) You may or may not want to retain any of this data. The data files can be useful for rebuilding the Owl manager's rrdtool databases (or building with different parameters.) On both the Owl manager and the Owl sensors, sensor data are stored in a data directory. After a few days, the data files are moved to a data archive directory. A full month's data files will be collapsed into a compressed format. The Owl Monitoring System provides tools that assist with managing Owl sensor data. After the sensor data have been moved from the sensor to the manager, the sensor has no further need of the data. On the manager, after the data have been moved into the data archive, the Owl Monitoring System has no further need of the old data; it may be retained or deleted. (The data will have been moved into the manager's rrdtool databases by then, so the data is not lost.) Each installation must decide their own data retention policy.
2. Owl Manager Installation
This section gives installation procedures for setting up the Owl Monitoring software on manager hosts. There are two sections, an installation section and a configuration section. The installation instructions discuss software installation of Owl software and supporting third-party software packages. System configuration is also detailed in this section, but only for non-Owl-specific configuration. There are a number of Owl-specific configuration actions that must be performed and these are detailed in section 4. The Owl manager requires Perl. You must install it on your system if it isn't already available. 2.1. Manager InstallationThe basic Owl manager configuration consists of the Owl software and the Nagios monitoring system. This configuration allows for monitoring of current sensor status, but does not provide any view into past monitoring data. In order to get the historical view, especially as provided by a graphing capability, several additional software packages must be installed on the manager. Since the historical view provides a great deal of useful information, these instructions will assume that you will be installing the additional software.It is possible (and not too difficult) to run multiple instances of the third-party software packages (e.g., Nagios, nagiosgraph) on a single host, but care must be taken when configuring, building, and installing the software. Without careful work, an existing installation may be overwritten by a later installation. These instructions assume that only one instance of each package will be installed on the manager host. Below is a diagram of the flow of Owl sensor data on the Owl manager. The details may not make sense at this stage, but this will show how the various pieces fit together.
![]() 2.1.1. Preparatory StepsSeveral actions must be taken on the manager host to prepare it for installing the required software. These steps are listed below. During the installation procedures, don't be shy about backing up things; it might not be necessary, but better safe than sorry.
2.1.2. Installing Third-Party SoftwareThe third-party software packages should be built and installed as specified by each package. Owl-specific notes for each will be given below. While many of these packages are available as pre-built distributions, configuring and building them yourself will provide for additional control that may be useful. This includes the installation location and the user and group that will be used for executing the commands.Nagios, along with a web server, should be installed such that they will be restarted at system boot time. This table shows the most recent software versions that were tested with the Owl Monitor.
2.1.2.1. Nagios Core and PluginsIf building Nagios Core and Nagios Plugins from source, it is recommended that the configure command be given arguments that will allow specification of the installation-path prefix, the Nagios user, and the Nagios group. A possible command line could be:$ ./configure --with-command-group --prefix=/owl --with-nagios-user=owl-monitor --with-nagios-group=owl-monitor The same prefix, username, and groupname must be used when building Nagios Core and Nagios Plugins. Multiple instances of Nagios can be installed on a single machine. This can be useful for segregating different things being monitored. However, you must be careful when doing this. In particular, even if the --prefix option was given to configure, the "make install-init" and "make install-webconf" commands will overwrite existing Nagios files. 2.1.2.2. nagiosgraphThe README file for nagiosgraph contains instructions for automated installation (using install.pl) and for manual installation. You should read this file and decide which method you want to use.If you choose the manual installation, you may skip most of this section and pick up at the discussion of the map file. If you choose to install with install.pl, you must know that it looks like several of the steps are not performed by the script. After running install.pl you must verify that the manual steps from "Restart nagios" to the end have actually been taken. install.pl will configure the package, install its files in the appropriate places, and modify Nagios and httpd files as needed. A variety of options allow for site-specific installation layouts. In addition, you will be prompted for many directory and file names prior to install.pl completing its work. If it can't find one or another file that it must update, then at the end of execution it will provide a set of steps that you must manually perform in order to complete the installation. The --dry-run option will show you what install.pl would do, so that you may verify where things are being copied. It isn't required, but it wouldn't be a bad idea to use this option prior to the actual install. One thing install.pl is likely to say is that you must add a Nagios command file. This file probably already exists, and you must find it. It is likely to be called something like commands.cfg and be located in the etc/objects directory in your Nagios installation. Check names carefully when making this modification, as commands.cfg already has a process-service-perfdata command and install.pl will have you add a process-service-perfdata-by-nagiosgraph command. install.pl logs all its actions, including the manual-intervention actions, in the install-log file. Unfortunately, the command line is not recorded in the log file, so it might be useful to save that somewhere yourself. I have used this command to successfully install nagiosgraph: $ install.pl --prefix /owl/nagiosgraph --nagios-user owl-manager --nagios-cgi-url /nagios-owl/cgi-bin \ --nagios-perfdata-file /owl/var/service-perfdata.out --log-dir /owl/var --layout standaloneI do not let install.pl update the Nagios or Apache configurations, which it will prompt for; I prefer to make those modifications myself. Refer to the install-log file to see what remains to be done. The README file contains steps for manual installation and it looks like several of the steps are not performed by install.pl. If you use install.pl to install nagiosgraph, verify that the manual steps from "Restart nagios" to the end have actually been taken. You may have to manually copy the nagiosgraph.ssi file into place. This file is in .../nagiosgraph/examples/nagiosgraph.ssi and it should be copied into your equivalent of /owl/share/nagiosgraph.ssi. Before nagiosgraph can be used by the Owl manager, the map file must be modified to recognize data from the Owl plugins for Nagios. The required modifications are discussed in section 2.2.7. 2.1.2.3. rrdtoolNo Owl-specific configuration or actions were required for building and installing rrdtool.2.1.2.4. drraw.cgiTo install drraw.cgi, copy the drraw.cgi and the drraw.conf files into the Nagios sbin/ directory. So, if you have installed Nagios in /owl/nagios, then you should copy the two drraw files into /owl/nagios/sbin.The drraw.conf must be edited for this installation. Some of these edits are required and some are optional. Each will be marked as to whether or not you must make the change. This file is a Perl file, so the configuration values are set according to Perl's language rules.
2.1.2.5. Perl modulesThere is a set of Perl modules required by the Owl manager and the third-party software packages. Some of the software packages may install these packages themselves, but you must ensure that these are available. These should be available through CPAN.Modules required by the Owl manager:
It is likely that some of the third-party software may require other Perl modules. The installation documentation or process should inform you of what must be installed on your system. 2.1.3. Installing the Owl Monitor SoftwareThere is a small amount of manager-specific software in the Owl Monitor. Most of the work on the manager is performed by Nagios and the other third-party software. The Owl Monitor software on the manager host consists of two Nagios plugins that will interpret Owl sensor data for Nagios and add Owl sensor data to graphing databases. There is also a small number of administrator programs for managing the large amounts of Owl sensor data.2.1.3.1. Unpacking the Owl Monitor SoftwareThe Owl Monitor software comes in a bzip2'd tar file. In most modern Unix systems, this can be unpacked with this command:$ tar xzf owl-monitor.tar.gzThis command unpacks the Owl distribution in the current directory. This can be unpacked anywhere you want, but it is a good idea to do it in the Owl manager's home directory. Most of the Owl commands will be in the owl-manager/bin directory. These commands can stay in that directory or be moved elsewhere. If they remain in their own directory, it would be helpful to add the directory to your shell's path. 2.1.3.2. owl-dnswatchThe owl-dnswatch command is a Nagios plugin that reads Owl monitoring data and reports the latest results to Nagios. Each execution of owl-dnswatch requires an Owl sensor name, a target host, a DNS server, and a DNS query type. The appropriate monitor datafile is consulted based on those inputs. The following actions must be taken to install owl-dnswatch.
2.1.3.3. owl-perfdataThe owl-perfdata command is a Nagios plugin that acts as a front-end for inserting Owl monitoring data into an RRD database. These databases are used for graphing the monitoring data. The following actions must be taken to install owl-perfdata.
2.1.3.4. owl-newsensorThere are a set of Nagios objects that must be created for each Owl sensor. Once the Owl manager has sensor data files in its data directory, the owl-newsensor program can be used to create these objects. The actions required for this are described in section 4.7.No local modifications are required for owl-newsensor. Copy owl-newsensor to a directory which will be its home. This could be anything that is convenient for you; e.g., /usr/bin, /usr/local/bin, ~/mystuff, or /owl/scripts. 2.1.3.5. owl-transfer-mgrIf the Owl manager transfers data from a sensor, it will use the owl-transfer-mgr to retrieve the data.No local modifications are required for owl-newsensor. 2.1.3.6. Data-Archiving ProgramsThe following programs assist the administrator with managing the large amount of Owl monitoring data that accumulates on the Owl manager host. These programs work together to create a two-stage archival process. It is helpful to have cron run these programs, though this is not essential.The first stage of the archive process moves old data from the sensor's data directory to the sensor's archive directory. Data is considered old if the file's timestamp (in its name) is older than two days. This part of the data archiving should be performed once per day, depending on how frequently the sensors perform their own data archiving. The second stage of the archive process combines a month's worth of sensor data into a single compressed tarfile. This should be performed several days after the end of the previous month. The Owl data-archival programs assume that all sensor data are stored in a common directory and all the archived data are stored in a common directory. They will segregate data by sensor in order to prevent it from getting mixed together. For example, when handling data for the sensors dresden and kvothe, these programs might expect the following directories to exist:
/owl/data/dresden /owl/data/kvothe /owl/data-archive/dresden /owl/data-archive/kvothe This first set of programs will archive data for all the sensors that report to an Owl manager. They must be told where the sensor data are stored and where the data archive is kept.
This second set of programs will archive data for a single sensor that reports to an Owl manager. They must be told where the sensor data are stored and where the data archive is kept.
More information about each of these programs is available in their man pages. Copy these archiving programs to a directory which will be their home. This could be anything that is convenient for you; e.g., /usr/bin, /usr/local/bin, ~/mystuff, or /owl/scripts. 2.2. Modifying Configuration FilesSeveral files must be modified in order to provide Nagios monitoring and graphing. These files belong to Nagios, Apache, and nagiosgraph. This section describes the required modifications. These modifications do not provide any monitoring of Owl sensor data; further steps to supply that functionality will be given later.In the basic Nagios installation, /nagios/etc holds sample configuration files, which can be modified for use. 2.2.1. Save Files!Before starting to modify any existing file -- e.g., Nagios object files and Apache configuration files -- it would be a very good idea to make a back-up copy of the files. This will provide reference copies of the files in case they might be needed later.2.2.2. nagios.cfg (Nagios)This file is the primary configuration file for Nagios. There are quite a few changes that must be made to this file. The modifications listed in this section will not incorporate monitoring of Owl sensor data. That will be described in section 4.8.1.The various configuration parameters for Nagios could be included in a small number of files, but the standard arrangement groups the parameters into separate files. The nagios.cfg file is the primary configuration file for Nagios, and at run-time it acts as the focal point to collect the disparate configuration files for the Nagios daemon. A cfg_file line is used to tell Nagios to include the specified line. Adding or deleting these lines will control which files are included in the Nagios configuration. As a result of the Nagios build and installation process, the nagios.cfg file should be mostly configured already. The Nagios configuration file is well-documented, so figuring out how to adjust its fields shouldn't be too difficult. There are a few things that must be done to prepare it for Owl monitoring. These directions assume that Owl monitoring is the only monitoring that will be performed by this installation.
2.2.3. cgi.cfg (Nagios)This is the configuration file for CGI scripts run by Nagios. Most of it is set up during the Nagios configuration process. The following things must be set to ensure your installation's needs are met.
2.2.4. resource.cfg (Nagios)Ensure USER1 points to the plugins directory. This should be something like /owl/libexec.2.2.5. share/config.inc.php (Nagios)Nagios uses this file to hold a small amount of file-location data. The following actions must be taken:
2.2.6. nagiosgraph.conf (nagiosgraph)This is the configuration file for Nagiosgraph. Several settings must be checked and some Owl-specific setting copied in. The following actions must be taken:
2.2.7. map (nagiosgraph)nagiosgraph uses a file of data definitions that describe how to extract data from Nagios plugins and how to handle the extracted data. The file is a collection of Perl expressions that are evaluated by nagiosgraph. This file is named something like /owl/nagiosgraph/etc/map. The Owl manager distribution contains a set of additional, Owl-specific definitions that must be added to the map file. These definitions are in ng-owl-map. Copy the contents of this file into the map. 2.2.8. htpasswd.users (Apache)Strictly speaking, this file is not a Nagios file, but is instead an httpd file. The Nagios configuration file for httpd gives the name of an htpasswd.users file. Use the htpasswd command to create and populate this file with the users that should be allowed to access the Owl monitor. These users must have their permissions defined in the Nagios cgi.cfg file. 3. An Interlude on Sensor QueriesBefore running your Owl Monitoring System, you must consider a number of questions. How many Owl sensors will be providing data to the Owl manager? How many queries will the sensors be performing? What queries will the sensors be performing? Will each sensor be performing the same queries? Will the sensor be providing "heartbeat" information to the manager? Who will transfer the sensor data to the manager? These questions are important for the sensor administrator, but they are critical for the manager administrator. 3.1. Gathering and Storing Sensor DataA query consists of three parts: a nameserver, a target host, and a query type. Responses to each query will be kept in their own datafile. The amount of data generated by each sensor will depend on the number of queries and the frequency with which the queries are performed.The sensors will have to gather the data and more queries will result in more data that will be collected. The data will all have to be transferred from the sensor to the manager. If the manager is configured for graphing, then data from each query will be stored in its own database. These databases can be quite large, so the manager host must have a good amount of available disk space. The administrators of an Owl sensor and the Owl manager (if they are different people) must coordinate the queries that the sensor will be performing. The manager's administrator can request that a set of queries be performed, but it is up to the sensor's administrator to actually configure the sensor for the queries. The manager is effectively at the mercy of the sensor. Sensor data filenames contain metadata about the sensor data contained therein. Therefore, the filenames will reflect the queries that generated the files. The fields are separated by commas and have this format: 121129.1701,seattle-sensor,example.com,a.root-servers.net,A.dns 121129.1701,seattle-sensor,example.com,m.root-servers.net,A.dns 121129.1701,seattle-sensor,example.com,m.root-servers.net,NS.dns 121129.1701,portland-sensor,example.com,a.root-servers.net,A.dns Breaking out the fields of the last example above gives this table:
3.2. Transferring Sensor DataThe Owl Monitoring System provides utilities to transfer sensor data from the sensor to the manager. One utility, owl-transfer runs on the sensor and transfers the data to the manager. The other utility, owl-transfer-mgr runs on the manager and transfers the data from the sensor. Both utilities are front-ends for the rsync program and run using an ssh-initiated connection. In both cases, the transferring host must be able to ssh into the remote host without a password. The FOSS rrsync ("restricted rsync", written by Joe Smith, modified by Wayne Davison) is used to restrict the transferring host's access to the remote host. The administrators of the sensor and the manager must agree on which host will be transferring data to the manager. The appropriate SSH public keys from the transferring host must be placed on the other host, so that the password-less ssh (as described above) may be performed. An Owl system with multiple sensors may have a mixed transfer configuration. In such a system, some sensors will transfer their data to the manager, while the manager will transfer data from other sensors. 3.3. Sensor HeartbeatsThe "heartbeat" facility allows an Owl sensor to periodically contact a manager to let it know that the sensor is still alive and network accessible. The heartbeat is sent by contacting a particular webpage on the Owl manager.The heartbeat facility is not necessary for Owl operation and Owl runs perfectly well without it. However, it can provide a useful assurance to the manager administrator that the sensor is still working. Enabling this facility on the sensor is at the discretion of the sensor administrator. Since this requires a webserver to be running on the manager host, that aspect is at the discretion of the manager administrator. 4. Adding Owl Sensors to an Existing Owl InstallationWhenever a new Owl sensor is added to those handled by an Owl manager, there are a number of actions that must take place on both the sensor and the manager. These actions should be followed consecutively within each section. However, there is some amount of necessary interleaving of actions. For example, a particular sensor action may be required before a particular manager action. It is acceptable for the Owl sensor and Owl manager to be under completely different administrative control. Each host may even be owned by different commercial, educational, governmental, or other such entity. All that is required is that there be some initial coordination between administrators when the sensor is first configured, along with (probably) aperiodic support from time to time later. The discussions on adding Owl sensors assumes that the sensor and manager are under different administrative control. Required actions will not change if they are under unified administrative control, but they may be easier to accomplish. This section describes the various actions that must be performed on the Owl manager in order to configure a new sensor for it. This assumes the installation and configuration procedures detailed in section 2 have been completed. 4.1. Create Directories and Files for Sensor's DataA set of directories and files must be created for a new sensor. These are the sensor directory, the sensor's data directory, the history directory, the data archive directory. The heartbeat file, if the sensor will be providing heartbeats, must also be created. These files are best created by the owl-initsensor command, but they may also be created manually. If they are created manually, you must use the names and organization as given in the examples below. The locations of the directories may change (i.e., the /owl/data and /owl/archive portions), but that's it. For example, when adding the new sensors helsinki and canberra, you might create the following sets of directories:
The archive directory must be writable by the manager's Owl user, but the others must be writable by the manager's Owl user and the manager's web server. 4.2. Firewall ConfigurationThe Owl manager and the Owl sensor communicate by way of SSH and HTTP. If the Owl manager is protected by a firewall (either on the manager host itself or by an enterprise-level firewall) then the firewall must be configured to allow data to be transferred between the manager and the sensor. The direction of this transfer (initiated by sensor or initiated by manager) depends on the Owl configuration. These firewall modifications are far beyond the scope of this document. 4.3. SSH Set-upFirst, you must generate a new SSH key for the manager's Owl user. If the manager will be pulling data from the sensor, then the you must provide the sensor administrator with your Owl user's public key. This key will allow owl-transfer-mgr to retrieve Owl sensor data. You must generate your key in compliance with particular key requirements (e.g., length and type) specified by the sensor administrator. If the sensor will be transferring data to the manager, then the sensor administrator must provide you with the public key of their Owl sensor user. This key will allow owl-transfer to pass Owl sensor data to the your manager. You must add this key to your .ssh/authorized_keys file. The key must be generated with key characteristics (e.g., length and type) that are acceptable to you. See the example below. The authorized_keys file restricts access to known sensor hosts and controls where each Owl sensor's data are stored. This file should only have one entry per sensor, and each sensor should have a unique data directory. Example entries (with abbreviated keys): command="/usr/local/bin/rrsync /owl/data/sensor1" ssh-rsa AAAA...Qw== sensor@sensor1.example.com command="/usr/local/bin/rrsync /owl/data/sensor8" ssh-rsa AAAA...gM== sensor@sensor8.example.auThe parts of the line from "ssh-rsa " to the end of the line are the public SSH key provided by the sensor's administrator. You will add everything before the "ssh-rsa ", using the proper paths for rrsync and the sensor's data directory. 4.4. Configuration SettingsWith Owl's capability of having either the sensor or the manager transfer sensor data to the manager, there are configuration settings that must be made to allow the transfer. These are described in the subsections below. Regardless of how your Owl installation handles data transfer, you should read both subsections to ensure you are making all the required settings. It would be a good idea for you and the sensor administrator to agree on a name for the sensor. This isn't required, but it will probably make things easier for you both in the long run to refer to the sensor by the same name. The sensor name can be very generic, such as sensor42, sensor-d, or owl-us-east. It can also be very specific, such as washdc-1600-penn-ave-nw or cheltenham-bldg4. The intent is to provide distinguishing information to the intended audience of the DNS monitoring data. You should use names that easily distinguish sensors and are acceptable to the manager and sensor administrators. 4.4.1. Provide Configuration Information to Sensor AdministratorIf the Owl sensor will be transferring data to the manager, then you must provide the sensor administrator with an SSH user. All Owl sensors may use this single SSH user, and the data will be distinguished by the SSH key used to connect from the sensor. The sensor will use the heartbeat URL to provide a heartbeat to the manager. If this is to be used, it must be set on the sensor regardless of who initiates sensor data transfer. The data values will be added to the sensor's configuration file. The following example data will allow the sensor to work with the Owl manager as expected. These data are used, in conjunction with the remote keyword, in the sensor's configuration file.
4.4.2. Receive Configuration Information from Sensor AdministratorIf the Owl manager will be pulling data from the sensor, then the sensor administrator must provide you with an SSH user. The owl-transfer-mgr program will use this SSH user to copy data from the sensor. The following data will allow the manager to connect to the Owl sensor as expected. These data are used, in conjunction with the remote keyword, in the sensor's configuration file.
4.5. Test Transfer from SensorAt this point, you are ready to test your Owl manager. When the sensor administrator reaches section 4.5 in their installation instructions, both sensor and manager are ready to test the data transfer. Coordinate with the sensor administrator to test data transfer. The sensor administrator must put a data file (of any sort, it doesn't have to be an Owl data file) in their data directory. If the manager will be transferring data, you must run owl-transfer-mgr to attempt to retrieve the test file. If the sensor will be transferring data, the sensor administrator must run owl-transfer to attempt to retrieve the test file. After the transfer command appears to have transferred the file without error, you must check that sensor's data directory on the manager to ensure the file has arrived as expected. Once the test file successfully appears in the new sensor's data directory, the sensor is ready to start collecting data and transferring it to the manager. You must inform the sensor administrator of this, so they can start the Owl sensor daemons. 4.6. Wait for Sensor DataThe sensor should now be in the process of collecting DNS response data. Whichever host will be transferring data must have the appropriate transfer daemon executing. You must now wait for sensor data to show up in the sensor's data directory. The time this takes will depend on how frequently the data is transferred to the manager. This is set in the configuration file. You will know when the sensor is transferring data because the new sensor's data directory will start holding files whose names reflect the queries you are expecting. You may proceed when you have found files for all the queries the sensor will be performing. 4.7. Build Nagios Sensor ObjectsOnce all the sensor's data directory contains files for all the queries expected from the new sensor, Nagios objects must be built for those queries. This may be done automatically using the owl-newsensor command. A file containing these Nagios objects must be created, and it will be added to the Nagios environment in section 4.8.1. owl-newsensor can be used like this to create the Nagios objects: $ owl-newsensor -out sensor8.cfg /owl/data/sensor8/data Passing owl-newsensor the -heartbeat option will cause an object to be created that will allow Nagios to display heartbeat information about the new sensor. 4.8. Nagios ModificationsSeveral Nagios configuration files must be modified to account for the new sensor. These changes will allow Nagios to start reporting current status of the new sensor as well as saving historical data to be used in graphing. 4.8.1. nagios.cfgThe following modifications must be made to the nagios.cfg file to prepare Nagios for monitoring Owl sensor data.
Examples for the cfg_file modifications may be found in nagios.cfg-owl.mods in the Owl manager distribution. 4.8.2. owl-hostgroups.cfgThis file contains the "DNS Response Time Sensors" Nagios hostgroup object that lists the Owl sensor hosts. All your Owl sensors should be listed in the "DNS Response Time Sensors" object.
You may add your own hostgroup objects to this file. Use the "DNS Response Time Sensors" object as a guide to create custom groups. This can be used, for example, to group all the sensors in a geographical region or those that are running a particular operating system. The hostgroup object allows you to group a set of hosts in a manner that makes sense for your purposes. 4.9. Graphing ModificationsThe new sensor's data must be made available to the drraw.cgi script. To do this, the new sensor's data sources must be added to the %datadirs hash in the drraw.conf file. See 2.1.2.4. drraw.cgi for more details. 4.10. Restart NagiosNagios must be restarted in order for it to see your new sensor's objects. Prior to the restart, verify that your object modifications won't cause a problem for Nagios. Execute this command (assuming you are in the directory containing the Nagios files): $ nagios -v nagios.cfgIt will read the configuration files and ensure they all look okay. If there are problems, they must be resolved before Nagios is started. Once the Nagios configuration file is validated and without problems, Nagios may be restarted: $ nagios stop $ nagios start It will probably take several minutes for Nagios to check all its services. Clicking on the "Services" link in the left-hand sidebar will bring up the configured services and you can watch the status of your new sensor. 5. Defining GraphsAfter Owl sensor data has been gathered for several hours and rrdtool has been adding it to the RRD databases, you can consider how you will graph the data. Graphing capabilities are provided by nagiosgraph and drraw.cgi. Both are very useful and have their place in your Owl monitor. It isn't a question of which to use; the question is how you want to use each and how you want to display your monitoring data. 5.1. Graphing with nagiosgraphnagiosgraph provides menus for easy selection of data, available through three CGI scripts. These are simple graphs that only display a single sensor/root server/DNS record per graph. For these scripts, the host is an Owl sensor and the service is a root server/DNS record type. Links for these scripts are included in the example-side.php file.
5.2. Graphing with drraw.cgidrraw.cgi provides a much more complex graphing capability than is available through nagiosgraph. It allows data from multiple sensor/root server/DNS record groups to be displayed on a single graph. Set-up for the graphs is not difficult and is done through a browser-based interface. However, there are a number of things that must be specified before the graph can be displayed. The graph-definition webpage talks about selecting data sources to be displayed. This is referring to the group of RRD databases you want drraw.cgi to use for this particular graph. Defining simple graphs, even those with multiple data sources, is not complicated. However, you can define fairly involved graphs (e.g., averaging data from different databases) with drraw.cgi. 5.3. Adding Graphs to the Nagios SidebarThe share/side.php script builds the left sidebar of links on the Nagios page. Several simple modifications will improve access to the graphing facilities. Obviously, this step must wait until after the system has been set up and site-specific graphs have been defined.The example-side.php file (in the Owl manager distribution) contains sample lines for both drraw.cgi and nagiosgraph. Add the contents of this file to the share/side.php file after the "Quick Search" text-entry box. The sample lines must be modified for your installation in the following ways:
6. Changing Queries on Existing Owl SensorsFrom time to time, you might decide to change the queries being performed by the Owl sensors. This could be adding new queries to some or all of the sensors, stopping some of the queries from being performed, or modifying existing queries. Regardless of the changes, there are actions that must be taken on the Owl manager and all affected Owl sensors. 6.1. Adding New QueriesAdding new queries to the Owl manager is a matter of informing Nagios that it should report on new data. This is accomplished by adding new Nagios objects for the sensor. The following actions must be performed:
6.2. Deleting Old QueriesDeleting existing queries to the Owl manager is a matter of informing Nagios that it should stop reporting on existing data. This is accomplished by deleting Nagios objects for the sensor. The following actions must be performed:
7. Owl Manager CommandsThe Owl manager has a number of Owl-specific commands required for its operation. This section provides a brief overview of the commands available to it. These are all Perl scripts and have self-contained man pages, which contain greater details about their use. These scripts are intended to run as an unprivileged user. This section briefly describes the software used on the Owl manager.
Sponsors:This work was implemented by the following organizations:
This work is funded in part by the following organizations: U.S. Department of Homeland Security/Science & Technology (S&T)
The Owl logo is copyright © 2012 by Wayne Morrison (Wayne@WayneMorrison.com).
All Rights Reserved.
|