Separate collectd log

Issue

Collectd in DEBUG mode can be very chatty. If you have many nodes, then the logs will rotate very quickly and you can’t find anything. Also grep(ing) is often not possible, because of the numerous amount of log entries which will be written in DEBUG mode.

Solution

This is how to adapt the log4j2.xml file to generate a new log file that will only have data for that node label you are troubleshooting.

What needs to be accomplished will be to add a new appender and logger to the log4j2 configuration.

  • the new appender will set the log file name and will filter the data to only output the messages for the node label we are looking for.
  • the new logger will allow us to configure the logging level of the data going to the new appender.

How to

Copy the default log4j2.xml

cp /opt/opennms/etc/log4j2.xml{,.orig}

Add the new logger’

In the ‘’‘loggers’’’ section of the /opt/opennms/etc/log4j2.xml file, add the below to create a new logger. This will send DEBUG level messages to an appender named Collectd-single-node. This does not effect the normal collectd logging level!

<logger name="org.opennms.netmgt.collectd" additivity="false" level="DEBUG">
<appender-ref ref="Collectd-single-node"/>
</logger>

Add a new appender

This will be placed in the appender section of the file after the closing Routing.

(ie between </Routing> and </appenders> lines of the log4j2.xml file )

In the KeyValuePair filter line, put in the specific node label of the node you are looking for. In this example, the node label of the server I want to log is pennms-server.

You can also change the name of the log files if warranted. I kept the file names the same as the server I was logging. Just remember to change the fileName and the filePattern lines for easier logging.

The ACCEPT part will log any data that matches the Context filter and the DENY will discard any messages that do not.

You can always change the output pattern to what you need. I have it also outputting the node label for validation.

<!-- ADDED BELOW -->
<RollingFile name="Collectd-single-node" fileName="${logdir}/opennms-server-collectd.log"
filePattern="${logdir}/opennms-server-collectd.%i.log.gz">
 <ContextMapFilter onMatch="ACCEPT" onMismatch="DENY">
  <KeyValuePair key="nodeLabel" value="opennms-server"/>
 </ContextMapFilter>
 <PatternLayout>
  <pattern>%d %-5p [%t] NODE:%X{nodeLabel} %c{1.}: %m%n</pattern>
 </PatternLayout>
 <!-- Rotate logs at 100MB-->
 <SizeBasedTriggeringPolicy size="100MB" />
 <!-- Rotate through 4 logs -->
 <DefaultRolloverStrategy max="4" fileIndex="min" />
</RollingFile>
<!-- ADDED ABOVE-->

Restart OpenNMS or wait till OpenNMS reads the new configuration changes (2-3 minutes)

A new file named opennms-server-collectd.log will be created in /opt/opennms/logs and any collectd messages that have the Context id of nodeLabel with the name opennms-server will be put in that log. The normal collected log will keep logging its currently configured logging level and not be affected.


Note

This configuration uses the Context map to find the node label because the node label is NOT in the raw message. To find all possible Context’s that can be used in the particular log, change the default output pattern by adding %X to the output pattern.

FROM


<pattern>%d %-5p [%t] %c{1.}: %m%n</pattern>

TO

<pattern>%d %-5p [%t] %c{1.}: %X: NODE:%X{nodeLabel}: %m%n</pattern>

An example output is below:

2018-05-30 06:30:02,726 INFO [Collectd-Thread-4-of-50] {foreignId=1526522493594, foreignSource=OpenNMS, ipAddress=192.168.205.185, nodeId=2, nodeLabel=test logger, prefix=collectd, service=SNMP, sysObjectId=.1.3.6.1.4.1.8072.3.2.10}: NODE:test logger updateRRD: updating RRD file /opt/opennms/share/rrd/snmp/fs/OpenNMS/1526522493594/eth0-525400cae48b/mib2-X-interfaces.rrd with values '1527676203:234510354:3118946'

From the output of this collectd information, the possible contexts that can be used are:

  • foreignId
  • foreignSource
  • ipAddress
  • nodeid
  • nodeLabel
  • prefix
  • service
  • sysObjectid
2 Likes