How to analyse a Java thread dump


All OpenNMS components run in a Java Virtual Machine. If you run OpenNMS in very large or complex environments, it could be necessary to leave the default settings and you need to tweak the Java and OpenNMS settings. A first common indication is Pollerd or Collectd don’t have enough threads to get through the number of services and metrics to collect. Another indication can be you have integrations which consume resources during Event processing or forwarding Alarms into other applications.

This Article describes how to analyse Java Thread dumps to investigate how different components in OpenNMS interact. Be aware the size of these dumps depend on the size of your OpenNMS server and can be very large.


Step 1: Generate the thread dump:

Get the Java process ID (PID) from the running OpenNMS application.

Run systemctl status opennms and note the main PID or look into ${OPENNMS_HOME}/logs/ file. Generate a thread dump using jstack -l ${pid} > /tmp/jstack.out. If you run on a Windows system you can use the Java provided command jps -v.

If you run our Docker container images you can get a thread dump also by sending SIGQUIT(3) to the Java process. You can send a signal to the Docker container with

docker-compose kill -s 3 <service-name>

or using just the Docker command

docker kill -s QUIT <container-id>

The thread dump will be generated on the standard out of the Java process which runs in foreground. You can capture the output with docker-compose logs or docker logs.

Step 2: Analyze the thread dump

Upload the thread dump to an analyser. You can use for example or if you want to analyse it locally with a community hosted Thread Dump Analyzer.

Step 3: Explaining outputs

Possible outputs for the thread status can be:

  • NEW: The thread is created but has not been processed yet.
  • RUNNABLE: The thread is occupying the CPU and processing a task.
  • BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock.
  • WAITING: The thread is waiting by using a wait, join or park method.
  • TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method.
Example of locked threads below 392 threads found 91 threads with this stack:
"Collectd-Thread-1-of-50": awaiting notification on [0x00000005c6f0d820]
"Collectd-Thread-10-of-50": awaiting notification on [0x00000005c6f0d820]
"Collectd-Thread-11-of-50": awaiting notification on [0x00000005c6f0d820]
"Collectd-Thread-12-of-50": awaiting notification on [0x00000005c6f0d820]
"Collectd-Thread-13-of-50": awaiting notification on [0x00000005c6f0d820]
"Collectd-Thread-14-of-50": awaiting notification on [0x00000005c6f0d820]
"Collectd-Thread-15-of-50": awaiting notification on [0x00000005c6f0d820]

Notice that these 91 threads are locked

If you click that process ([0x00000005c6f0d820]) it takes you to this:

AbstractQueuedSynchronizer$ConditionObject  49 threads waiting for notification on lock:

so 49 of 50 Collectd threads are locked. If we go to the Pollerd threads

30 threads waiting for notification on lock:

All of the Poller threads are locked (30 of 30)