H29+ won't start with permission error to open ICMP socket

Problem

We have noticed some people ran into issues with ICMP and problems during startup when they have tried to upgrade to Horizon 29+. The biggest change in H29 is running now as a non-privileged user and it requires having a Linux Kernel > 3.10 which introduced a way to give users a way to use datagram socket for ICMP echo/reply datagrams. During the upgrade we install a sysctl file that controls this behavior:

cat /etc/sysctl.d/99-opennms-non-root-icmp.conf
net.ipv4.ping_group_range=1 995

If you run on very old Kernels this option is not available to you.

Symptoms

When you try to start OpenNMS you get error messages

2021-12-20 11:37:25,554 ERROR [Main] o.o.n.i.j.Jni6Pinger: Permission error received while attempting to open ICMP socket. See https://wiki.opennms.org/wiki/ICMP for information on configuring ICMP for non-root.

You might also see in journalctl -u opennms an error message like this:

Dec 20 11:16:52 localhost.localdomain systemd[1]: Started OpenNMS server.
Dec 20 11:18:05 localhost.localdomain systemd[1]: opennms.service: Supervising process 81854 which is not our child. We'll most likely not notice when it exits.

Solution

The opennms user requires permission to use a datagram socket to be able to send ICMP (ping) messages. By default, a user doesn’t have these permissions. During the installation set sysctl net.ipv4.ping_group_range and give the opennms group the permissions accordingly. If you don’t have a modern Linux Kernel you probably don’t have this option and you need to add the permissions on a process level. You need to add network capabilities to the use of raw datagram sockets. The easiest way to assign them is by modifying the systemd unit of OpenNMS to assign the capabilities accordingly.

Edit the systemd unit /etc/systemd/system/multi-user.target.wants/opennms.service by adding the lines in the [service] section as the following:

AmbientCapabilities=CAP_NET_RAW

Reload the sytemd unit with systemctl daemon-reload and restart the service with systemctl restart opennms.

Kudos to jesk and @UberPinguin for help digging through these issues.

Considerations with SELinux

If this doesn’t solve the problem verify if you have SELinux running and enabled. You can verify this with the sestatus command and gives you information if it’s enabled and the working mode like here:

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

With audit2allow -a you can verify if opennms violates SELinux policies. In case the command is missing, run dnf install policycoreutils-devel package.

Look for entries preventing to bind ICMP sockets like this:

#============= unconfined_service_t ==============
allow unconfined_service_t node_t:icmp_socket node_bind;

============= unconfined_service_t ==============
allow unconfined_service_t port_t:icmp_socket name_bind;

Create a type enforcement file like this: JniPing.te

module JniPing 1.0;

require {
        type unconfined_service_t;
        type node_t;
        type port_t;
        class icmp_socket { name_bind node_bind };
}

#============= unconfined_service_t ==============
allow unconfined_service_t node_t:icmp_socket node_bind;
allow unconfined_service_t port_t:icmp_socket name_bind;

Create an SElinux module and compile it with:

checkmodule -M -m -o JniPing.mod JniPing.te
semodule_package -o JniPing.pp -m JniPing.mod

Install the SELinux package and enforce it with

semodule -i JniPing.pp
getenforce

You can verify the package with semodule --list-modules | grep JniPing. The permission denied message should be fixed now.

Kudos to @va13 for the help figuring this out. :metal:


:woman_facepalming: You can fix me, I’m a wiki post.

We have added an enhancement to cover this topic also in our installation documentation: [NMS-13866] Add additional steps running as non-root on old Kernels, e.g. RHEL7 - The OpenNMS Issue Tracker

Workaround not working.

Adding these two lines to OpenNMS Service: /etc/systemd/system/multi-user.target.wants/opennms.service
on /etc/systemd/system/multi-user.target.wants/opennms.service

CapabilityBoundingSet=CAP_NET_RAW
AmbientCapabilities=CAP_NET_RAW

Could not connect to the OpenNMS JVM (OpenNMS might not be running or could be starting up or shutting down): Unable to retrieve status from running services.
opennms is stopped
Can’t open PID file /opt/opennms/logs/opennms.pid (yet?) after start: No such file or directory

Service file details:

[Unit]
Description=OpenNMS server
Wants=postgresql.service
Requires=network.target network-online.target
After=postgresql.service postgresql-10.service postgresql-11.service postgresql-12.service postgresql-13.service network.target network-online.target

[Service]
User=opennms
Environment="OPENNMS_HOME=/opt/opennms"
CapabilityBoundingSet=CAP_NET_RAW
AmbientCapabilities=CAP_NET_RAW

Type=forking
RuntimeDirectory=opennms
PIDFile=/opt/opennms/logs/opennms.pid

ExecStart=/etc/init.d/opennms -s start
ExecStop=/etc/init.d/opennms stop

[Install]
WantedBy=multi-user.target

For SELinux (RHEL8, AlmaLinux, …) additional steps are necessary:

# dnf install policycoreutils-devel
# audit2allow -a
============= unconfined_service_t ==============
allow unconfined_service_t port_t:icmp_socket name_bind;
# audit2allow -a -M JniPing
******************** IMPORTANT ***********************

To make this policy package active, execute:

semodule -i JniPing.pp
# ls -l
-rw-r--r--. 1 root root    968 30. Mär 17:04 JniPing.pp
-rw-r--r--. 1 root root    215 30. Mär 17:04 JniPing.te
# semodule -i JniPing.pp
# getenforce
Enforcing

See also Chapter 8:

1 Like

Your problems don’t seem to be related to the permission error described in this knowledge base article. Please open a community support issue in 🏅 Community Support - OpenNMS Community and don’t use this knowledgebase article for discussions.