Upgrading Horizon from 28 to 29

[tobbe]

Problem:
Trying to upgrade from 28 to 29.
Opennms is run as the opennms user.
When starting it up, the following is displayed in the log file:

2022-04-04 12:22:41,801 DEBUG [Main] o.o.n.i.j.JnaIcmpMessenger: Unable to initialize IPv6 Pinger.
java.lang.reflect.InvocationTargetException: null
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.opennms.jicmp.jna.NativeDatagramSocket.create(NativeDatagramSocket.java:94) ~[org.opennms.core.icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.V6Pinger.<init>(V6Pinger.java:57) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.JnaIcmpMessenger.<init>(JnaIcmpMessenger.java:67) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.JnaPinger.initialize(JnaPinger.java:72) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.JnaPinger.isV4Available(JnaPinger.java:101) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPingerFactory.tryPinger(BestMatchPingerFactory.java:72) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPingerFactory.findPinger(BestMatchPingerFactory.java:137) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPinger.initialize(BestMatchPinger.java:129) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPinger.isV4Available(BestMatchPinger.java:101) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Manager.testPinger(Manager.java:264) ~[org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Manager.lambda$doTestLoadLibraries$6(Manager.java:247) ~[org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71) [org.opennms.core.logging-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Manager.doTestLoadLibraries(Manager.java:246) [org.opennms.core.daemon-29.0.8.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [?:?]
        at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [?:?]
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [?:?]
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) [?:?]
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) [?:?]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) [?:?]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [?:?]
        at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:333) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:258) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:157) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:95) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150) [org.opennms.core.daemon-29.0.8.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:531) [opennms_bootstrap.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: com.sun.jna.LastErrorException: [1] Operation not permitted
        at org.opennms.jicmp.jna.UnixV6NativeSocket.socket(Native Method) ~[org.opennms.core.icmp-jna-29.0.8.jar:?]
        at org.opennms.jicmp.jna.UnixV6NativeSocket.<init>(UnixV6NativeSocket.java:56) ~[org.opennms.core.icmp-jna-29.0.8.jar:?]
        ... 45 more
2022-04-04 12:22:41,815 DEBUG [Main] o.o.n.i.j.JnaPingRequest: 1649067761815: Sending Ping Request: [ID=JnaPingRequestId[addr = /127.0.0.1, ident = 24578, seqNum = 1, tId = 1],Retries=0,Timeout=500,Packet-Size=64,Expiration=-1,Callback=org.opennms.netmgt.icmp.LogPrefixPreservingPingResponseCallback@13d010fe]
2022-04-04 12:22:41,818 DEBUG [Main] o.o.p.r.RequestTracker: Scheding timeout for request to [ID=JnaPingRequestId[addr = /127.0.0.1, ident = 24578, seqNum = 1, tId = 1],Retries=0,Timeout=500,Packet-Size=64,Expiration=1649067762315,Callback=org.opennms.netmgt.icmp.LogPrefixPreservingPingResponseCallback@13d010fe] in 497 ms
2022-04-04 12:22:41,818 INFO  [Main] o.o.n.i.SinglePingResponseCallback: waiting for ping to /127.0.0.1 to finish
2022-04-04 12:22:41,822 INFO  [JNA-ICMP-24578-Callback-Processor] o.o.n.i.SinglePingResponseCallback: got response for address /127.0.0.1, thread 24578, seq 1 with a responseTime 0.686691ms
2022-04-04 12:22:41,822 INFO  [Main] o.o.n.i.SinglePingResponseCallback: finished waiting for ping to /127.0.0.1 to finish
2022-04-04 12:22:41,822 INFO  [Main] o.o.n.i.b.BestMatchPingerFactory: Best available pinger is: class org.opennms.netmgt.icmp.jni.JniPinger
2022-04-04 12:22:41,823 INFO  [Main] o.o.n.v.Manager: Using ICMP implementation: org.opennms.netmgt.icmp.best.BestMatchPinger
2022-04-04 12:22:41,824 INFO  [Main] o.o.n.v.Manager: IPv4 ICMP available? true
2022-04-04 12:22:41,824 INFO  [Main] o.o.n.v.Manager: IPv6 ICMP available? false
2022-04-04 12:22:41,825 INFO  [Main] o.o.n.v.Invoker: Invocation doTestLoadLibraries successful for MBean OpenNMS:Name=TestLoadLibraries
2022-04-04 12:22:41,826 INFO  [Main] o.o.n.v.Invoker: Invoking init on object OpenNMS:Name=Eventd```

Process seems to be stuck:

# opennms -v status
WARNING: relaunching as opennms
OpenNMS.Name=Eventd                 : start_pending
OpenNMS.Name=Alarmd                 : start_pending
OpenNMS.Name=Bsmd                   : start_pending
OpenNMS.Name=Ticketer               : start_pending
OpenNMS.Name=Queued                 : start_pending
OpenNMS.Name=Actiond                : start_pending
OpenNMS.Name=Notifd                 : start_pending
OpenNMS.Name=Scriptd                : start_pending
OpenNMS.Name=Rtcd                   : start_pending
OpenNMS.Name=Pollerd                : start_pending
OpenNMS.Name=EnhancedLinkd          : start_pending
OpenNMS.Name=Collectd               : start_pending
OpenNMS.Name=Discovery              : start_pending
OpenNMS.Name=Vacuumd                : start_pending
OpenNMS.Name=EventTranslator        : start_pending
OpenNMS.Name=PassiveStatusd         : start_pending
OpenNMS.Name=Statsd                 : start_pending
OpenNMS.Name=Provisiond             : start_pending
OpenNMS.Name=Reportd                : start_pending
OpenNMS.Name=Ackd                   : start_pending
OpenNMS.Name=JettyServer            : start_pending
OpenNMS.Name=KarafStartupMonitor    : start_pending
OpenNMS.Name=Telemetryd             : start_pending
OpenNMS.Name=Trapd                  : start_pending
OpenNMS.Name=PerspectivePoller      : start_pending
opennms is partially running

OpenNMS version:
29.0.8

Running as root works fine so seems opennms user lacks some privileges.
Any idea what needs to be done? IPv4 ping seems to be fine and the setting net.ipv4.ping_group_range = 1 993 is there. Is there something similar for IPv6?

[dino2gnt]

I don’t like the ping_group_range solution. I think adding the required capabilities to startup is more elegant:

Allow privileged port bindings when running OpenNMS with a system account

Problem With Horizon 29+ we run no longer as a root user by default. An unprivileged system account opennms is created and used for the process. Running with an unprivileged user increases some security aspects, but the drawback is limitations by spawning processes trying to bind privileged ports < 1024. Especially for monitoring the ports 514/udp for Syslog and 162/udp can be an issue, which is the reason the ports have changed in our default configuration to 10514/udp and 10162/udp. For thes…

Add CAP_NET_RAW to the AmbientCapabilities list described in this post, so you have:

AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW

…and you’re off to the races.

[tobbe]

Thanks a lot, that got me past that problem. Now I’m getting loads of parse errors in the event files instead. I will check those and get back to you in case I can’t figure it out myself.

[tobbe]

It seems to get stuck when starting up Eventd. Any experience with that and what could be the reason? Changing to RUNAS=root makes it work again.

It never gets to these rows when running as the unprivileged user:

2022-04-04 21:37:24,804 INFO  [Main] o.s.c.s.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@6cbf0fc4: startup date [Mon Apr 04 21:37:24 CEST 2022]; parent: ApplicationContext 'daemonContext'
2022-04-04 21:37:24,805 INFO  [Main] o.s.b.f.x.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/opennms/applicationContext-dao.xml]
2022-04-04 21:37:24,819 INFO  [Main] o.s.b.f.x.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/opennms/applicationConte

It just sits there with this log row being the last:

2022-04-04 21:37:23,958 WARN  [Main] o.a.a.j.p.PooledConnectionFactory: Create pooled connection during start failed. This exception will be ignored.
javax.jms.JMSException: Error while attempting to add new Connection to the pool
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createJmsException(PooledConnectionFactory.java:266) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:225) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:204) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.start(PooledConnectionFactory.java:290) [activemq-jms-pool-5.14.5.jar:5.14.5]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1713) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1650) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1579) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:843) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) [?:?]
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1148) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1050) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:140) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1148) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1050) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648) [org.apache.servicemix.bundle...skipping...
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:275) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:95) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:92) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:221) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        ... 94 more
Caused by: java.io.IOException: Broker named 'localhost' does not exist.
        at org.apache.activemq.transport.vm.VMTransportFactory.doCompositeConnect(VMTransportFactory.java:119) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.vm.VMTransportFactory.doConnect(VMTransportFactory.java:56) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.TransportFactory.connect(TransportFactory.java:65) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:331) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:346) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:304) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:244) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:275) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:95) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:92) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:221) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        ... 94 more
Caused by: java.io.IOException: Broker named 'localhost' does not exist.
        at org.apache.activemq.transport.vm.VMTransportFactory.doCompositeConnect(VMTransportFactory.java:119) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.vm.VMTransportFactory.doConnect(VMTransportFactory.java:56) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.TransportFactory.connect(TransportFactory.java:65) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:331) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:346) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:304) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:244) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:275) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:95) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:92) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:221) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        ... 94 more

[dino2gnt]

Have you changed anything with the embedded ActiveMQ broker config? Are you using ActiveMQ in your environment?

Have you run $OPENNMS_HOME/bin/fix-permissions ?

[tobbe]

Yes, we do indeed use ActiveMQ in our environment. When running as the unprivileged user the AMQ port (61616) does not come up.

Yes, I have run $OPENNMS_HOME/bin/fix-permissions.

[tobbe]

I found the problem. In opennms-activemq.xml the data directory was pointing to a directory where the permissions were wrong since it was outside the /opt/opennms hierarchy:

        <property name="properties">
             <value>
                 <!--activemq.data=/usr/share/opennms/data/tmp/activemq -->
                 activemq.data=/opt/opennms/data/tmp/activemq
             </value>
        </property>

Commented out the wrong dir, added the correct dir to the conf, created the dir, ran fix-permissions.sh and restarted opennms.

I also added the following row to /etc/security/limits.d/20-nproc.conf since it was complaining of running out of processes for Provisiond:

opennms soft nproc unlimited

Everything is working now, thanks for the support.

[tobbe]

I spoke too soon. We have a listener specified in opennms.properties that binds to 443 and the opennms user is not allowed to bind to the privileged port number:

org.opennms.netmgt.jetty.https-port = 443

Any idea how to solve this the best way or should we set up a proxy in front, for example nginx instead?

[dino2gnt]

This is what the CAP_NET_BIND_SERVICE capability specifically allows…?

However, for SSL termination in general, i recommend putting Apache or nginx on top of it as a reverse proxy. Configuring SSL in Jetty is too much of a headache.

[tobbe]

That was a very good suggestion. Set up an NGINX proxy in front of OpenNMS and it works well.

I still have a problem with what appears to be OpenNMS fork bombing the system.

# opennms -v status
WARNING: relaunching as opennms
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: Resource temporarily unavailable

Here is the settings I have in the systemd file for opennms:

[Unit]
Description=OpenNMS server
Wants=postgresql.service
Requires=network.target network-online.target
After=postgresql.service postgresql-10.service postgresql-11.service postgresql-12.service postgresql-13.service network.target network-online.target

[Service]
User=opennms
Environment="OPENNMS_HOME=/opt/opennms"
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW
LimitNOFILE=16000
LimitNOFILESoft=16000
LimitNProc=unlimited

Type=forking
PIDFile=/opt/opennms/logs/opennms.pid

ExecStart=/opt/opennms/bin/opennms -s start
ExecStop=/opt/opennms/bin/opennms stop

[Install]
WantedBy=multi-user.target

Any idea where I’ve gone wrong here? When I type lsof | wc -l it just hangs.

[dino2gnt]

LimitNOFILE=16000

Too small. Try adding a zero or two.

[tobbe]

There was an offending line under /etc/security/limits.d/20-nproc.conf that overrode what I had in the systemd file:

* soft nproc 4096

After I commented that out it seems to be working flawlessly.

Many thanks for the support!