Upgrading Horizon from 28 to 29

Problem:
Trying to upgrade from 28 to 29.
Opennms is run as the opennms user.
When starting it up, the following is displayed in the log file:

2022-04-04 12:22:41,801 DEBUG [Main] o.o.n.i.j.JnaIcmpMessenger: Unable to initialize IPv6 Pinger.
java.lang.reflect.InvocationTargetException: null
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.opennms.jicmp.jna.NativeDatagramSocket.create(NativeDatagramSocket.java:94) ~[org.opennms.core.icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.V6Pinger.<init>(V6Pinger.java:57) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.JnaIcmpMessenger.<init>(JnaIcmpMessenger.java:67) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.JnaPinger.initialize(JnaPinger.java:72) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.jna.JnaPinger.isV4Available(JnaPinger.java:101) ~[opennms-icmp-jna-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPingerFactory.tryPinger(BestMatchPingerFactory.java:72) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPingerFactory.findPinger(BestMatchPingerFactory.java:137) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPinger.initialize(BestMatchPinger.java:129) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.icmp.best.BestMatchPinger.isV4Available(BestMatchPinger.java:101) ~[opennms-icmp-best-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Manager.testPinger(Manager.java:264) ~[org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Manager.lambda$doTestLoadLibraries$6(Manager.java:247) ~[org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71) [org.opennms.core.logging-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Manager.doTestLoadLibraries(Manager.java:246) [org.opennms.core.daemon-29.0.8.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [?:?]
        at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [?:?]
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [?:?]
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) [?:?]
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) [?:?]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) [?:?]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [?:?]
        at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:333) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:258) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:157) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:95) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173) [org.opennms.core.daemon-29.0.8.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150) [org.opennms.core.daemon-29.0.8.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:531) [opennms_bootstrap.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: com.sun.jna.LastErrorException: [1] Operation not permitted
        at org.opennms.jicmp.jna.UnixV6NativeSocket.socket(Native Method) ~[org.opennms.core.icmp-jna-29.0.8.jar:?]
        at org.opennms.jicmp.jna.UnixV6NativeSocket.<init>(UnixV6NativeSocket.java:56) ~[org.opennms.core.icmp-jna-29.0.8.jar:?]
        ... 45 more
2022-04-04 12:22:41,815 DEBUG [Main] o.o.n.i.j.JnaPingRequest: 1649067761815: Sending Ping Request: [ID=JnaPingRequestId[addr = /127.0.0.1, ident = 24578, seqNum = 1, tId = 1],Retries=0,Timeout=500,Packet-Size=64,Expiration=-1,Callback=org.opennms.netmgt.icmp.LogPrefixPreservingPingResponseCallback@13d010fe]
2022-04-04 12:22:41,818 DEBUG [Main] o.o.p.r.RequestTracker: Scheding timeout for request to [ID=JnaPingRequestId[addr = /127.0.0.1, ident = 24578, seqNum = 1, tId = 1],Retries=0,Timeout=500,Packet-Size=64,Expiration=1649067762315,Callback=org.opennms.netmgt.icmp.LogPrefixPreservingPingResponseCallback@13d010fe] in 497 ms
2022-04-04 12:22:41,818 INFO  [Main] o.o.n.i.SinglePingResponseCallback: waiting for ping to /127.0.0.1 to finish
2022-04-04 12:22:41,822 INFO  [JNA-ICMP-24578-Callback-Processor] o.o.n.i.SinglePingResponseCallback: got response for address /127.0.0.1, thread 24578, seq 1 with a responseTime 0.686691ms
2022-04-04 12:22:41,822 INFO  [Main] o.o.n.i.SinglePingResponseCallback: finished waiting for ping to /127.0.0.1 to finish
2022-04-04 12:22:41,822 INFO  [Main] o.o.n.i.b.BestMatchPingerFactory: Best available pinger is: class org.opennms.netmgt.icmp.jni.JniPinger
2022-04-04 12:22:41,823 INFO  [Main] o.o.n.v.Manager: Using ICMP implementation: org.opennms.netmgt.icmp.best.BestMatchPinger
2022-04-04 12:22:41,824 INFO  [Main] o.o.n.v.Manager: IPv4 ICMP available? true
2022-04-04 12:22:41,824 INFO  [Main] o.o.n.v.Manager: IPv6 ICMP available? false
2022-04-04 12:22:41,825 INFO  [Main] o.o.n.v.Invoker: Invocation doTestLoadLibraries successful for MBean OpenNMS:Name=TestLoadLibraries
2022-04-04 12:22:41,826 INFO  [Main] o.o.n.v.Invoker: Invoking init on object OpenNMS:Name=Eventd```


Process seems to be stuck:

# opennms -v status
WARNING: relaunching as opennms
OpenNMS.Name=Eventd                 : start_pending
OpenNMS.Name=Alarmd                 : start_pending
OpenNMS.Name=Bsmd                   : start_pending
OpenNMS.Name=Ticketer               : start_pending
OpenNMS.Name=Queued                 : start_pending
OpenNMS.Name=Actiond                : start_pending
OpenNMS.Name=Notifd                 : start_pending
OpenNMS.Name=Scriptd                : start_pending
OpenNMS.Name=Rtcd                   : start_pending
OpenNMS.Name=Pollerd                : start_pending
OpenNMS.Name=EnhancedLinkd          : start_pending
OpenNMS.Name=Collectd               : start_pending
OpenNMS.Name=Discovery              : start_pending
OpenNMS.Name=Vacuumd                : start_pending
OpenNMS.Name=EventTranslator        : start_pending
OpenNMS.Name=PassiveStatusd         : start_pending
OpenNMS.Name=Statsd                 : start_pending
OpenNMS.Name=Provisiond             : start_pending
OpenNMS.Name=Reportd                : start_pending
OpenNMS.Name=Ackd                   : start_pending
OpenNMS.Name=JettyServer            : start_pending
OpenNMS.Name=KarafStartupMonitor    : start_pending
OpenNMS.Name=Telemetryd             : start_pending
OpenNMS.Name=Trapd                  : start_pending
OpenNMS.Name=PerspectivePoller      : start_pending
opennms is partially running

OpenNMS version:
29.0.8

Running as root works fine so seems opennms user lacks some privileges.
Any idea what needs to be done? IPv4 ping seems to be fine and the setting net.ipv4.ping_group_range = 1 993 is there. Is there something similar for IPv6?

I don’t like the ping_group_range solution. I think adding the required capabilities to startup is more elegant:

Add CAP_NET_RAW to the AmbientCapabilities list described in this post, so you have:

AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW

…and you’re off to the races.

Thanks a lot, that got me past that problem. Now I’m getting loads of parse errors in the event files instead. I will check those and get back to you in case I can’t figure it out myself.

It seems to get stuck when starting up Eventd. Any experience with that and what could be the reason? Changing to RUNAS=root makes it work again.

It never gets to these rows when running as the unprivileged user:

2022-04-04 21:37:24,804 INFO  [Main] o.s.c.s.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@6cbf0fc4: startup date [Mon Apr 04 21:37:24 CEST 2022]; parent: ApplicationContext 'daemonContext'
2022-04-04 21:37:24,805 INFO  [Main] o.s.b.f.x.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/opennms/applicationContext-dao.xml]
2022-04-04 21:37:24,819 INFO  [Main] o.s.b.f.x.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/opennms/applicationConte

It just sits there with this log row being the last:

2022-04-04 21:37:23,958 WARN  [Main] o.a.a.j.p.PooledConnectionFactory: Create pooled connection during start failed. This exception will be ignored.
javax.jms.JMSException: Error while attempting to add new Connection to the pool
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createJmsException(PooledConnectionFactory.java:266) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:225) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:204) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.start(PooledConnectionFactory.java:290) [activemq-jms-pool-5.14.5.jar:5.14.5]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1713) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1650) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1579) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:843) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105) [org.apache.servicemix.bundles.spring-context-4.2.9.RELEASE_1.jar:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) [?:?]
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1148) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1050) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:140) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1148) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1050) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) [org.apache.servicemix.bundles.spring-beans-4.2.9.RELEASE_1.jar:?]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648) [org.apache.servicemix.bundle...skipping...
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:275) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:95) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:92) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:221) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        ... 94 more
Caused by: java.io.IOException: Broker named 'localhost' does not exist.
        at org.apache.activemq.transport.vm.VMTransportFactory.doCompositeConnect(VMTransportFactory.java:119) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.vm.VMTransportFactory.doConnect(VMTransportFactory.java:56) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.TransportFactory.connect(TransportFactory.java:65) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:331) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:346) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:304) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:244) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:275) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:95) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:92) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:221) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        ... 94 more
Caused by: java.io.IOException: Broker named 'localhost' does not exist.
        at org.apache.activemq.transport.vm.VMTransportFactory.doCompositeConnect(VMTransportFactory.java:119) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.vm.VMTransportFactory.doConnect(VMTransportFactory.java:56) ~[activemq-broker-5.14.5.jar:5.14.5]
        at org.apache.activemq.transport.TransportFactory.connect(TransportFactory.java:65) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:331) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:346) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:304) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:244) ~[activemq-client-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:275) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:95) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.makeObject(PooledConnectionFactory.java:92) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1221) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:221) ~[activemq-jms-pool-5.14.5.jar:5.14.5]
        ... 94 more

Have you changed anything with the embedded ActiveMQ broker config? Are you using ActiveMQ in your environment?

Have you run $OPENNMS_HOME/bin/fix-permissions ?

Yes, we do indeed use ActiveMQ in our environment. When running as the unprivileged user the AMQ port (61616) does not come up.

Yes, I have run $OPENNMS_HOME/bin/fix-permissions.

I found the problem. In opennms-activemq.xml the data directory was pointing to a directory where the permissions were wrong since it was outside the /opt/opennms hierarchy:

        <property name="properties">
             <value>
                 <!--activemq.data=/usr/share/opennms/data/tmp/activemq -->
                 activemq.data=/opt/opennms/data/tmp/activemq
             </value>
        </property>

Commented out the wrong dir, added the correct dir to the conf, created the dir, ran fix-permissions.sh and restarted opennms.

I also added the following row to /etc/security/limits.d/20-nproc.conf since it was complaining of running out of processes for Provisiond:

opennms soft nproc unlimited

Everything is working now, thanks for the support.

I spoke too soon. We have a listener specified in opennms.properties that binds to 443 and the opennms user is not allowed to bind to the privileged port number:

org.opennms.netmgt.jetty.https-port = 443

Any idea how to solve this the best way or should we set up a proxy in front, for example nginx instead?

This is what the CAP_NET_BIND_SERVICE capability specifically allows…?

However, for SSL termination in general, i recommend putting Apache or nginx on top of it as a reverse proxy. Configuring SSL in Jetty is too much of a headache.

That was a very good suggestion. Set up an NGINX proxy in front of OpenNMS and it works well.

I still have a problem with what appears to be OpenNMS fork bombing the system.

# opennms -v status
WARNING: relaunching as opennms
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: retry: No child processes
/opt/opennms/bin/_lib.sh: fork: Resource temporarily unavailable

Here is the settings I have in the systemd file for opennms:

[Unit]
Description=OpenNMS server
Wants=postgresql.service
Requires=network.target network-online.target
After=postgresql.service postgresql-10.service postgresql-11.service postgresql-12.service postgresql-13.service network.target network-online.target

[Service]
User=opennms
Environment="OPENNMS_HOME=/opt/opennms"
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW
LimitNOFILE=16000
LimitNOFILESoft=16000
LimitNProc=unlimited

Type=forking
PIDFile=/opt/opennms/logs/opennms.pid

ExecStart=/opt/opennms/bin/opennms -s start
ExecStop=/opt/opennms/bin/opennms stop

[Install]
WantedBy=multi-user.target

Any idea where I’ve gone wrong here? When I type lsof | wc -l it just hangs.

LimitNOFILE=16000

Too small. Try adding a zero or two.

There was an offending line under /etc/security/limits.d/20-nproc.conf that overrode what I had in the systemd file:

* soft nproc 4096

After I commented that out it seems to be working flawlessly.

Many thanks for the support!