OpenNMS Telemetryd on Oracle Linux 8.x not starting

Problem:
OpenNMS Telemetryd on Oracle Linux 8.x not starting. The startup process for opennms hangs at Telemetryd and finally times out and stops the opennms startup. We had a successful install directly from the install notes on this server. We have this up and running on several boxes. CentOS 7.x CentOS8.x RHEL VM 7.x-8.x This is the first time running into this error.

We did comment out the service in the xml file to see if would make it all the way through the startup process. It did finish startup with this commented out, but it needs to be in there to work, so it was just a quick test to check. Telemetryd not starting and hanging is the issue.

OpenNMS version:
Version 27.1.0-1
Other relevant data:
[e.g. logs from OpenNMS, error messages etc]
telemetry.log

2021-03-26 16:47:42,563 DEBUG [Main] o.o.n.v.Invoker: Invocation start successful for MBean OpenNMS:Name=KarafStartupMonitor
2021-03-26 16:47:42,563 DEBUG [Main] o.o.n.v.Invoker: pass 1 on service OpenNMS:Name=Telemetryd will invoke method "start"
2021-03-26 16:47:42,563 DEBUG [Main] o.o.n.v.Invoker: Invoking start on object OpenNMS:Name=Telemetryd```
Telemetryd.log
```2021-03-26 16:00:37,106 INFO  [Main] o.s.c.s.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@64a61e86: startup date [Fri Mar 26 16:00:37 CDT 2021]; parent: ApplicationContext 'eventDaemonContext'
2021-03-26 16:00:37,107 INFO  [Main] o.s.b.f.x.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/opennms/applicationContext-telemetryDaemon.xml]
2021-03-26 16:00:37,121 INFO  [Main] o.s.b.f.a.AutowiredAnnotationBeanPostProcessor: JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2021-03-26 16:08:27,247 ERROR [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: Could not start daemon: java.lang.IllegalArgumentException: Each parser must be of type UdpParser but was not.
java.lang.IllegalArgumentException: Each parser must be of type UdpParser but was not.
        at org.opennms.netmgt.telemetry.listeners.factory.UdpListenerFactory.createBean(UdpListenerFactory.java:63) ~[org.opennms.features.telemetry.listeners-27.1.0.jar:?]
        at org.opennms.netmgt.telemetry.listeners.factory.UdpListenerFactory.createBean(UdpListenerFactory.java:42) ~[org.opennms.features.telemetry.listeners-27.1.0.jar:?]
        at org.opennms.netmgt.telemetry.protocols.registry.impl.TelemetryServiceRegistryImpl.getService(TelemetryServiceRegistryImpl.java:121) ~[org.opennms.features.telemetry.registry-27.1.0.jar:?]
        at org.opennms.netmgt.telemetry.protocols.registry.impl.TelemetryServiceRegistryImpl.getService(TelemetryServiceRegistryImpl.java:56) ~[org.opennms.features.telemetry.registry-27.1.0.jar:?]
        at org.opennms.netmgt.telemetry.protocols.registry.impl.TelemetryRegistryImpl.getListener(TelemetryRegistryImpl.java:85) ~[org.opennms.features.telemetry.registry-27.1.0.jar:?]
        at org.opennms.netmgt.telemetry.daemon.Telemetryd.start(Telemetryd.java:140) ~[org.opennms.features.telemetry.daemon-27.1.0.jar:?]

From the telemetryd.log

2021-03-26 16:08:27,247 ERROR [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: Could not start daemon: java.lang.IllegalArgumentException: Each parser must be of type UdpParser but was not.

Sorry all… I hit the save button before finishing the post.

This is the error for Telemetryd not starting.

We do have the full install of Oracle Linux along with gnome installed. It is not a minimal install. I did read this may be an issue.

Looking for the gurus to chime in and tell us what bone head mistakes we have made.

Looks like a Karaf cache problem to me.

Stop OpenNMS
Run ${OPENNMS_HOME}/bin/fix-karaf-setup.sh
Start OpenNMS.

Super thx for the response. We have run the fix-karaf-setup.sh already and cleared the data directory. But we did it again just to make sure. We are still getting no love there.

Last lines of the manager.log after running fix-karaf-seup.sh this time…

2021-03-26 21:55:51,443 DEBUG [Main] o.o.n.v.Invoker: Invoking start on object OpenNMS:Name=KarafStartupMonitor
2021-03-26 21:56:11,445 DEBUG [Main] o.o.n.v.Invoker: Invocation start successful for MBean OpenNMS:Name=KarafStartupMonitor
2021-03-26 21:56:11,445 DEBUG [Main] o.o.n.v.Invoker: pass 1 on service OpenNMS:Name=Telemetryd will invoke method "start"
2021-03-26 21:56:11,445 DEBUG [Main] o.o.n.v.Invoker: Invoking start on object OpenNMS:Name=Telemetryd

Hangs on that last line for 5 minutes or so and then times out and does a full shutdown.

Trying to think of several things we have checked.
We have checked the etc hosts. good there.
Compared this Oracle Linux 8x server to another server CentOS 8x server with netstat -tulpn with IPs and ports. The 127.0.0.1:1190 was not started on our non-working server. But that is grasping at straws since some of the ports are randomly assigned.

Is collectd disabled in service-configuration.xml? that will break telemetryd.

collectd section from service-configuration.xml all looks correct. Not disabled. start … pass=1

      <name>OpenNMS:Name=Collectd</name>
      <class-name>org.opennms.netmgt.collectd.jmx.Collectd</class-name>
      <invoke method="init" pass="0" at="start"/>
      <invoke method="start" pass="1" at="start"/>
      <invoke method="status" pass="0" at="status"/>
      <invoke method="stop" pass="0" at="stop"/>
   </service>

We are working from a base install. No configuration changes. We actually have 3 identical servers with the same OS load and are trying things on 2 boxes and leaving 1 as the base install so we can go back to it and compare, since it’s settings are not changed.

Our next check will be to reload Oracle Linux Minimal install and try again. Will report back here with results.

Thanks.

Oracle Linux isn’t part of the testing regime afaik, but since its just a RHEL ripoff, it shouldn’t cause a problem.

Is there anything potentially illuminating in the output of journalctl -u opennms and/or /var/log/opennms/karaf.log ?

dino2gnt, this is the karaf log from our install:

2021-03-26T16:47:38,507 | ERROR | FelixStartLevel  | ServiceRecipe                    | 18 - org.apache.aries.blueprint.core - 1.10.2 | Error retrieving service from ServiceRecipe[name='.component-2']
org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean dnsResolver
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:593) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.ServiceRecipe.internalGetService(ServiceRecipe.java:252) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.ServiceRecipe.internalCreate(ServiceRecipe.java:149) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) [18:org.apache.aries.blueprint.core:1.10.2]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:360) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:190) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:737) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:433) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:298) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:311) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:280) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:276) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:266) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1179) [?:?]
	at org.apache.felix.framework.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:730) [?:?]
	at org.apache.felix.framework.EventDispatcher.fireBundleEvent(EventDispatcher.java:485) [?:?]
	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4579) [?:?]
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2174) [?:?]
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1373) [?:?]
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	... 35 more
Caused by: io.netty.channel.ChannelException: failed to open a new selector
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	... 35 more
Caused by: java.io.IOException: Too many open files
	at sun.nio.ch.IOUtil.makePipe(Native Method) ~[?:?]
	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:83) ~[?:?]
	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	... 35 more
2021-03-26T16:47:38,532 | ERROR | FelixStartLevel  | ServiceRecipe                    | 18 - org.apache.aries.blueprint.core - 1.10.2 | Error retrieving service from ServiceRecipe[name='.component-2']
org.osgi.service.blueprint.container.ComponentDefinitionException: org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean dnsResolver
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:310) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.ServiceRecipe.internalGetService(ServiceRecipe.java:252) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.ServiceRecipe.internalCreate(ServiceRecipe.java:149) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) [18:org.apache.aries.blueprint.core:1.10.2]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:360) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:190) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:737) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:433) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:298) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:311) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:280) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:276) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:266) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1179) [?:?]
	at org.apache.felix.framework.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:730) [?:?]
	at org.apache.felix.framework.EventDispatcher.fireBundleEvent(EventDispatcher.java:485) [?:?]
	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4579) [?:?]
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2174) [?:?]
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1373) [?:?]
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean dnsResolver
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:593) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more
Caused by: java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more
Caused by: io.netty.channel.ChannelException: failed to open a new selector
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more
Caused by: java.io.IOException: Too many open files
	at sun.nio.ch.IOUtil.makePipe(Native Method) ~[?:?]
	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:83) ~[?:?]
	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more
2021-03-26T16:47:38,556 | WARN  | FelixStartLevel  | BeanRecipe                       | 18 - org.apache.aries.blueprint.core - 1.10.2 | Object to be destroyed is not an instance of UnwrapperedBeanHolder, type: null
2021-03-26T16:47:38,560 | ERROR | FelixStartLevel  | BlueprintContainerImpl           | 18 - org.apache.aries.blueprint.core - 1.10.2 | Unable to start container for blueprint bundle org.opennms.features.dnsresolver.netty/27.1.0
org.osgi.service.blueprint.container.ComponentDefinitionException: org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean dnsResolver
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:310) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.ServiceRecipe.internalGetService(ServiceRecipe.java:252) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.ServiceRecipe.internalCreate(ServiceRecipe.java:149) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:360) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:190) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:737) ~[18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:433) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:298) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:311) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:280) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:276) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:266) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422) [18:org.apache.aries.blueprint.core:1.10.2]
	at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1179) [?:?]
	at org.apache.felix.framework.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:730) [?:?]
	at org.apache.felix.framework.EventDispatcher.fireBundleEvent(EventDispatcher.java:485) [?:?]
	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4579) [?:?]
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2174) [?:?]
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1373) [?:?]
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean dnsResolver
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:593) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more
Caused by: java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more
Caused by: io.netty.channel.ChannelException: failed to open a new selector
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more
Caused by: java.io.IOException: Too many open files
	at sun.nio.ch.IOUtil.makePipe(Native Method) ~[?:?]
	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:83) ~[?:?]
	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyResolverContext.init(NettyResolverContext.java:92) ~[?:?]
	at org.opennms.netmgt.dnsresolver.netty.NettyDnsResolver.init(NettyDnsResolver.java:156) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:337) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:835) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703) ~[?:?]
	at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90) ~[?:?]
	at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:62) ~[?:?]
	at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:108) ~[?:?]
	at org.apache.aries.blueprint.container.ServiceRecipe.createService(ServiceRecipe.java:285) ~[?:?]
	... 27 more```

Thanks for posting this for us Dennis.

I keep going back to this error. We see in the code that this is not matching on length with another value, so the UDP IP:Port seems to not be getting set/allowed? TCP makes it through it seems. Maybe Oracle Linux has something blocking this? guessing?

o.o.n.d.AbstractSpringContextJmxServiceDaemon: Could not start daemon: 
java.lang.IllegalArgumentException: Each parser must be of type 
UdpParser but was not.```

I would address this first. How to investigate file descriptor issues

Update… We fiddled with the settings on the ulimit trying to adjust for the error “to many files open”. No luck.

We ended up going back to a different server, loading Oracle Linux 8.3 and installing opennms there. This was a standard server and not raid 5. As far as we can tell, the raid controller on all of our new servers is giving us grief.

Happy to report the opennms is working just fine on Oracle Linux 8.3 with the standard install notes. But we could not sort out why we were getting the errors on all the servers that had the raid controllers.

We will continue to poke at it and see if we can see what settings are poking us in the eye on this one.