Karaf doesn't start after upgrade from OpenNMS 28.1.1 to 29.0.0

Karaf doesn’t start in root and non-root enviroment. I removed all plugins from system but problem is still exists. I cleared all files and directories from /opt/opennms/data directory but I still have an error. Karaf log is empty but in karafStartupMonitor.log are logs as below.

OpenNMS version:
29.0.0

Other relevant data:
Operating system: CentOS 7.9
JAVA: Open JDK 11.0.13.0.8-1.el7_9
Postgresql: postgresql12-12.8-1PGDG

logs

2021-11-12 11:17:55,081 INFO  [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: karafStartupMonitor initializing.
2021-11-12 11:17:55,082 DEBUG [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: SPRING: thread.classLoader=java.net.FactoryURLClassLoader@6356695f
2021-11-12 11:17:55,084 INFO  [Main] o.s.c.s.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@35e3f377: startup date [Fri Nov 12 11:17:55 CET 2021]; root of context hierarchy
2021-11-12 11:17:55,084 INFO  [Main] o.s.b.f.x.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/opennms/applicationContext-karafStartupMonitor.xml]
2021-11-12 11:17:55,097 DEBUG [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: SPRING: context.classLoader= java.net.FactoryURLClassLoader@6356695f
2021-11-12 11:17:55,098 INFO  [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: karafStartupMonitor initialization complete.
2021-11-12 11:18:41,072 INFO  [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: karafStartupMonitor starting.
2021-11-12 11:18:41,072 DEBUG [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: SPRING: thread.classLoader=java.net.FactoryURLClassLoader@6356695f
2021-11-12 11:18:41,072 INFO  [Main] o.o.f.k.h.d.KarafStartupMonitor: KarafStartupMonitor is starting.
2021-11-12 11:18:41,073 INFO  [Main] o.o.f.k.h.d.KarafStartupMonitor: Waiting for loading of org.opennms.features.karaf.health.service.KarafHealthService, will block startup until service is available.
2021-11-12 11:23:41,096 ERROR [Main] o.o.n.d.AbstractSpringContextJmxServiceDaemon: Could not start daemon: java.lang.IllegalStateException: KarafStartupMonitor: It seems Karaf can't be started properly. This is bad, will fail startup.
What can you do about this?
1.) check in logs/karaf.log for problems
2.) clear the 'data' folder - it contains Karaf's cache
3.) run the script bin/fix-karaf-setup.sh
java.lang.IllegalStateException: KarafStartupMonitor: It seems Karaf can't be started properly. This is bad, will fail startup.
What can you do about this?
1.) check in logs/karaf.log for problems
2.) clear the 'data' folder - it contains Karaf's cache
3.) run the script bin/fix-karaf-setup.sh
        at org.opennms.features.karaf.health.daemon.KarafStartupMonitor.start(KarafStartupMonitor.java:63) ~[org.opennms.features.karaf-health-29.0.0.jar:?]
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon$2.run(AbstractSpringContextJmxServiceDaemon.java:128) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71) [org.opennms.core.logging-29.0.0.jar:?]
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon.start(AbstractSpringContextJmxServiceDaemon.java:118) [org.opennms.core.daemon-29.0.0.jar:?]
        at jdk.internal.reflect.GeneratedMethodAccessor509.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [?:?]
        at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [?:?]
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [?:?]
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) [?:?]
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) [?:?]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) [?:?]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [?:?]
        at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:277) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:206) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:157) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:95) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150) [org.opennms.core.daemon-29.0.0.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:531) [opennms_bootstrap.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2021-11-12 11:23:41,103 INFO  [Main] o.s.c.s.ClassPathXmlApplicationContext: Closing ApplicationContext 'karafStartupMonitorContext': startup date [Fri Nov 12 11:17:55 CET 2021]; root of context hierarchy
2021-11-12 11:23:41,104 INFO  [Main] o.o.f.k.h.d.KarafStartupMonitor: KarafStartupMonitor is stopped.

Try this step from the log message:

:eyes:

The script has been runed. No efects.

Nothing I found in karaf.log.

Jetty errors can also prevent Karaf from starting, is there anything of note in the jetty-server.log or the web.log ?

I found, that jetty server didn’t start too. Server said that no felix extension in karaf.

Post the entire exception if you can.

piÄ…, 12 lis 2021, 13:46:07 CET
begin ulimit settings:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 385736
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 12000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 385736
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
end ulimit settings
Executing command: /opt/opennms/bin/runjava -r -- --add-modules=java.base,java.compiler,java.datatransfer,java.desktop,java.instrument,java.logging,java.management,java
.management.rmi,java.naming,java.prefs,java.rmi,java.scripting,java.security.jgss,java.security.sasl,java.sql,java.sql.rowset,java.xml,jdk.attach,jdk.httpserver,jdk.jdi
,jdk.sctp,jdk.security.auth,jdk.xml.dom -Djava.locale.providers=CLDR,COMPAT -Dopennms.pidfile=/opt/opennms/logs/opennms.pid -agentlib:jdwp=transport=dt_socket,server=y,
address=*:8001,suspend=n -Dopennms.home=/opt/opennms -XX:+HeapDumpOnOutOfMemoryError -Xmx75000m -Djava.net.preferIPv4Stack=true -XX:+UseG1GC -Dcom.sun.management.jmxrem
ote.port=18980 -Dcom.sun.management.jmxremote.local.only=true -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.manage
ment.jmxremote.login.config=opennms -Dcom.sun.management.jmxremote.access.file=/opt/opennms/etc/jmxremote.access -Djdk.attach.allowAttachSelf=true -DisThreadContextMapI
nheritable=true -Dgroovy.use.classvalue=true -XX:MaxMetaspaceSize=512m -Djava.io.tmpdir=/opt/opennms/data/tmp -XX:+StartAttachListener -jar /opt/opennms/lib/opennms_boo
tstrap.jar start
Listening for transport dt_socket at address: 8001
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/opt/opennms/lib/org.apache.servicemix.bundles.spring-core-4.2.9.RELEASE_1.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
lis 12, 2021 1:46:22 PM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 4.3.2.Final
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
init...
java.lang.RuntimeException: Could not resolve mvn:org.apache.felix/org.apache.felix.framework/5.6.12
        at org.apache.karaf.main.util.SimpleMavenResolver.resolve(SimpleMavenResolver.java:59)
        at org.apache.karaf.main.Main.createClassLoader(Main.java:512)
        at org.apache.karaf.main.Main.launch(Main.java:282)
        at org.opennms.container.web.WebAppListener.contextInitialized(WebAppListener.java:75)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1067)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
        at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:996)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:910)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:517)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:157)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173)
        at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:428)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:785)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:754)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:605)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:252)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.opennms.netmgt.jetty.JettyServer.onStart(JettyServer.java:111)
        at org.opennms.netmgt.daemon.AbstractServiceDaemon$4.run(AbstractServiceDaemon.java:297)
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71)
        at org.opennms.netmgt.daemon.AbstractServiceDaemon.start(AbstractServiceDaemon.java:290)
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon$2.run(AbstractSpringContextJmxServiceDaemon.java:128)
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71)
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon.start(AbstractSpringContextJmxServiceDaemon.java:118)
        at jdk.internal.reflect.GeneratedMethodAccessor511.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260)
        at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at java.management/com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at java.management/com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:277)
        at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:206)
        at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:157)
        at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:95)
        at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173)
        at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:531)
        at java.base/java.lang.Thread.run(Thread.java:829)
lis 12, 2021 1:50:03 PM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 4.3.2.Final
java.net.SocketException: Protocol family unavailable
        at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:117)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:42)
        at org.snmp4j.MessageDispatcherImpl.sendMessage(MessageDispatcherImpl.java:203)
        at org.snmp4j.MessageDispatcherImpl.sendPdu(MessageDispatcherImpl.java:519)
        at org.snmp4j.Snmp.sendMessage(Snmp.java:1063)
        at org.snmp4j.Snmp.send(Snmp.java:1033)
        at org.snmp4j.Snmp.send(Snmp.java:1017)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.sendNextPdu(Snmp4JWalker.java:236)
        at org.opennms.netmgt.snmp.SnmpWalker.buildAndSendNextPdu(SnmpWalker.java:104)
        at org.opennms.netmgt.snmp.SnmpWalker.start(SnmpWalker.java:88)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.start(Snmp4JWalker.java:210)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.walk(SnmpProxyRpcModule.java:164)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:94)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:63)
        at org.opennms.core.rpc.camel.CamelRpcClientFactory$1.execute(CamelRpcClientFactory.java:125)
        at org.opennms.netmgt.snmp.proxy.common.LocationAwareSnmpClientRpcImpl.execute(LocationAwareSnmpClientRpcImpl.java:126)
        at org.opennms.netmgt.snmp.proxy.common.AbstractSNMPRequestBuilder.execute(AbstractSNMPRequestBuilder.java:108)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryCdp.collect(NodeDiscoveryCdp.java:86)
        at org.opennms.netmgt.enlinkd.common.NodeCollector.runDiscovery(NodeCollector.java:84)
        at org.opennms.netmgt.enlinkd.common.Discovery.run(Discovery.java:105)
        at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
        at java.base/java.lang.Thread.run(Thread.java:829)
java.net.SocketException: Protocol family unavailable
        at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:117)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:42)
        at org.snmp4j.MessageDispatcherImpl.sendMessage(MessageDispatcherImpl.java:203)
        at org.snmp4j.MessageDispatcherImpl.sendPdu(MessageDispatcherImpl.java:519)
        at org.snmp4j.Snmp.sendMessage(Snmp.java:1063)
        at org.snmp4j.Snmp.send(Snmp.java:1033)
        at org.snmp4j.Snmp.send(Snmp.java:1017)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.sendNextPdu(Snmp4JWalker.java:236)
        at org.opennms.netmgt.snmp.SnmpWalker.buildAndSendNextPdu(SnmpWalker.java:104)
        at org.opennms.netmgt.snmp.SnmpWalker.start(SnmpWalker.java:88)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.start(Snmp4JWalker.java:210)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.walk(SnmpProxyRpcModule.java:164)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:94)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:63)
        at org.opennms.core.rpc.camel.CamelRpcClientFactory$1.execute(CamelRpcClientFactory.java:125)
        at org.opennms.netmgt.snmp.proxy.common.LocationAwareSnmpClientRpcImpl.execute(LocationAwareSnmpClientRpcImpl.java:126)
        at org.opennms.netmgt.snmp.proxy.common.AbstractSNMPRequestBuilder.execute(AbstractSNMPRequestBuilder.java:108)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.getVtpVlanMap(NodeDiscoveryBridge.java:240)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.collect(NodeDiscoveryBridge.java:105)
        at org.opennms.netmgt.enlinkd.common.NodeCollector.runDiscovery(NodeCollector.java:84)
        at org.opennms.netmgt.enlinkd.common.Discovery.run(Discovery.java:105)
        at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
        at java.base/java.lang.Thread.run(Thread.java:829)
java.net.SocketException: Protocol family unavailable
        at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:117)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:42)
        at org.snmp4j.MessageDispatcherImpl.sendMessage(MessageDispatcherImpl.java:203)
        at org.snmp4j.MessageDispatcherImpl.sendPdu(MessageDispatcherImpl.java:519)
        at org.snmp4j.Snmp.sendMessage(Snmp.java:1063)
        at org.snmp4j.Snmp.send(Snmp.java:1033)
        at org.snmp4j.Snmp.send(Snmp.java:1017)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.sendNextPdu(Snmp4JWalker.java:236)
        at org.opennms.netmgt.snmp.SnmpWalker.buildAndSendNextPdu(SnmpWalker.java:104)
        at org.opennms.netmgt.snmp.SnmpWalker.start(SnmpWalker.java:88)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.start(Snmp4JWalker.java:210)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.walk(SnmpProxyRpcModule.java:164)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:94)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:63)
        at org.opennms.core.rpc.camel.CamelRpcClientFactory$1.execute(CamelRpcClientFactory.java:125)
        at org.opennms.netmgt.snmp.proxy.common.LocationAwareSnmpClientRpcImpl.execute(LocationAwareSnmpClientRpcImpl.java:126)
        at org.opennms.netmgt.snmp.proxy.common.AbstractSNMPRequestBuilder.execute(AbstractSNMPRequestBuilder.java:108)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryIpNetToMedia.collect(NodeDiscoveryIpNetToMedia.java:105)
        at org.opennms.netmgt.enlinkd.common.NodeCollector.runDiscovery(NodeCollector.java:84)
        at org.opennms.netmgt.enlinkd.common.Discovery.run(Discovery.java:105)
        at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
        at java.base/java.lang.Thread.run(Thread.java:829)
java.net.SocketException: Protocol family unavailable
        at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:117)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:42)
        at org.snmp4j.MessageDispatcherImpl.sendMessage(MessageDispatcherImpl.java:203)
        at org.snmp4j.MessageDispatcherImpl.sendPdu(MessageDispatcherImpl.java:519)
        at org.snmp4j.Snmp.sendMessage(Snmp.java:1063)
        at org.snmp4j.Snmp.send(Snmp.java:1033)
        at org.snmp4j.Snmp.send(Snmp.java:1017)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.sendNextPdu(Snmp4JWalker.java:236)
        at org.opennms.netmgt.snmp.SnmpWalker.buildAndSendNextPdu(SnmpWalker.java:104)
        at org.opennms.netmgt.snmp.SnmpWalker.start(SnmpWalker.java:88)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.start(Snmp4JWalker.java:210)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.walk(SnmpProxyRpcModule.java:164)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:94)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:63)
        at org.opennms.core.rpc.camel.CamelRpcClientFactory$1.execute(CamelRpcClientFactory.java:125)
        at org.opennms.netmgt.snmp.proxy.common.LocationAwareSnmpClientRpcImpl.execute(LocationAwareSnmpClientRpcImpl.java:126)
        at org.opennms.netmgt.snmp.proxy.common.AbstractSNMPRequestBuilder.execute(AbstractSNMPRequestBuilder.java:108)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.walkDot1dBasePortTable(NodeDiscoveryBridge.java:297)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.collect(NodeDiscoveryBridge.java:129)
        at org.opennms.netmgt.enlinkd.common.NodeCollector.runDiscovery(NodeCollector.java:84)
        at org.opennms.netmgt.enlinkd.common.Discovery.run(Discovery.java:105)
        at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
        at java.base/java.lang.Thread.run(Thread.java:829)
java.net.SocketException: Protocol family unavailable
        at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:117)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:42)
        at org.snmp4j.MessageDispatcherImpl.sendMessage(MessageDispatcherImpl.java:203)
        at org.snmp4j.MessageDispatcherImpl.sendPdu(MessageDispatcherImpl.java:519)
        at org.snmp4j.Snmp.sendMessage(Snmp.java:1063)
        at org.snmp4j.Snmp.send(Snmp.java:1033)
        at org.snmp4j.Snmp.send(Snmp.java:1017)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.sendNextPdu(Snmp4JWalker.java:236)
        at org.opennms.netmgt.snmp.SnmpWalker.buildAndSendNextPdu(SnmpWalker.java:104)
        at org.opennms.netmgt.snmp.SnmpWalker.start(SnmpWalker.java:88)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.start(Snmp4JWalker.java:210)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.walk(SnmpProxyRpcModule.java:164)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:94)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:63)
        at org.opennms.core.rpc.camel.CamelRpcClientFactory$1.execute(CamelRpcClientFactory.java:125)
        at org.opennms.netmgt.snmp.proxy.common.LocationAwareSnmpClientRpcImpl.execute(LocationAwareSnmpClientRpcImpl.java:126)
        at org.opennms.netmgt.snmp.proxy.common.AbstractSNMPRequestBuilder.execute(AbstractSNMPRequestBuilder.java:108)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.getDot1dBridgeBase(NodeDiscoveryBridge.java:183)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.collect(NodeDiscoveryBridge.java:137)
        at org.opennms.netmgt.enlinkd.common.NodeCollector.runDiscovery(NodeCollector.java:84)
        at org.opennms.netmgt.enlinkd.common.Discovery.run(Discovery.java:105)
        at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
        at java.base/java.lang.Thread.run(Thread.java:829)
java.net.SocketException: Protocol family unavailable
        at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:117)
        at org.snmp4j.transport.DefaultUdpTransportMapping.sendMessage(DefaultUdpTransportMapping.java:42)
        at org.snmp4j.MessageDispatcherImpl.sendMessage(MessageDispatcherImpl.java:203)
        at org.snmp4j.MessageDispatcherImpl.sendPdu(MessageDispatcherImpl.java:519)
        at org.snmp4j.Snmp.sendMessage(Snmp.java:1063)
        at org.snmp4j.Snmp.send(Snmp.java:1033)
        at org.snmp4j.Snmp.send(Snmp.java:1017)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.sendNextPdu(Snmp4JWalker.java:236)
        at org.opennms.netmgt.snmp.SnmpWalker.buildAndSendNextPdu(SnmpWalker.java:104)
        at org.opennms.netmgt.snmp.SnmpWalker.start(SnmpWalker.java:88)
        at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.start(Snmp4JWalker.java:210)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.walk(SnmpProxyRpcModule.java:164)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:94)
        at org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule.execute(SnmpProxyRpcModule.java:63)
        at org.opennms.core.rpc.camel.CamelRpcClientFactory$1.execute(CamelRpcClientFactory.java:125)
        at org.opennms.netmgt.snmp.proxy.common.LocationAwareSnmpClientRpcImpl.execute(LocationAwareSnmpClientRpcImpl.java:126)
        at org.opennms.netmgt.snmp.proxy.common.AbstractSNMPRequestBuilder.execute(AbstractSNMPRequestBuilder.java:108)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.walkDot1qTpFdb(NodeDiscoveryBridge.java:516)
        at org.opennms.netmgt.enlinkd.NodeDiscoveryBridge.collect(NodeDiscoveryBridge.java:169)
        at org.opennms.netmgt.enlinkd.common.NodeCollector.runDiscovery(NodeCollector.java:84)
        at org.opennms.netmgt.enlinkd.common.Discovery.run(Discovery.java:105)
        at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
        at java.base/java.lang.Thread.run(Thread.java:829)
An error occurred while attempting to start the "OpenNMS:Name=KarafStartupMonitor" service (class org.opennms.netmgt.daemon.SimpleSpringContextJmxServiceDaemon).  Shutting down and exiting.
javax.management.RuntimeMBeanException: java.lang.IllegalStateException: KarafStartupMonitor: It seems Karaf can't be started properly. This is bad, will fail startup.
What can you do about this?
1.) check in logs/karaf.log for problems
2.) clear the 'data' folder - it contains Karaf's cache
3.) run the script bin/fix-karaf-setup.sh
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:829)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:842)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:811)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:277)
        at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:206)
        at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:157)
        at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:95)
        at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173)
        at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:531)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: KarafStartupMonitor: It seems Karaf can't be started properly. This is bad, will fail startup.
What can you do about this?
1.) check in logs/karaf.log for problems
2.) clear the 'data' folder - it contains Karaf's cache
3.) run the script bin/fix-karaf-setup.sh
        at org.opennms.features.karaf.health.daemon.KarafStartupMonitor.start(KarafStartupMonitor.java:63)
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon$2.run(AbstractSpringContextJmxServiceDaemon.java:128)
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71)
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon.start(AbstractSpringContextJmxServiceDaemon.java:118)
        at jdk.internal.reflect.GeneratedMethodAccessor511.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260)
        at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at java.management/com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at java.management/com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809)
        ... 13 more
2021-11-12 15:11:53,303 DEBUG [Main] o.a.c.s.SpringCamelContext: onApplicationEvent: org.springframework.context.event.ContextRefreshedEvent[source=Root WebApplicationC
ontext: startup date [Fri Nov 12 15:11:47 CET 2021]; parent: ApplicationContext 'webContext']
2021-11-12 15:11:53,303 DEBUG [Main] o.a.c.s.SpringCamelContext: onApplicationEvent: org.springframework.context.event.ContextRefreshedEvent[source=Root WebApplicationC
ontext: startup date [Fri Nov 12 15:11:47 CET 2021]; parent: ApplicationContext 'webContext']
2021-11-12 15:11:53,304 DEBUG [Main] o.e.j.j.InitialContextFactory: InitialContextFactory.getInitialContext()
2021-11-12 15:11:53,305 DEBUG [Main] o.e.j.j.InitialContextFactory: Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@5
91297af
2021-11-12 15:11:53,308 DEBUG [Main] o.e.j.j.j.javaURLContextFactory: >>> new root context requested
2021-11-12 15:11:53,308 DEBUG [Main] o.e.j.j.NamingContext: Looking up name="comp/env/spring.liveBeansView.mbeanDomain"
2021-11-12 15:11:53,308 DEBUG [Main] o.e.j.j.ContextFactory: Trying thread context classloader
2021-11-12 15:11:53,311 DEBUG [Main] o.e.j.j.NamingContext: Looking up name="env/spring.liveBeansView.mbeanDomain"
2021-11-12 15:11:53,312 DEBUG [Main] o.e.j.j.InitialContextFactory: InitialContextFactory.getInitialContext()
2021-11-12 15:11:53,312 DEBUG [Main] o.e.j.j.InitialContextFactory: Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@2
fdbaa07
2021-11-12 15:11:53,312 DEBUG [Main] o.e.j.j.l.localContextRoot: Looking up name="spring.liveBeansView.mbeanDomain"
2021-11-12 15:11:53,313 INFO  [Main] o.s.w.c.ContextLoader: Root WebApplicationContext: initialization completed in 5389 ms
2021-11-12 15:11:53,313 DEBUG [Main] o.e.j.s.h.ContextHandler: contextInitialized: javax.servlet.ServletContextEvent[source=ServletContext@o.e.j.w.WebAppContext@85516b5
{opennms,/opennms,[file:///opt/opennms/jetty-webapps/opennms/, jar:file:///opt/opennms/jetty-webapps/opennms/WEB-INF/lib/swagger-ui-3.13.0.jar!/META-INF/resources],STAR
TING}{/opt/opennms/jetty-webapps/opennms}]->org.opennms.container.web.WebAppListener@e8a561a
2021-11-12 15:11:53,315 INFO  [Main] o.e.j.s.h.C.opennms: contextInitialized
2021-11-12 15:11:53,315 INFO  [Main] o.e.j.s.h.C.opennms: Root: /opt/opennms
2021-11-12 15:11:53,397 WARN  [Main] o.e.j.s.h.C.opennms: Unexpected exception while starting Karaf
java.lang.RuntimeException: Could not resolve mvn:org.apache.felix/org.apache.felix.framework/5.6.12
        at org.apache.karaf.main.util.SimpleMavenResolver.resolve(SimpleMavenResolver.java:59) ~[org.apache.karaf.main-4.3.2.jar:4.3.2]
        at org.apache.karaf.main.Main.createClassLoader(Main.java:512) ~[org.apache.karaf.main-4.3.2.jar:4.3.2]
        at org.apache.karaf.main.Main.launch(Main.java:282) ~[org.apache.karaf.main-4.3.2.jar:4.3.2]
        at org.opennms.container.web.WebAppListener.contextInitialized(WebAppListener.java:75) [org.opennms.container.servlet-29.0.0.jar:?]
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1067) [jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) [jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:996) [jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) [jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) [jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) [jetty-webapp-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) [jetty-webapp-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:910) [jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) [jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) [jetty-webapp-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:517) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:157) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:428) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:785) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:754) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:605) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:252) [jetty-deploy-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.Server.start(Server.java:423) [jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) [jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.Server.doStart(Server.java:387) [jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) [jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.opennms.netmgt.jetty.JettyServer.onStart(JettyServer.java:111) [opennms-jetty-29.0.0.jar:?]
        at org.opennms.netmgt.daemon.AbstractServiceDaemon$4.run(AbstractServiceDaemon.java:297) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71) [org.opennms.core.logging-29.0.0.jar:?]
        at org.opennms.netmgt.daemon.AbstractServiceDaemon.start(AbstractServiceDaemon.java:290) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon$2.run(AbstractSpringContextJmxServiceDaemon.java:128) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.core.logging.Logging.withPrefix(Logging.java:71) [org.opennms.core.logging-29.0.0.jar:?]
        at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon.start(AbstractSpringContextJmxServiceDaemon.java:118) [org.opennms.core.daemon-29.0.0.jar:?]
        at jdk.internal.reflect.GeneratedMethodAccessor506.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [?:?]
        at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [?:?]
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [?:?]
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [?:?]
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) [?:?]
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) [?:?]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) [?:?]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [?:?]
        at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:277) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:206) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:157) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:95) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173) [org.opennms.core.daemon-29.0.0.jar:?]
        at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150) [org.opennms.core.daemon-29.0.0.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:531) [opennms_bootstrap.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2021-11-12 15:11:53,404 DEBUG [Main] o.e.j.s.h.ContextHandler: contextInitialized: javax.servlet.ServletContextEvent[source=ServletContext@o.e.j.w.WebAppContext@85516b5{opennms,/opennms,[file:///opt/opennms/jetty-webapps/opennms/, jar:file:///opt/opennms/jetty-webapps/opennms/WEB-INF/lib/swagger-ui-3.13.0.jar!/META-INF/resources],STARTING}{/opt/opennms/jetty-webapps/opennms}]->org.apache.felix.http.proxy.ProxyListener@140150d1
2021-11-12 15:11:53,404 DEBUG [Main] o.e.j.u.c.AbstractLifeCycle: starting characterEncodingFilter==org.springframework.web.filter.CharacterEncodingFilter@25c43eb7{inst=false,async=false,src=DESCRIPTOR:file:///opt/opennms/jetty-webapps/opennms/WEB-INF/web.xml}
2021-11-12 15:11:53,406 DEBUG [Main] o.e.j.s.BaseHolder: Holding class org.springframework.web.filter.CharacterEncodingFilter from java.net.FactoryURLClassLoader@5ffead27
2021-11-12 15:11:53,406 DEBUG [Main] o.e.j.u.c.AbstractLifeCycle: STARTED @221183ms characterEncodingFilter==org.springframework.web.filter.CharacterEncodingFilter@25c43eb7{inst=false,async=false,src=DESCRIPTOR:file:///opt/opennms/jetty-webapps/opennms/WEB-INF/web.xml}
2021-11-12 15:11:53,407 DEBUG [Main] o.e.j.j.InitialContextFactory: InitialContextFactory.getInitialContext()
2021-11-12 15:11:53,407 DEBUG [Main] o.e.j.j.InitialContextFactory: Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@1efc3dfd
2021-11-12 15:11:53,409 DEBUG [Main] o.e.j.s.FilterHolder: Filter.init org.springframework.web.filter.CharacterEncodingFilter@2691b724
2021-11-12 15:11:53,425 DEBUG [Main] o.e.j.u.c.AbstractLifeCycle: starting jSessionIdNoCacheFilter==org.opennms.web.security.JSessionIdNoCacheFilter@6c6c1442{inst=false,async=false,src=DESCRIPTOR:file:///opt/opennms/jetty-webapps/opennms/WEB-INF/web.xml}
2021-11-12 15:11:53,426 DEBUG [Main] o.e.j.s.BaseHolder: Holding class org.opennms.web.security.JSessionIdNoCacheFilter from WebAppClassLoader{opennms}@57473e50
2021-11-12 15:11:53,426 DEBUG [Main] o.e.j.u.c.AbstractLifeCycle: STARTED @221203ms jSessionIdNoCacheFilter==org.opennms.web.security.JSessionIdNoCacheFilter@6c6c1442{inst=false,async=false,src=DESCRIPTOR:file:///opt/opennms/jetty-webapps/opennms/WEB-INF/web.xml}

I found many problems in version 29.0.0.
In non-root enviroment I could not change the number of file descriptors by opennms.conf.
In root and non-root enviroment ssl did not start any more. I can not see any security java classes in log when I turn on debug mod on jetty. I didn’t touch https config from many years.
I had to copy all files from etc-pristine to rerun opennms process. Very buggy verssion.

No arguments here.