Opennms 30 after upgrade, Pollerd ~1300sec to successful init

Problem - Pollerd takes 21 minutes to respond with successful init
First off this is an unsupported platform - openSUSE Leap 15.3
I’ve never had issues I couldn’t fix before (been running since the 1.0 days) but I’m stumped now. When I start opennms from opennms/bin a status check shows all daemons at start pending. I give it a few minutes and everything remains at stat pending, so I kill process. I find no errors in the logs that are unexpected (ip address, spring warnings, etc). Did this multiple times over the last few days.

This morning I left the start running and Pollerd finished startup but jetty server quickly crashed afterwards. I’ll take a look at the jetty process but how can I troubleshoot the Pollerd issue? 21 minutes is a long time. Pollerd doesn’t appear to be stuck as the log continually shows Jdbc calls.

I’ve copied below the opennms -v start output up to the jetty crash. I’ve also got the pollerd debug logs (81M) but they don’t show an error. Other than installing the pristine config are there other changes I can make to troubleshoot the Pollerd delay?

Expected outcome:
pollerd less than 5 minutes to start

Opennms Version 30

opennms -v -f start

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 257193
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 50480
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 257193
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[0.010s][info][gc] Using G1
[0.630s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 52M->25M(1006M) 9.135ms
[0.838s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 67M->26M(1006M) 3.607ms
[1.379s][info][gc] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 74M->28M(1006M) 3.190ms
[1.524s][info][gc] GC(3) Pause Young (Concurrent Start) (Metadata GC Threshold) 44M->28M(1006M) 5.606ms
[1.524s][info][gc] GC(4) Concurrent Cycle
[1.533s][info][gc] GC(4) Pause Remark 29M->29M(1006M) 1.981ms
[1.535s][info][gc] GC(4) Pause Cleanup 29M->29M(1006M) 0.127ms
[1.537s][info][gc] GC(4) Concurrent Cycle 13.216ms
[INFO] Invoking init on object OpenNMS:Name=Manager
[INFO] Invocation init successful for MBean OpenNMS:Name=Manager
[INFO] Invoking doTestLoadLibraries on object OpenNMS:Name=TestLoadLibraries
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
[INFO] Using ICMP implementation: org.opennms.netmgt.icmp.best.BestMatchPinger
[INFO] IPv4 ICMP available? true
[INFO] IPv6 ICMP available? true
[INFO] Invocation doTestLoadLibraries successful for MBean OpenNMS:Name=TestLoadLibraries
[INFO] Invoking init on object OpenNMS:Name=Eventd
[2.586s][info][gc] GC(5) Pause Young (Normal) (G1 Evacuation Pause) 140M->31M(1006M) 4.698ms
[3.079s][info][gc] GC(6) Pause Young (Concurrent Start) (Metadata GC Threshold) 113M->32M(1006M) 9.435ms
[3.079s][info][gc] GC(7) Concurrent Cycle
[3.088s][info][gc] GC(7) Pause Remark 34M->34M(1006M) 1.922ms
[3.090s][info][gc] GC(7) Pause Cleanup 34M->34M(1006M) 0.097ms
[3.093s][info][gc] GC(7) Concurrent Cycle 13.331ms
[3.993s][info][gc] GC(8) Pause Young (Normal) (G1 Evacuation Pause) 192M->36M(1006M) 7.157ms
[6.796s][info][gc] GC(9) Pause Young (Normal) (G1 Evacuation Pause) 602M->93M(1006M) 23.637ms
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/opt/opennms/lib/org.apache.servicemix.bundles.spring-core-4.2.9.RELEASE_1.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so.  Attempting to load jicmp6 library from this location.
[INFO] Successfully loaded jicmp6 library.
[9.163s][info][gc] GC(10) Pause Young (Normal) (G1 Evacuation Pause) 535M->109M(1006M) 49.818ms
[9.563s][info][gc] GC(11) Pause Young (Concurrent Start) (Metadata GC Threshold) 212M->107M(1208M) 14.857ms
[9.563s][info][gc] GC(12) Concurrent Cycle
[9.626s][info][gc] GC(12) Pause Remark 114M->114M(1208M) 14.139ms
[9.649s][info][gc] GC(12) Pause Cleanup 116M->116M(1208M) 0.140ms
[9.655s][info][gc] GC(12) Concurrent Cycle 92.056ms
Jun 14, 2022 6:38:47 AM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 4.3.2.Final
[11.832s][info][gc] GC(13) Pause Young (Concurrent Start) (G1 Evacuation Pause) 473M->119M(1208M) 30.414ms
[11.832s][info][gc] GC(14) Concurrent Cycle
[11.906s][info][gc] GC(14) Pause Remark 138M->138M(1208M) 5.747ms
[11.926s][info][gc] GC(14) Pause Cleanup 142M->142M(1208M) 0.144ms
[11.931s][info][gc] GC(14) Concurrent Cycle 98.779ms
[14.498s][info][gc] GC(15) Pause Young (Normal) (G1 Evacuation Pause) 647M->126M(1208M) 20.232ms
[18.245s][info][gc] GC(16) Pause Young (Normal) (G1 Evacuation Pause) 734M->140M(1208M) 24.632ms
[INFO] Invocation init successful for MBean OpenNMS:Name=Eventd
[INFO] Invoking init on object OpenNMS:Name=Alarmd
[INFO] Invocation init successful for MBean OpenNMS:Name=Alarmd
[INFO] Invoking init on object OpenNMS:Name=Bsmd
[INFO] Invocation init successful for MBean OpenNMS:Name=Bsmd
[INFO] Invoking init on object OpenNMS:Name=Ticketer
[INFO] Invocation init successful for MBean OpenNMS:Name=Ticketer
[INFO] Invoking init on object OpenNMS:Name=Queued
[INFO] Invocation init successful for MBean OpenNMS:Name=Queued
[INFO] Invoking init on object OpenNMS:Name=Actiond
[INFO] Invocation init successful for MBean OpenNMS:Name=Actiond
[INFO] Invoking init on object OpenNMS:Name=Notifd
[INFO] Invocation init successful for MBean OpenNMS:Name=Notifd
[INFO] Invoking init on object OpenNMS:Name=Scriptd
[INFO] Invocation init successful for MBean OpenNMS:Name=Scriptd
[INFO] Invoking init on object OpenNMS:Name=Rtcd
[INFO] Invocation init successful for MBean OpenNMS:Name=Rtcd
[INFO] Invoking init on object OpenNMS:Name=Pollerd
[32.992s][info][gc] GC(17) Pause Young (Normal) (G1 Evacuation Pause) 818M->160M(1208M) 23.073ms
[57.570s][info][gc] GC(18) Pause Young (Normal) (G1 Evacuation Pause) 818M->163M(1208M) 30.830ms
[83.062s][info][gc] GC(19) Pause Young (Normal) (G1 Evacuation Pause) 845M->162M(1208M) 13.353ms
[108.879s][info][gc] GC(20) Pause Young (Normal) (G1 Evacuation Pause) 844M->162M(1208M) 11.384ms
[134.744s][info][gc] GC(21) Pause Young (Normal) (G1 Evacuation Pause) 844M->161M(1208M) 13.503ms
[158.174s][info][gc] GC(22) Pause Young (Normal) (G1 Evacuation Pause) 845M->161M(1208M) 10.557ms
[184.228s][info][gc] GC(23) Pause Young (Normal) (G1 Evacuation Pause) 845M->161M(1208M) 13.176ms
[210.202s][info][gc] GC(24) Pause Young (Normal) (G1 Evacuation Pause) 845M->161M(1208M) 11.771ms
[236.155s][info][gc] GC(25) Pause Young (Normal) (G1 Evacuation Pause) 845M->162M(1208M) 18.899ms
[259.998s][info][gc] GC(26) Pause Young (Normal) (G1 Evacuation Pause) 844M->177M(1208M) 17.591ms
[285.689s][info][gc] GC(27) Pause Young (Normal) (G1 Evacuation Pause) 845M->164M(1208M) 14.262ms
[312.237s][info][gc] GC(28) Pause Young (Normal) (G1 Evacuation Pause) 862M->163M(1208M) 8.538ms
[337.222s][info][gc] GC(29) Pause Young (Normal) (G1 Evacuation Pause) 861M->162M(1208M) 9.061ms
[363.776s][info][gc] GC(30) Pause Young (Normal) (G1 Evacuation Pause) 862M->162M(1208M) 7.241ms
[387.930s][info][gc] GC(31) Pause Young (Normal) (G1 Evacuation Pause) 862M->163M(1208M) 7.361ms
[417.546s][info][gc] GC(32) Pause Young (Normal) (G1 Evacuation Pause) 861M->164M(1208M) 10.654ms
[449.086s][info][gc] GC(33) Pause Young (Normal) (G1 Evacuation Pause) 878M->164M(1208M) 5.669ms
[479.793s][info][gc] GC(34) Pause Young (Normal) (G1 Evacuation Pause) 880M->164M(1208M) 5.576ms
[506.805s][info][gc] GC(35) Pause Young (Normal) (G1 Evacuation Pause) 882M->164M(1208M) 4.592ms
[537.574s][info][gc] GC(36) Pause Young (Normal) (G1 Evacuation Pause) 882M->165M(1208M) 4.426ms
[564.899s][info][gc] GC(37) Pause Young (Normal) (G1 Evacuation Pause) 883M->165M(1208M) 4.120ms
[591.961s][info][gc] GC(38) Pause Young (Normal) (G1 Evacuation Pause) 883M->165M(1208M) 4.326ms
[617.845s][info][gc] GC(39) Pause Young (Normal) (G1 Evacuation Pause) 883M->166M(1208M) 5.949ms
[642.424s][info][gc] GC(40) Pause Young (Normal) (G1 Evacuation Pause) 882M->167M(1208M) 5.352ms
[669.603s][info][gc] GC(41) Pause Young (Normal) (G1 Evacuation Pause) 883M->167M(1208M) 4.065ms
[696.678s][info][gc] GC(42) Pause Young (Normal) (G1 Evacuation Pause) 883M->167M(1208M) 5.283ms
[724.048s][info][gc] GC(43) Pause Young (Normal) (G1 Evacuation Pause) 883M->167M(1208M) 4.808ms
[749.625s][info][gc] GC(44) Pause Young (Normal) (G1 Evacuation Pause) 883M->168M(1208M) 4.400ms
[777.690s][info][gc] GC(45) Pause Young (Normal) (G1 Evacuation Pause) 884M->168M(1208M) 4.766ms
[805.516s][info][gc] GC(46) Pause Young (Normal) (G1 Evacuation Pause) 884M->169M(1208M) 4.915ms
[832.542s][info][gc] GC(47) Pause Young (Normal) (G1 Evacuation Pause) 885M->168M(1208M) 5.695ms
[859.091s][info][gc] GC(48) Pause Young (Normal) (G1 Evacuation Pause) 884M->175M(1208M) 6.294ms
[884.594s][info][gc] GC(49) Pause Young (Normal) (G1 Evacuation Pause) 885M->170M(1208M) 6.279ms
[912.057s][info][gc] GC(50) Pause Young (Normal) (G1 Evacuation Pause) 886M->170M(1208M) 6.405ms
[937.992s][info][gc] GC(51) Pause Young (Normal) (G1 Evacuation Pause) 884M->170M(1208M) 5.430ms
[965.572s][info][gc] GC(52) Pause Young (Normal) (G1 Evacuation Pause) 884M->171M(1208M) 6.228ms
[990.475s][info][gc] GC(53) Pause Young (Normal) (G1 Evacuation Pause) 885M->171M(1208M) 7.857ms
[1019.428s][info][gc] GC(54) Pause Young (Normal) (G1 Evacuation Pause) 887M->172M(1208M) 5.867ms
[1051.561s][info][gc] GC(55) Pause Young (Normal) (G1 Evacuation Pause) 886M->171M(1208M) 5.414ms
[1084.758s][info][gc] GC(56) Pause Young (Normal) (G1 Evacuation Pause) 887M->172M(1208M) 5.837ms
[1114.944s][info][gc] GC(57) Pause Young (Normal) (G1 Evacuation Pause) 888M->172M(1208M) 7.725ms
[1143.089s][info][gc] GC(58) Pause Young (Normal) (G1 Evacuation Pause) 888M->172M(1208M) 5.953ms
[1170.846s][info][gc] GC(59) Pause Young (Normal) (G1 Evacuation Pause) 888M->172M(1208M) 6.300ms
[1198.100s][info][gc] GC(60) Pause Young (Normal) (G1 Evacuation Pause) 888M->173M(1208M) 4.611ms
[1221.669s][info][gc] GC(61) Pause Young (Normal) (G1 Evacuation Pause) 889M->173M(1208M) 6.185ms
[1248.833s][info][gc] GC(62) Pause Young (Normal) (G1 Evacuation Pause) 889M->173M(1208M) 5.377ms
[1276.031s][info][gc] GC(63) Pause Young (Normal) (G1 Evacuation Pause) 889M->174M(1208M) 6.562ms
[1303.239s][info][gc] GC(64) Pause Young (Normal) (G1 Evacuation Pause) 890M->174M(1208M) 5.177ms
[INFO] Invocation init successful for MBean OpenNMS:Name=Pollerd
[INFO] Invoking init on object OpenNMS:Name=EnhancedLinkd
[1316.313s][info][gc] GC(65) Pause Young (Normal) (G1 Evacuation Pause) 890M->180M(1208M) 7.968ms
[1316.956s][info][gc] GC(66) Pause Young (Normal) (G1 Evacuation Pause) 890M->181M(1208M) 6.118ms
[1317.535s][info][gc] GC(67) Pause Young (Normal) (G1 Evacuation Pause) 891M->182M(1208M) 6.255ms
[INFO] Invocation init successful for MBean OpenNMS:Name=EnhancedLinkd
[INFO] Invoking init on object OpenNMS:Name=Collectd
[INFO] Invocation init successful for MBean OpenNMS:Name=Collectd
[INFO] Invoking init on object OpenNMS:Name=Discovery
[INFO] Invocation init successful for MBean OpenNMS:Name=Discovery
[INFO] Invoking init on object OpenNMS:Name=Vacuumd
[INFO] Invocation init successful for MBean OpenNMS:Name=Vacuumd
[INFO] Invoking init on object OpenNMS:Name=EventTranslator
[INFO] Invocation init successful for MBean OpenNMS:Name=EventTranslator
[INFO] Invoking init on object OpenNMS:Name=PassiveStatusd
[INFO] Invocation init successful for MBean OpenNMS:Name=PassiveStatusd
[INFO] Invoking init on object OpenNMS:Name=Statsd
[INFO] Invocation init successful for MBean OpenNMS:Name=Statsd
[INFO] Invoking init on object OpenNMS:Name=Provisiond
[1318.977s][info][gc] GC(68) Pause Young (Concurrent Start) (Metadata GC Threshold) 727M->184M(1208M) 10.523ms
[1318.977s][info][gc] GC(69) Concurrent Cycle
[1319.077s][info][gc] GC(69) Pause Remark 194M->194M(1208M) 7.049ms
[1319.122s][info][gc] GC(69) Pause Cleanup 196M->196M(1208M) 0.174ms
[1319.125s][info][gc] GC(69) Concurrent Cycle 147.305ms
[INFO] Invocation init successful for MBean OpenNMS:Name=Provisiond
[INFO] Invoking init on object OpenNMS:Name=Reportd
[INFO] Invocation init successful for MBean OpenNMS:Name=Reportd
[INFO] Invoking init on object OpenNMS:Name=Ackd
[INFO] Invocation init successful for MBean OpenNMS:Name=Ackd
[INFO] Invoking init on object OpenNMS:Name=JettyServer
[ERROR] An error occurred invoking operation init on MBean OpenNMS:Name=JettyServer
javax.management.RuntimeMBeanException: org.springframework.beans.factory.access.BootstrapException: Unable to return specified BeanFactory instance: factory key [jettyServerContext], from group with resource name [classpath*:beanRefContext.xml]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jettyServerContext' defined in URL [jar:file:/opt/opennms/lib/opennms-jetty-30.0.0.jar!/beanRefContext.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'daemon' defined in class path resource [META-INF/opennms/applicationContext-jettyServer.xml]: Invocation of init method failed; nested exception is java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

wes