Issue after 23 to 24 upgrade on CentOS

upgrade
#1

OS: CentOS Linux release 7.6.1810 (Core)

The upgrade succeeded as far as the logs go and the config files pass the config-tester after I merged my changes and ran runjava -s . I can’t get OpenNMS to start.

Running /opt/opennms/bin/opennms -vf start ends with the following:

[28.825s][info][gc] GC(33) Pause Young (Normal) (G1 Evacuation Pause) 278M->142M(332M) 37.247ms
An error occurred while attempting to start the "OpenNMS:Name=Manager" service (class org.opennms.netmgt.vmmgr.Manager).  Shutting down and exiting.
javax.management.InstanceAlreadyExistsException: OpenNMS:Name=Manager
        at java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at org.opennms.netmgt.vmmgr.Invoker.instantiateClasses(Invoker.java:129)
        at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:152)
        at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:94)
        at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173)
        at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:527)
        at java.base/java.lang.Thread.run(Thread.java:834)
#2

Can you please check the logs for exceptions and share the exceptions here? Here are the ones I would look into:

  • manager.log: it shows which daemons are triggered to startup. If there is one daemon which fails, it is an indicator to look in more detail in the corresponding daemon log file
  • uncatorized.log: Might not exist, it is created if unexpected errors occur
#3

I am not seeing a failed daemon. Uncategorized.log doesn’t exist. Here is the error. It won’t let me post the whole log as it is too many characters and I can’t attach it because it is not a picture.

2019-05-06 07:43:53,321 DEBUG [Main] o.o.n.v.Invoker: loading class org.opennms.netmgt.vmmgr.Manager
2019-05-06 07:43:53,321 DEBUG [Main] o.o.n.v.Invoker: create new instance of org.opennms.netmgt.vmmgr.Manager
2019-05-06 07:43:53,321 DEBUG [Main] o.o.n.v.Invoker: registering mbean instance OpenNMS:Name=Manager
2019-05-06 07:43:53,321 ERROR [Main] o.o.n.v.Invoker: An error occurred loading the mbean OpenNMS:Name=Manager of type org.opennms.netmgt.vmmgr.Manager
javax.management.InstanceAlreadyExistsException: OpenNMS:Name=Manager
	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320) ~[?:?]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) ~[?:?]
	at org.opennms.netmgt.vmmgr.Invoker.instantiateClasses(Invoker.java:129) [org.opennms.core.daemon-24.0.0.jar:?]
	at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:152) [org.opennms.core.daemon-24.0.0.jar:?]
	at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:94) [org.opennms.core.daemon-24.0.0.jar:?]
	at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173) [org.opennms.core.daemon-24.0.0.jar:?]
	at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150) [org.opennms.core.daemon-24.0.0.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:527) [opennms_bootstrap.jar:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
#4

Interesting further up:

2019-05-06 08:37:57,182 DEBUG [Main] o.o.n.v.Starter: Beginning startup
2019-05-06 08:37:57,191 DEBUG [Main] o.o.n.c.ServiceConfigFactory: ServiceConfigFactory.init: config file path /opt/opennms/etc/service-configuration.xml
2019-05-06 08:37:58,296 DEBUG [Main] o.o.n.v.Invoker: loading class org.opennms.netmgt.vmmgr.Manager
2019-05-06 08:37:58,299 DEBUG [Main] o.o.n.v.Invoker: create new instance of org.opennms.netmgt.vmmgr.Manager
2019-05-06 08:37:58,299 DEBUG [Main] o.o.n.v.Invoker: registering mbean instance OpenNMS:Name=Manager
2019-05-06 08:37:58,300 DEBUG [Main] o.o.n.v.Invoker: loading class org.opennms.netmgt.vmmgr.Manager
2019-05-06 08:37:58,300 DEBUG [Main] o.o.n.v.Invoker: create new instance of org.opennms.netmgt.vmmgr.Manager

It seems to be following along with the service start up and then tries doing it again:

2019-05-06 08:37:58,355 DEBUG [Main] o.o.n.v.Invoker: registering mbean instance OpenNMS:Name=Telemetryd
2019-05-06 08:37:58,355 DEBUG [Main] o.o.n.v.Invoker: setting attribute LoggingPrefix
2019-05-06 08:37:58,355 DEBUG [Main] o.o.n.v.Invoker: setting attribute SpringContext
2019-05-06 08:37:58,355 DEBUG [Main] o.o.n.v.Invoker: loading class org.opennms.netmgt.vmmgr.Manager
2019-05-06 08:37:58,355 DEBUG [Main] o.o.n.v.Invoker: create new instance of org.opennms.netmgt.vmmgr.Manager
2019-05-06 08:37:58,355 DEBUG [Main] o.o.n.v.Invoker: registering mbean instance OpenNMS:Name=Manager
2019-05-06 08:37:58,355 ERROR [Main] o.o.n.v.Invoker: An error occurred loading the mbean OpenNMS:Name=Manager of type org.opennms.netmgt.vmmgr.Manager
javax.management.InstanceAlreadyExistsException: OpenNMS:Name=Manager

The error occurs after it finishes the list of services and appears to be trying again. Hence the "already exists" error. Of course all of the services following that error are also errored out with the same error message:

2019-05-06 08:37:58,376 DEBUG [Main] o.o.n.v.Invoker: loading class org.opennms.netmgt.daemon.SimpleSpringContextJmxServiceDaemon
2019-05-06 08:37:58,377 DEBUG [Main] o.o.n.v.Invoker: create new instance of org.opennms.netmgt.daemon.SimpleSpringContextJmxServiceDaemon
2019-05-06 08:37:58,377 DEBUG [Main] o.o.n.v.Invoker: registering mbean instance OpenNMS:Name=Telemetryd
2019-05-06 08:37:58,377 ERROR [Main] o.o.n.v.Invoker: An error occurred loading the mbean OpenNMS:Name=Telemetryd of type org.opennms.netmgt.daemon.SimpleSpringContextJmxServiceDaemon
javax.management.InstanceAlreadyExistsException: OpenNMS:Name=Telemetryd
#5

I think I had a bad config somewhere. I removed all of the configs from the /opt/opennms/etc directory and then put all of the pristine configs in and copied the ones I had modified back into the directory.

It all came back up. Not sure which config was bad though.

1 Like