OpenNMS is not starting

I’ve hit a wall twice with installing and learning OpenNMS. I’ve installed what I believe to be the current version of OpenNMS and I’ve been following along with the videos that Tarus produced. When I move from lesson 4-1 to 4-2 a restart of OpenNMS is required. And when I attempt to restart OpenNMS, the status shows start_pending and then OpenNMS does not start. Instead I get this:

[root@nms etc]# opennms -v status
Could not connect to the OpenNMS JVM (OpenNMS might not be running or could be starting up or shutting down): Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)]
opennms is stopped

I’ve searched using Google for the error, and I can find it – but the problems don’t look similar to what I am seeing. I’m not a *nix person, but I am able to follow directions, and I have an idea of what I am doing and Google helps fill in the gaps. But I don’t know what this error is telling me.

The first time this occurred, I thought it had to do with downloading the notifd-configuration.xml and NotificationCommands.xml files. The formatting of these on my server versus the versions in the opennms.org/Training are different. So this time I used cut and paste for only the sections that are different – thus preserving the original files. But backing out those changes hasn’t allowed the system to be restarted. Which makes me wonder what is breaking the system. Am I finding some bug, or is something else breaking this.

I apologize if this question has been asked before. I’m new to OpenNMS so any help would be appreciated, otherwise, I’m reinstalling for the third time.

Thanks.

Dave

Hey Dave!

Which operation system do you use?

How did you install OpenNMS? Did you follow the official install documentation?

I was looking to learn about OpenNMS, so I’ve been watching the videos and following along with Tarus. He’s using version 18 in the videos, I think I am on version 23. Most of the differences I’ve noticed in the GUI have been cosmetic. That being said, I’m using CentOS 7 and I’ve patched to current. And the directions I’m using have been included in the videos.

Dave

Please check some things:

ps aux | grep java
-> It should not find any Java processes (if ONMS is the only Java application you are running on this server)

Also you can check systemctl status opennms and see if it’s currently running.
Use systemctl stop opennms to stop it.

The mallet method:
If it’s running just kill it using pkill java as root.

Then try systemctl start opennms.

I’m not certain if java is running, or the grep process is returning the answer. OpenNMS is not running. When I issue opennms -v status I get the java error. When I issue the command #systemctl start (restart) opennms I get the “start_pending” messages, so I know it is trying to start. But then checking the status at some later point, I’m back to the error.

Here’s the text to go along with the questions you asked.

[root@nms ~]# ps aux | grep java
root      6130  0.0  0.0 112708   980 pts/0    S+   14:08   0:00 grep --color=auto java
[root@nms ~]# systemctl status opennms
● opennms.service - OpenNMS server
   Loaded: loaded (/usr/lib/systemd/system/opennms.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2019-03-14 09:10:17 EDT; 4h 58min ago
  Process: 22664 ExecStop=/opt/opennms/bin/opennms stop (code=exited, status=0/SUCCESS)
  Process: 13106 ExecStart=/opt/opennms/bin/opennms -Q start (code=exited, status=0/SUCCESS)
 Main PID: 20746 (code=exited, status=0/SUCCESS)

Mar 14 09:09:42 nms.hacc.edu systemd[1]: Starting OpenNMS server...
Mar 14 09:09:55 nms.hacc.edu opennms[13106]: Starting OpenNMS: (not waiting for startup) [  OK  ]
Mar 14 09:09:55 nms.hacc.edu systemd[1]: Started OpenNMS server.
Mar 14 09:10:16 nms.hacc.edu opennms[22664]: Stopping OpenNMS: Trying to stop OpenNMS but it's already stopped.
Mar 14 09:10:16 nms.hacc.edu opennms[22664]: [  OK  ]

Thanks.

Dave

Chances are it’s puking on some invalid config. Check /var/log/opennms/output.log and you should be able to identify the culprit (file name, line number) from the exception.

You can also check the configs using /opt/opennms/bin/config-tester

Here is the config-tester output. I don’t see anything with an “error”.

[root@nms opennms]# /opt/opennms//bin/config-tester v
14:58:09.408 [Main] INFO  org.opennms.netmgt.config.tester.ConfigTesterFilterDao - Initializing ConfigTesterFilterDao
14:58:09.546 [Main] INFO  org.springframework.context.support.ClassPathXmlApplicationContext - Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@144c5e45: startup date [Thu Mar 14 14:58:09 EDT 2019]; root of context hierarchy
14:58:09.603 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.core.soa-23.0.3.jar!/beanRefContext.xml]
14:58:09.844 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-ackd-23.0.3.jar!/beanRefContext.xml]
14:58:09.860 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-alarmd-23.0.3.jar!/beanRefContext.xml]
14:58:09.873 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-asterisk-23.0.3.jar!/beanRefContext.xml]
14:58:09.886 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-config-23.0.3.jar!/beanRefContext.xml]
14:58:09.897 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-config-tester-23.0.3.jar!/beanRefContext.xml]
14:58:09.907 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-correlator-23.0.3.jar!/beanRefContext.xml]
14:58:09.917 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-dao-23.0.3.jar!/beanRefContext.xml]
14:58:09.927 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-jetty-23.0.3.jar!/beanRefContext.xml]
14:58:09.937 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-provisiond-23.0.3.jar!/beanRefContext.xml]
14:58:09.946 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-reportd-23.0.3.jar!/beanRefContext.xml]
14:58:09.955 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-reporting-23.0.3.jar!/beanRefContext.xml]
14:58:09.963 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms-services-23.0.3.jar!/beanRefContext.xml]
14:58:09.976 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/opennms_system_report.jar!/beanRefContext.xml]
14:58:09.985 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.core.daemon-23.0.3.jar!/beanRefContext.xml]
14:58:09.994 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.bsm.daemon-23.0.3.jar!/beanRefContext.xml]
14:58:10.003 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.discovery-23.0.3.jar!/beanRefContext.xml]
14:58:10.011 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.events.daemon-23.0.3.jar!/beanRefContext.xml]
14:58:10.020 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.events.syslog-23.0.3.jar!/beanRefContext.xml]
14:58:10.028 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.events.traps-23.0.3.jar!/beanRefContext.xml]
14:58:10.036 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.poller.remote-23.0.3.jar!/beanRefContext.xml]
14:58:10.045 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.reporting.availability-23.0.3.jar!/beanRefContext.xml]
14:58:10.053 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.reporting.core-23.0.3.jar!/beanRefContext.xml]
14:58:10.061 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.reporting.jasper-reports-23.0.3.jar!/beanRefContext.xml]
14:58:10.070 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.reporting.repository-23.0.3.jar!/beanRefContext.xml]
14:58:10.078 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.telemetry.daemon-23.0.3.jar!/beanRefContext.xml]
14:58:10.086 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/opt/opennms/lib/org.opennms.features.ticketing.daemon-23.0.3.jar!/beanRefContext.xml]
14:58:10.186 [Main] INFO  org.springframework.context.support.ClassPathXmlApplicationContext - Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@47de476: startup date [Thu Mar 14 14:58:10 EDT 2019]; root of context hierarchy
14:58:10.188 [Main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [META-INF/opennms/applicationContext-configTester.xml]
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:525)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: config 'v' is not a known config name
        at org.opennms.netmgt.config.tester.ConfigTester.checkConfigNameValid(ConfigTester.java:129)
        at org.opennms.netmgt.config.tester.ConfigTester.testConfig(ConfigTester.java:82)
        at org.opennms.netmgt.config.tester.ConfigTester.testConfig(ConfigTester.java:205)
        at org.opennms.netmgt.config.tester.ConfigTester.main(ConfigTester.java:194)
        ... 6 more
[root@nms opennms]#

In the log files, I do see this error:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tcpListener' defined in class path resource [META-INF/opennms/applicationContext-eventDaemon.xml]: Invocation of init method failed; nested exception is java.nio.channels.UnresolvedAddressException
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267)
        ... 44 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tcpListener' defined in class path resource [META-INF/opennms/applicationContext-eventDaemon.xml]: Invocation of init method failed; nested exception is java.nio.channels.UnresolvedAddressException

and this was the last line of that log file…

2019-03-14 14:12:14,882 ActiveMQ ShutdownHook WARN Unable to register Log4j shutdown hook because JVM is shutting down. Using SimpleLogger

You’re issuing the command: /opt/opennms//bin/config-tester v

and config-tester is returning : java.lang.IllegalArgumentException: config 'v' is not a known config name

Did you change any configs regarding to tcpListener?

dino: oops that was an error. The parameter was to be a -v. If there is no parameter included, the file line is (after showing it loads a bunch of other XML beans):

path resource [META-INF/opennms/applicationContext-configTester.xml]
Invalid usage: too few arguments. Use the ‘-h’ option for help.

mfuhrmann:

Where would I change anything with regards to the tcpListener in OpenNMS?

Thanks.

I don"t know. Did you change config files?

Try to grep it: grep tcpListener /opt/opennms/etc/* - R

I never could figure out what happened. I certainly didn’t change anything that I could recall regarding the tcpListener. I’ve decided to try installing once again, this time using Ubuntu. I’ll see what happens…

Thank you all.

Dave

Dave,

I struggled with an ONMS install on CentOS 7 as well. I followed much of the same path you did (online docs and Tarus’ YouTube tutorials). The one thing I tried (based on a comment Tarus made a few times) is to start with a minimal install of CentOS (i.e, no desktop GUI or other packages.) I updated the OS, then installed ONMS and it ran. I then added a few more things like a minimal install of Gnome and a browser. I haven’t had a problem since. It seems like the full install of CentOS with Gnome (and other packages) creates a Java conflict with ONMS.

Hope this helps,
Danny

Danny:

What I found was in the eventd-configuration.xml file:

Tarus specifically talks about the use of the TCPAddress and UDPAddress, and he states that the only two values you can use are 127.0.0.1 or the *. Depends on whether or not you will allow foreign devices to send alerts to OpenNMS. I’m not sure that I want devices sending alerts to OpenNMS, but I wasn’t sure that I didn’t. But he talks also about limiting which devices can send by using the firewall. So I changed the setting from 127.0.0.1 to *. And when I restarted OpenNMS (not just a service), it won’t load. Changing the values back to 127.0.0.1, OpenNMS works fine. So I’m not sure if changes were made between the version he uses in the videos (which I think is 18) and the current version. But the * in that file definitely breaks OpenNMS. And it was only after I reinstalled for about the fifth time - and purposely set out to discover what was breaking OpenNMS that I discovered this.

Dave

The problem you will get having a Java Runtime Engine instead of a Java Development Kit installed is described here: Installing OpenNMS fails with ClassNotFoundException: com.sun.tools.attach.AttachNotSupportedException.

The main issue here is the Java dependency is satisfied with the JRE which comes with CentOS 7 and OpenNMS will not install the JDK and use what’s there which causes the problem.

1 Like

Grrr. I think this has changed since I did the videos. You will want to use “0.0.0.0” instead of “*”.

I blame @indigo

Tarus:

Thanks for that confirmation! I started wondering about that, and whether other combinations might work. But I do appreciate the response - and after watching those videos over and over I’m connecting the dots and really beginning to understand what I’m doing with this.

Thanks!

Dave

@Sortova Is there a way you could note that in the video? Add a subtitle under that section, maybe? My youtube-fu isn’t great. It might help a lot of people to not end up in that situation, though.

Once Horizon 24 comes out I plan to redo all of them with the new UI

2 Likes