Karaf shell fails to start after upgrade to Horizon 24

I’ve got a test OpenNMS instance which I’m testing the Jira Plugin. This was originally a 23.0.4 OpenNMS instance, But I upgraded it to 24.0.0, and now the karaf shell fails to start.

I have another test OpenNMS instance that does not have the jira plugin, which the karaf shell works, it is only the Instance where I am testing Jira, where karaf shell is not starting.

the users.properties file is correct with:

admin = admin,_g_:admingroup
_g_\:admingroup = group,admin,manager,viewer,systembundles,ssh

the karaf.log contains the following:

2019-05-10T03:30:41,427 | ERROR | activator-1-thread-2 | BootFeaturesInstaller            | 11 - 
org.apache.karaf.features.core - 4.2.3 | Error installing boot feature repository mvn:org.opennms.karaf/opennms/23.0.4/xml/features
java.lang.RuntimeException: Error resolving artifact org.opennms.karaf:opennms:xml:features:23.0.4: [Could not find artifact org.opennms.karaf:opennms:xml:features:23.0.4] : mvn:org.opennms.karaf/opennms/23.0.4/xml/features
at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:116) ~[11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.service.RepositoryImpl.<init>(RepositoryImpl.java:50) ~[11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.service.RepositoryCacheImpl.create(RepositoryCacheImpl.java:51) ~[11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:380) ~[11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:375) ~[11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.addRepositories(BootFeaturesInstaller.java:123) [11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:93) [11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.start(BootFeaturesInstaller.java:87) [11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.features.internal.osgi.Activator.doStart(Activator.java:196) [11:org.apache.karaf.features.core:4.2.3]
at org.apache.karaf.util.tracker.BaseActivator.run(BaseActivator.java:292) [11:org.apache.karaf.features.core:4.2.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.io.IOException: Error resolving artifact org.opennms.karaf:opennms:xml:features:23.0.4: [Could not find artifact org.opennms.karaf:opennms:xml:features:23.0.4]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.configureIOException(AetherBasedResolver.java:803) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:774) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:555) ~[?:?]
at org.ops4j.pax.url.mvn.internal.Connection.getInputStream(Connection.java:123) ~[?:?]
at java.net.URL.openStream(URL.java:1117) ~[?:?]
at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:113) ~[?:?]
... 14 more
Suppressed: shaded.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.opennms.karaf:opennms:xml:features:23.0.4
	at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:403) ~[?:?]
	at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215) ~[?:?]
	at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192) ~[?:?]
	at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247) ~[?:?]
	at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767) ~[?:?]
	at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
	at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
	at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
	at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:555) ~[?:?]
	at org.ops4j.pax.url.mvn.internal.Connection.getInputStream(Connection.java:123) ~[?:?]
	at java.net.URL.openStream(URL.java:1117) ~[?:?]
	at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:113) ~[11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.service.RepositoryImpl.<init>(RepositoryImpl.java:50) ~[11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.service.RepositoryCacheImpl.create(RepositoryCacheImpl.java:51) ~[11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:380) ~[11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:375) ~[11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.service.BootFeaturesInstaller.addRepositories(BootFeaturesInstaller.java:123) [11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.service.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:93) [11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.service.BootFeaturesInstaller.start(BootFeaturesInstaller.java:87) [11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.features.internal.osgi.Activator.doStart(Activator.java:196) [11:org.apache.karaf.features.core:4.2.3]
	at org.apache.karaf.util.tracker.BaseActivator.run(BaseActivator.java:292) [11:org.apache.karaf.features.core:4.2.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: shaded.org.eclipse.aether.resolution.ArtifactResolutionException: Error resolving artifact org.opennms.karaf:opennms:xml:features:23.0.4
at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:413) ~[?:?]
at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215) ~[?:?]
at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192) ~[?:?]
at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:555) ~[?:?]
at org.ops4j.pax.url.mvn.internal.Connection.getInputStream(Connection.java:123) ~[?:?]
at java.net.URL.openStream(URL.java:1117) ~[?:?]
at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:113) ~[?:?]
... 14 more
2019-05-10T03:30:41,655 | INFO  | features-3-thread-1 | FeaturesServiceImpl              | 11 - org.apache.karaf.features.core - 4.2.3 | No deployment change.
2019-05-10T03:30:41,672 | INFO  | features-3-thread-1 | FeaturesServiceImpl              | 11 - org.apache.karaf.features.core - 4.2.3 | Done.
2019-05-10T03:30:41,713 | INFO  | features-3-thread-1 | FeaturesServiceImpl              | 11 - org.apache.karaf.features.core - 4.2.3 | No deployment change.
2019-05-10T03:30:41,714 | INFO  | features-3-thread-1 | FeaturesServiceImpl              | 11 - org.apache.karaf.features.core - 4.2.3 | Done.
2019-05-10T03:30:42,335 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final
May 10, 2019 3:48:40 AM org.apache.karaf.main.Main$KarafLockCallback lockAcquired
INFO: Lock acquired. Setting startlevel to 100
2019-05-10T03:48:43,605 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final
2019-05-10T03:53:25,563 | INFO  | fileinstall-/opt/opennms/etc | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | Updating configuration from org.apache.karaf.features.cfg
2019-05-10T03:54:31,605 | INFO  | fileinstall-/opt/opennms/etc | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | Updating configuration from org.apache.karaf.features.cfg
May 10, 2019 3:57:11 AM org.apache.karaf.main.Main$KarafLockCallback lockAcquired
INFO: Lock acquired. Setting startlevel to 100
2019-05-10T03:57:14,484 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final
May 10, 2019 4:37:40 AM org.apache.karaf.main.Main$KarafLockCallback lockAcquired
INFO: Lock acquired. Setting startlevel to 100
2019-05-10T04:37:42,648 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final
May 10, 2019 5:00:27 AM org.apache.karaf.main.Main$KarafLockCallback lockAcquired
INFO: Lock acquired. Setting startlevel to 100
2019-05-10T05:00:30,050 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final
May 10, 2019 5:32:24 AM org.apache.karaf.main.Main$KarafLockCallback lockAcquired
INFO: Lock acquired. Setting startlevel to 100
2019-05-10T05:32:27,182 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final
2019-05-10T05:36:13,066 | INFO  | fileinstall-/opt/opennms/etc | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | Updating configuration from org.apache.karaf.features.cfg
May 10, 2019 5:37:25 AM org.apache.karaf.main.Main$KarafLockCallback lockAcquired
INFO: Lock acquired. Setting startlevel to 100
2019-05-10T05:37:28,138 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final

the errors regarding v23.0.4 features I’ve resolved by removing the pluginManifestData.xml as this also resolved the same error on the server that does not have the jira plugin installed. but on the server that has the jira plugin, the karaf shell is not starting.

netstat shows that port 8101 is not being listened on.

Any help regarding how to resolve this would be much appreciated.

Regards,
John

Bump!

Anyone have any ideas on this?

Hi John. Not sure v24 will support a v23.x plugin but, you need to specify the real repository location in the org.ops4j.pax.url.mvn.cfg file under /etc . If you drill into the 23.0.4 version in maven, it informs of " Note : this artifact it located at OpenNMS repository (http://repo.opennms.org/maven2/)".

So set the following line in the above cfg file: org.ops4j.pax.url.mvn.repositories=http://repo.opennms.org/maven2/
I think it deploys commented out with no value.

Hi,

it looks to me like you might still need to upgrade the opennms-plugin-ticketer-jira package to v24, or you have a .cfg floating around that is improperly merged or referencing an old version of the plugin instead of v24.

[root@opennms opennms]# yum info opennms-plugin-ticketer-jira.noarch
Available Packages
Name        : opennms-plugin-ticketer-jira
Arch        : noarch
Version     : 24.0.0
# yum info opennms-plugin-ticketer-jira.noarch
Loaded plugins: ulninfo, versionlock
Installed Packages
Name        : opennms-plugin-ticketer-jira
Arch        : noarch
Version     : 24.0.0
Release     : 1
Size        : 8.3 M
Repo        : installed
From repo   : opennms-repo-stable-common
Summary     : JIRA Ticketer Plugin
URL         : http://www.opennms.org/
License     : LGPL/AGPL
Description : The JIRA ticketer plugin provides the ability to automatically create JIRA
            : issues from OpenNMS alarms.
            : 
            : This is an OpenNMS build from Git.  For a complete log, see:
            : https://github.com/OpenNMS/opennms/commit/d697aee0117681ceda94a3392b25d3a8aa79c4db

@mwright12, the artifact errors were caused by the file pluginManifestData.xml containing refences to the 23.0.4 plugin. I have manually updated that file so it is referencing 24.0.0, however, i’m not sure that update was correct - opennms startup nolonger has those errors in the karaf.log file. Infact, not much in there at all.:

2019-05-21T22:53:17,453 | INFO  | fileinstall-/opt/opennms/etc | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | Updating configuration from org.opennms.features.topology.app.icons.bsm.cfg
2019-05-21T22:54:49,485 | INFO  | fileinstall-/opt/opennms/etc | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | Updating configuration from org.opennms.features.topology.app.icons.application.cfg
May 21, 2019 11:00:57 PM org.apache.karaf.main.Main$KarafLockCallback lockAcquired
INFO: Lock acquired. Setting startlevel to 100
2019-05-21T23:01:00,360 | INFO  | Main             | Version                          |  -  -  | HV000001: Hibernate Validator 4.3.2.Final

The 2 files referenced there are 2 that had minor differences with the etc-pristine versions when I compared, which I copied in.

is there anyway to increase logging for karaf? there is nothing for karaf in log4j2.xml

Logging level can be increased by changing org.ops4j.pax.logging.cfg

Now I’m seeing this regularly in the karaf.log:

2019-05-21T23:43:44,227 | DEBUG | RMI TCP Connection(4)-127.0.0.1 | tcp                              |  -  -  | RMI TCP Connection(4)-127.0.0.1: (port 20303) op = 80
2019-05-21T23:43:44,227 | DEBUG | RMI TCP Connection(4)-127.0.0.1 | loader                           |  -  -  | RMI TCP Connection(4)-127.0.0.1: name = "[Ljava.lang.String;", codebase = ""
2019-05-21T23:43:44,229 | DEBUG | RMI TCP Connection(4)-127.0.0.1 | misc                             |  -  -  | JMXPluggableAuthenticator::authenticate
java.lang.SecurityException: Authentication failed! No LoginModules configured for opennms
        at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:227) ~[?:?]
        at com.sun.jmx.remote.security.JMXPluggableAuthenticator.<init>(JMXPluggableAuthenticator.java:135) ~[?:?]
        at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:229) ~[?:?]
        at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:231) ~[?:?]
        at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:198) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359) ~[?:?]
        at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:?]
        at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:?]
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562) ~[?:?]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796) ~[?:?]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.security.auth.login.LoginException: No LoginModules configured for opennms
        at javax.security.auth.login.LoginContext.init(LoginContext.java:261) ~[?:?]
        at javax.security.auth.login.LoginContext.<init>(LoginContext.java:412) ~[?:?]
        at com.sun.jmx.remote.security.JMXPluggableAuthenticator.<init>(JMXPluggableAuthenticator.java:104) ~[?:?]
        ... 20 more

Note: This particular opennms instance was initially built as V23.0.4 when the jira plugin was installed, and I was trying to get the plugin to work. Karaf shell was working at that time. Karaf shell stopped working following the upgrade to V24.

I built a new OpenNMS server from scratch with 24.0.0, and installed the jira plugin.

I added my settings to jira.properties, and restarted opennms.

After this, I could connect to the karaf shell, and the jira:verify command verified the configuration was correct, and was able to connect to the jira instance…

I then added the following to the opennms.properties file:

opennms.ticketer.plugin=org.opennms.netmgt.ticketd.OSGiBasedTicketerPlugin

and jira-troubleticketer to the featuresBoot property in the ${OPENNMS_HOME}/etc/org.apache.karaf.features.cfg

I then restarted opennms, and tried connecting to karaf shell so I could run

feature:install jira-troubleticketer

However, I found that I could not connect to the karaf shell, and noted that port 8101 was no longer being listened to.

I reverted the 2 files back to their etc-pristine versions and restarted opennms, but karaf shell is still not accessible.

Any help would be much appreciated!

NOTE: I don’t have the output of the successful jira:verify command, as it has since scrolled out of the terminal history, and I can no longer connect to karaf shell to re-run :frowning:

Regards,
John