Horizon 27.0.3 crashes with stack traces

I have a new install of Horizon 27.0.3 (CentOS 7, 4 CPU, 32 GB, jdk1.8) but it won’t stay running longer than a couple of minutes, and stack traces with the following error:

2021-01-06T02:56:08,306 | ERROR | activator-1-thread-2 | BootFeaturesInstaller | 11 - org.apache.karaf.features.core - 4.2.6 | Error installing boot feature repository mvn:org.opennms.karaf/opennms/27.0.2/xml/spring-legacy
java.lang.RuntimeException: Error resolving artifact org.opennms.karaf:opennms:xml:spring-legacy:27.0.2: [Could not find artifact org.opennms.karaf:opennms:xml:spring-legacy:27.0.2] : mvn:org.opennms.karaf/opennms/27.0.2/xml/spring-legacy
at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:116) ~[11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.service.RepositoryImpl.(RepositoryImpl.java:50) ~[11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.service.RepositoryCacheImpl.create(RepositoryCacheImpl.java:51) ~[11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:380) ~[11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:375) ~[11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.addRepositories(BootFeaturesInstaller.java:123) [11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:93) [11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.start(BootFeaturesInstaller.java:87) [11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.features.internal.osgi.Activator.doStart(Activator.java:196) [11:org.apache.karaf.features.core:4.2.6]
at org.apache.karaf.util.tracker.BaseActivator.run(BaseActivator.java:292) [11:org.apache.karaf.features.core:4.2.6]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]

So, I downgraded to 27.0.2 and the stack traces have gone away but the load average on the host drives way up to 50+ soon after starting up, and now some of the Horizon GUI pages end before rendering (the Resource Graph is particularly problematic). I’d tried head sizes of 4GB, 8GB, 10GB but it’s the same result.

I need to resolve both issues ultimately, but we can focus on the first one for now. Any ideas?

The error message carries 27.0.2 version numbers instead 27.0.3. it seems to me the upgrade broke things. You can run the config diff command in the bin directory. In doubt take the config files from the pristine directory in share.