How to Upgrade Apache Karaf

how-to
osgi
karaf

#1

This guide gives some guidelines on how to best update Apache Karaf.
Please note, that this is probably nothing anyone besides the OpenNMS Development Team should do.
This is only a very short list of hints, and not of any kind a full “Upgrade Guide”. However, I couldn’t finde any documentation anywhere at all, so here we go:

Update Root POM

First the karafVersion property in the root pom.xml should be updated with the new Karaf Version, e.g. 4.2.3.

Afterwards I usually trigger a build, just to have the latest artifacts locally in my repository.

Update feature definitions

In order to embed Apache Karaf (and in detail the OSGI Http Service`) with OpenNMS (and in detail with Jetty), we have to update the default feature.xml files from Apache Karaf, to disable/overwrite some http-services.

In detail this means. Copy the following files from your local maven repository (see previous step) to container/features/src/main/resources. They are all located in ~/.m2/repository/org/apache/karaf/features

  • cp ~/.m2/repository/org/apache/karaf/features/spring/4.2.3/spring-4.2.3-features.xml ~/dev/opennms/container/features/src/main/resources/karaf/spring.xml
  • cp ~/.m2/repository/org/apache/karaf/features/spring-legacy/4.2.3/spring-legacy-4.2.3-features.xml ~/dev/opennms/container/features/src/main/resources/karaf/spring-legacy.xml
  • cp ~/.m2/repository/org/apache/karaf/features/standard/4.2.3/standard-4.2.3-features.xml ~/dev/opennms/container/features/src/main/resources/karaf/standard.xml

As we “hack” those, simply git diff and see what has actually changed and apply our hacks accordingly.
This mostly means, don’t install pax-http and all related bundles.

Be also aware that some of the versions referenced in standard.xml are also defined in some pom.xml files. Ensure that they also match versions. Here are some examples:

  • <paxExamVersion>
  • <paxWebVersion>
  • there may be more

Download Apache Karaf Distribution

Download the Apache Karaf Distribution from https://karaf.apache.org to which to update, e.g. 4.2.3.
This is required for all the configuration files, which must be updated as well.

Update Configuration files

We overwrite a lot of default configuration files to bend Karaf to our needs.
In order to not miss anything changed there, simply copy over all default configuration files from the downloaded Apache Karaf Distribution folder.

The files which must be copied over are located in

  • ~/dev/opennms/container/karaf/src/main/filtered-resources/etc
  • ~/dev/opennms/container/shared/src/filtered-resources/etc
  • ~/dev/opennms/features/container/sentinel/src/filtered-resources/etc
  • ~/dev/opennms/features/container/minion/src/filtered-resources/etc

Afterwards verify what actually was added by us with git diff or your IDE of choice and apply them accordingly.

Build OpenNMS and hope verify that everything woks

Now do a full build of OpenNMS (./clean.pl && ./compile.pl -DskipTests && ./assemble.pl -p dir -DskipTests).

If successful, start OpenNMS and see if it comes up.
Some basic checks:

If everything returned a 200 OK with actual content, you are good to do some manual testing.
Good candiates are:

  • Login
  • Verify Topology Map
  • Verify JMX Configurator UI
  • Verify Plugin Manager
  • Verify Rest (see above)
  • Login to Karaf Shell

If that all worked, verify all logs and check for Exceptions.
Each Exception should be investigated and must be solved.

Please keep in mind, that you actually should rename your local maven repository to ensure your system directory was build correctly. You may skip it here, but Bamboo may fail then instead

Verifying Minion

Do a ./runInPlace.sh in features/minion and verify that the minion is coming up.
A health:check will most likely tell you if it is at least working.

Verifying Sentinel

Do a ./runInPlace.sh in features/sentinel and verify that the sentinel is coming up.
As sentinel is not installing any features by default, install at least sentinel-core, maybe even sentinel-persistence and run a health:check to see if it should work.

Bamboo

If until now everything worked, push your changes and Bamboo will probably spit out some test failures. Fix them and you are good to go.

Troubleshooting

When updating Karaf (especially major versions) a lot of things will break.
Most likely you will encounter dependency related issues.
So here is how to debug:

  • Verify all installed bundles in karaf shell :
    list. If nothing is shown, you can also try a list -t 0 to see in more details what is wrong. log:tail will show any issues.
  • Verify startup.properties and see, that the versions actually match the startup.properties of the distribution. Keep in mind, we also install additional libraries, such as mina.jar
  • Verify system directory. If a bundle cannot be resolved it is probably not in the system directory. If so, some dependencies are probably overritten by OpenNMS (see “Update Feature Definitions” and verify everything was updated correctly)
  • Wondering what felixVersion is used? Check the framework/framework-features.xml in ~/.m2/repository/org/apache/karaf/features
  • If something is not working as expected, check how it would behave in a plain Apache Karaf Container or maybe in the Container before upgrade.
  • Cannot figure out some problems, try isolating it. Maybe even create a plain maven project.
  • Debug
  • Debug even more
  • Swear a lot. Believe me it helps (-:
  • (╯’□’)╯︵ ┻━┻

Recent Upgrades

  • 4.1.5 to 4.2.3 PR