Opennms.web.base-url not always working

Problem:
We have been running OpenNMS behind an Apache reverse proxy for many years. After the last update, from 29.0.6 to 30.0.2, we have been experiencing some weird inconsistencies in the URL formatting behaviour.

In opennms.properties, we have set opennms.web.base-url = https://%x%c/ and our Apache config looks like this

    ProxyPass http://127.0.0.1:8980/opennms
    ProxyPassReverse http://127.0.0.1:8980/opennms

Instead of forwarding to the X-Forwarded-Host sometimes the localhost address is used instead, which causes confusion for the web browser. See the following three requests made with curl to the webserver https://www.example.com/:

> GET /opennms HTTP/1.1
< Location: https://opennms.example.com/opennms/

> GET /opennms/ HTTP/1.1
< Location: https://127.0.0.1:8980/opennms/frontPage.htm

> GET /opennms/frontPage.htm HTTP/1.1
< Location: https://opennms.example.com/opennms/login.jsp;jsessionid=node02c374aiyvs4v176rxxjdf9kc0438.node0

Expected outcome:
If opennms.web.base-url is configured, it should always be used when rewriting URLs.

OpenNMS version:
30.0.2 on RHEL 7.9

Other relevant data:

This appears to be a failure of Apache to correctly rewrite the response from Jetty; the Location: shouldn’t be 127.0.0.1. baseurl shouldn’t even be at play here, as the only change to baseurl is to switch it to https instead of http and all the responses come back as https URLs.

Is this consistently reproducible? Can you share the exact method you’re using to reproduce this?

It is consistently reproducible and I have reconfigured and restarted OpenNMS for the tests in this follow-up. Apache is usually configured with the following redirect of http to https, but I have disabled it during the tests in this post. The ProxyPass is only configured in the https virtual host in Apache.

RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

This is the test script:

#!/bin/bash

curl https://opennms.example.com/opennms -v 2>&1 | egrep "GET|Location"
curl https://opennms.example.com/opennms/ -v 2>&1 | egrep "GET|Location"
curl https://opennms.example.com/opennms/frontPage.htm -v 2>&1 | egrep "GET|Location"
curl https://opennms.example.com/opennms/login.jsp -v 2>&1 | egrep "GET|base"

When opennms.web.base-url is unset, the test script outputs the following, likely because it defaults to %s://%x%c/ in calculatedUrlBase() in opennms-web-api/src/main/java/org/opennms/web/api/Util.java. The outcome is the same when this default setting is manually configured.

> GET /opennms HTTP/1.1
< Location: https://opennms.example.com/opennms/
> GET /opennms/ HTTP/1.1
< Location: https://opennms.example.com/opennms/frontPage.htm
> GET /opennms/frontPage.htm HTTP/1.1
< Location: https://opennms.example.com/opennms/login.jsp;jsessionid=node0113fjnzc2b20zj2a63f9rt6p5843.node0
> GET /opennms/login.jsp HTTP/1.1
    <base href="http://opennms.example.com/opennms/" />

In this case, when accessing OpenNMS from a web browser, I reach the login page, but no graphics and other content works as those are requested over http and not https.

When opennms.web.base-url = https://%x%c/ everything looks nice except for one of the redirects

> GET /opennms HTTP/1.1
< Location: https://opennms.example.com/opennms/
> GET /opennms/ HTTP/1.1
< Location: https://127.0.0.1:8980/opennms/frontPage.htm
> GET /opennms/frontPage.htm HTTP/1.1
< Location: https://opennms.example.com/opennms/login.jsp;jsessionid=node01w0o5tl32ehga1t7xjwe8khrju1711.node0
> GET /opennms/login.jsp HTTP/1.1
    <base href="https://opennms.example.com/opennms/" />

Just as a test I configured opennms.web.base-url = https://%x%c//// but that only seems to affect <base> and not Location:. It is still presenting localhost though.

> GET /opennms HTTP/1.1
< Location: https://opennms.example.com/opennms/
> GET /opennms/ HTTP/1.1
< Location: https://127.0.0.1:8980/opennms/frontPage.htm
> GET /opennms/frontPage.htm HTTP/1.1
< Location: https://opennms.example.com/opennms/login.jsp;jsessionid=node01gjvao6m2blpbuvdq2jm3ygbn191.node0
> GET /opennms/login.jsp HTTP/1.1
    <base href="https://opennms.example.com/opennms////" />

I don’t see the same result.

[root@30horizon etc]# cat opennms.properties.d/baseurl.properties
opennms.web.base-url = https://%x%c/
(07:34AM) dino@lab ~ $ bash /tmp/foo.sh
> GET /opennms HTTP/1.1
< Location: http://30horizon.ad.secretlab.com/opennms/
> GET /opennms HTTP/1.1
< Location: http://30horizon.ad.secretlab.com/opennms/
> GET /opennms/frontPage.htm HTTP/1.1
< Location: https://30horizon.ad.secretlab.com/opennms/login.jsp;jsessionid=node01cz1k66c0pkr91lqcshc306rqy6001.node0
> GET /opennms/login.jsp HTTP/1.1
    <base href="https://30horizon.ad.secretlab.com/opennms/" />

I don’t have an RHEL7 environment handy, this is on RHEL8.

What version of Apache is this?

I did some further testing with Apache and Nginx and found that the Host header had to be set to the hostname of the reverse proxy for the redirect from /opennms/ to /opennms/frontPage.htm to work properly. It seems this particular redirect doesn’t use the X-Forwarded-Host.

This workaround in Apache works:

ProxyPreserveHost On

Would be interesting to find out what’s wrong with our setup that causes this, but I’m not sure what to look for.