Page Sequence Monitor - Cannot identify XDH private key - Following Upgrade

Problem: Since Upgrading to Release 27.1.1 from Release 27.0.3; Iā€™m getting the below a failure on the polling service when utilising Page-Sequence. The error returned is " cannot identify XDH private key." The systems that are being monitored are behind an Nginx proxy; while the proxy has a wildcard certificate.

Polling outcome: New Error: ā€œA Web-Landing-Page-SSL outage was identified on interface 10.255.200.14 because of the following condition: cannot identify XDH private key.ā€

Expected outcome:
Restore normal operation; so that the sites are monitored as they were before the upgrade.

OpenNMS version: 27.1.1

Other relevant data:

Exmaple Cert:

Service Configuration

      <service name="Web-Landing-Page-SSL" interval="300000" user-defined="true" status="on">
         <parameter key="retry" value="1"/>
         <parameter key="timeout" value="5000"/>
         <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
         <parameter key="ds-name" value="webLandingSsl"/>
         <parameter key="page-sequence">
            <page-sequence xmlns="">
               <page disable-ssl-verification="true" host="${nodelabel}" http-version="1.1" method="GET" path="/" port="443" requireIPv4="false" requireIPv6="false" response-range="200-399" scheme="https" virtual-host="${nodelabel}" xmlns=""/>
            </page-sequence>
         </parameter>
      </service>


Debug of failing poll

2021-04-27 21:42:37,174 DEBUG [pool-13-thread-321] o.a.h.i.c.DefaultHttpClientConnectionOperator: Connecting to wiki.abcd.tv/10.255.200.14:443
2021-04-27 21:42:37,175 DEBUG [pool-13-thread-321] o.a.h.c.s.SSLConnectionSocketFactory: Connecting socket to wiki.abcd.tv/10.255.200.14:443 with timeout 5000
2021-04-27 21:42:37,175 DEBUG [pool-13-thread-321] o.a.h.c.s.SSLConnectionSocketFactory: Enabled protocols: [TLSv1.3, TLSv1.2]
2021-04-27 21:42:37,175 DEBUG [pool-13-thread-321] o.a.h.c.s.SSLConnectionSocketFactory: Enabled cipher suites:[TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2021-04-27 21:42:37,175 DEBUG [pool-13-thread-321] o.a.h.c.s.SSLConnectionSocketFactory: Starting handshake
2021-04-27 21:42:37,177 DEBUG [Poller-Thread-17-of-30] o.o.n.p.QueryManagerDaoImpl: Successfully updated last good/fail timestamp for service named SNMP on node id 109 and interface /10.70.16.10.
2021-04-27 21:42:37,177 DEBUG [Poller-Thread-17-of-30] o.o.n.p.p.PollableService: Finish Scheduled Poll of service PollableService[location=Default, interface=PollableInterface [PollableNode [109]:10.70.16.10], svcName=SNMP], started at 2021-04-27T21:42:36.912+0000
2021-04-27 21:42:37,177 DEBUG [Poller-Thread-17-of-30] o.o.n.s.LegacyScheduler: schedule: Adding ready runnable ScheduleEntry[expCode=1] for PollableService[location=Default, interface=PollableInterface [PollableNode [109]:10.70.16.10], svcName=SNMP] (ready in 300000ms) at interval 300000
2021-04-27 21:42:37,177 DEBUG [Poller-Thread-17-of-30] o.o.n.s.LegacyScheduler: schedule: queue element added, notification not performed
2021-04-27 21:42:37,173 DEBUG [pool-13-thread-304] o.o.n.p.m.PageSequenceMonitor: cannot identify XDH private key
java.security.InvalidKeyException: cannot identify XDH private key
        at org.bouncycastle.jcajce.provider.asymmetric.edec.KeyAgreementSpi.engineDoPhase(Unknown Source) ~[bcprov-jdk15on-1.66.jar:1.66.0]
        at javax.crypto.KeyAgreement.doPhase(KeyAgreement.java:579) ~[?:?]
        at sun.security.ssl.KAKeyDerivation.t13DeriveKey(KAKeyDerivation.java:104) ~[?:?]
        at sun.security.ssl.KAKeyDerivation.deriveKey(KAKeyDerivation.java:63) ~[?:?]
        at sun.security.ssl.ServerHello$T13ServerHelloConsumer.consume(ServerHello.java:1256) ~[?:?]
        at sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(ServerHello.java:990) ~[?:?]
        at sun.security.ssl.ServerHello$ServerHelloConsumer.consume(ServerHello.java:878) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411) ~[?:?]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar:4.5.2]
        at org.opennms.core.web.HttpClientWrapper.execute(HttpClientWrapper.java:353) ~[org.opennms.core.web-27.1.1.jar:?]
        at org.opennms.netmgt.poller.monitors.PageSequenceMonitor$HttpPage.execute(PageSequenceMonitor.java:379) [opennms-services-27.1.1.jar:?]
        at org.opennms.netmgt.poller.monitors.PageSequenceMonitor$HttpPageSequence.execute(PageSequenceMonitor.java:202) [opennms-services-27.1.1.jar:?]
        at org.opennms.netmgt.poller.monitors.PageSequenceMonitor$HttpPageSequence.access$100(PageSequenceMonitor.java:160) [opennms-services-27.1.1.jar:?]
        at org.opennms.netmgt.poller.monitors.PageSequenceMonitor.poll(PageSequenceMonitor.java:685) [opennms-services-27.1.1.jar:?]
        at org.opennms.netmgt.poller.client.rpc.PollerClientRpcModule$1.get(PollerClientRpcModule.java:77) [org.opennms.features.poller.client-rpc-27.1.1.jar:?]
        at org.opennms.netmgt.poller.client.rpc.PollerClientRpcModule$1.get(PollerClientRpcModule.java:71) [org.opennms.features.poller.client-rpc-27.1.1.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]```

There is a known issue that us being tracked. See [NMS-13111] BouncyCastle breaks SSL connector in Jetty for details. You may ned to roll back Java to a previous issue.

1 Like

@mmahacek thanks for the rapid response; understood :slight_smile:

I had come across the exact same issue on my test 27.1.1 server, so it was fresh in my mind.

1 Like

This is resolved the issue for now :slight_smile:

Useful commands for anyone viewing this thread:

  • apt-cache policy openjdk-11-jre-headless
  • sudo apt-get install openjdk-11-jre-headless=11.0.9.1+1-1~deb10u2 openjdk-11-jdk-headless=11.0.9.1+1-1~deb10u2
  • sudo apt-mark hold openjdk-11-jre-headless openjdk-11-jdk-headless