Syslogd: Unable to parse message

I’ve enabled syslogd and made sure that the default syslogd event files are included.
Also I’ve configured rsyslog to forward syslogs to OpenNMS.

Now I’ve simulated a login failure on a server, but I didn’t get an event into the database.
Syslogd tells me: unable to parse.

So I’ve investigated a little bit more.

Syslog entry in auth.log looks like:
Mar 8 08:21:11 rmvm056 sshd[5773]: Failed password for invalid user hans from port 44486 ssh2

Match rule provided in OpenSSH.syslog.xml:
<match type="regex" expression="^Failed (.*?) for invalid user (\S+) from (.*?) port (\d+) ssh(\d)$"/>

Debug log while generating auth error:

2019-03-08 09:50:11,110 DEBUG [AggregatorFlush-Syslog] o.o.n.s.CustomSyslogParser: Regexp not matched: rmvm056 sshd[27873]: Failed password for invalid user hans from port 45260 ssh2
2019-03-08 09:50:11,110 INFO  [AggregatorFlush-Syslog] o.o.n.s.SyslogSinkConsumer: Message discarded, returning without enqueueing event.
org.opennms.netmgt.syslogd.MessageDiscardedException: Unable to parse message: '<38>Mar  8 08:50:10 rmvm056 sshd[27873]: Failed password for invalid user hans from port 45260 ssh2'
        at org.opennms.netmgt.syslogd.ConvertToEvent.<init>( ~[]
        at org.opennms.netmgt.syslogd.SyslogSinkConsumer.toEventLog( ~[]
        at org.opennms.netmgt.syslogd.SyslogSinkConsumer.handleMessage( ~[]
        at org.opennms.netmgt.syslogd.SyslogSinkConsumer.handleMessage( ~[]
        at org.opennms.core.ipc.sink.common.AbstractMessageConsumerManager.lambda$dispatch$0( ~[org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at java.lang.Iterable.forEach( [?:1.8.0_191]
        at org.opennms.core.ipc.sink.common.AbstractMessageConsumerManager.dispatch( [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.camel.client.CamelLocalMessageDispatcherFactory.dispatch( [org.opennms.core.ipc.sink.camel.client-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.camel.client.CamelLocalMessageDispatcherFactory.dispatch( [org.opennms.core.ipc.sink.camel.client-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory.timedDispatch( [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory.access$000( [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory$1.dispatch( [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory$1.dispatch( [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.aggregation.Aggregator$ [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at java.util.TimerThread.mainLoop( [?:1.8.0_191]
        at [?:1.8.0_191]

I’ve also tried to change the file format setting in rsyslog.conf:

# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Then the logs look like:

2019-03-08T09:20:50.639859+00:00 rmvm056 sshd[16052]: Failed none for invalid user hans from port 46388 ssh2

But I get the same parsing error.

Also I’ve verified the regex and the syslog entry with and it worked.

Which parser did you specify in syslogd-configuration.xml ?

It’s the default config:


My forwarding regex is the only difference here. I am using : forwarding-regexp="^((.+?) (.*))\r?\n?$"

But isn’t the log already forwarded when the log is complaining about the matching in OpenSSH.syslog.xml?

Got me! I only know mine is working and yours isn’t :clown_face:

1 Like

So you have changed yours in past? Why? Was it not working? :joy:

@dino2gnt I’ve tested your regex. Now I’m getting:
2019-03-08 20:27:06,310 ERROR [AggregatorFlush-Syslog] o.o.n.s.SyslogSinkConsumer: Unexpected exception while processing SyslogConnection java.lang.IndexOutOfBoundsException: No group 8

What I didn’t mention: The syslog client VM is an Ubuntu 18 running rsyslog. Not sure if that makes a difference. My OpenNMS version is 23.0.3.

Show me the entire ueiMatch stanza for that message please.

  <process-match expression="^sshd$"/>
  <match type="regex" expression="^Failed (.*?) for invalid user (\S+) from (.*?) port (\d+) ssh(\d)$"/>
  <parameter-assignment matching-group="1" parameter-name="authMethod"/>
  <parameter-assignment matching-group="2" parameter-name="user"/>
  <parameter-assignment matching-group="3" parameter-name="remoteHost"/>
  <parameter-assignment matching-group="4" parameter-name="remotePort"/>
  <parameter-assignment matching-group="5" parameter-name="protocolVersion"/>


I think you want the RadixTreeSyslogParser, assuming you’re on a release that includes it. This parser is the default in Horizon 24.

It works with the RadixTreeSyslogParser. Thanks @jeffg
Good to here, that this one is the default one in H24, since it seems to work out of the box.