Syslogd: Unable to parse message

troubleshooting
syslogd

#1

I’ve enabled syslogd and made sure that the default syslogd event files are included.
Also I’ve configured rsyslog to forward syslogs to OpenNMS.

Now I’ve simulated a login failure on a server, but I didn’t get an event into the database.
Syslogd tells me: unable to parse.

So I’ve investigated a little bit more.

Syslog entry in auth.log looks like:
Mar 8 08:21:11 rmvm056 sshd[5773]: Failed password for invalid user hans from 10.30.10.24 port 44486 ssh2

Match rule provided in OpenSSH.syslog.xml:
<match type="regex" expression="^Failed (.*?) for invalid user (\S+) from (.*?) port (\d+) ssh(\d)$"/>

Debug log while generating auth error:

2019-03-08 09:50:11,110 DEBUG [AggregatorFlush-Syslog] o.o.n.s.CustomSyslogParser: Regexp not matched: rmvm056 sshd[27873]: Failed password for invalid user hans from 10.30.10.24 port 45260 ssh2
2019-03-08 09:50:11,110 INFO  [AggregatorFlush-Syslog] o.o.n.s.SyslogSinkConsumer: Message discarded, returning without enqueueing event.
org.opennms.netmgt.syslogd.MessageDiscardedException: Unable to parse message: '<38>Mar  8 08:50:10 rmvm056 sshd[27873]: Failed password for invalid user hans from 10.30.10.24 port 45260 ssh2'
        at org.opennms.netmgt.syslogd.ConvertToEvent.<init>(ConvertToEvent.java:304) ~[org.opennms.features.events.syslog-23.0.3.jar:?]
        at org.opennms.netmgt.syslogd.SyslogSinkConsumer.toEventLog(SyslogSinkConsumer.java:124) ~[org.opennms.features.events.syslog-23.0.3.jar:?]
        at org.opennms.netmgt.syslogd.SyslogSinkConsumer.handleMessage(SyslogSinkConsumer.java:101) ~[org.opennms.features.events.syslog-23.0.3.jar:?]
        at org.opennms.netmgt.syslogd.SyslogSinkConsumer.handleMessage(SyslogSinkConsumer.java:61) ~[org.opennms.features.events.syslog-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageConsumerManager.lambda$dispatch$0(AbstractMessageConsumerManager.java:90) ~[org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_191]
        at org.opennms.core.ipc.sink.common.AbstractMessageConsumerManager.dispatch(AbstractMessageConsumerManager.java:90) [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.camel.client.CamelLocalMessageDispatcherFactory.dispatch(CamelLocalMessageDispatcherFactory.java:52) [org.opennms.core.ipc.sink.camel.client-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.camel.client.CamelLocalMessageDispatcherFactory.dispatch(CamelLocalMessageDispatcherFactory.java:45) [org.opennms.core.ipc.sink.camel.client-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory.timedDispatch(AbstractMessageDispatcherFactory.java:80) [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory.access$000(AbstractMessageDispatcherFactory.java:61) [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory$1.dispatch(AbstractMessageDispatcherFactory.java:121) [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.common.AbstractMessageDispatcherFactory$1.dispatch(AbstractMessageDispatcherFactory.java:118) [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.aggregation.Aggregator.run(Aggregator.java:189) [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at org.opennms.core.ipc.sink.aggregation.Aggregator$1.run(Aggregator.java:104) [org.opennms.core.ipc.sink.common-23.0.3.jar:?]
        at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_191]
        at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_191]

I’ve also tried to change the file format setting in rsyslog.conf:

# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Then the logs look like:

2019-03-08T09:20:50.639859+00:00 rmvm056 sshd[16052]: Failed none for invalid user hans from 10.30.10.24 port 46388 ssh2

But I get the same parsing error.

Also I’ve verified the regex and the syslog entry with https://debuggex.com and it worked.


#2

Which parser did you specify in syslogd-configuration.xml ?


#3

It’s the default config:

    <configuration
            syslog-port="10514"
            new-suspect-on-message="false"
            parser="org.opennms.netmgt.syslogd.CustomSyslogParser"
            forwarding-regexp="^.*\s(19|20)\d\d([-/.])(0[1-9]|1[012])\2(0[1-9]|[12][0-9]|3[01])(\s+)(\S+)(\s)(\S.+)"
            matching-group-host="6"
            matching-group-message="8"
            discard-uei="DISCARD-MATCHING-MESSAGES"
            timezone=""
            />

#4

My forwarding regex is the only difference here. I am using : forwarding-regexp="^((.+?) (.*))\r?\n?$"


#5

But isn’t the log already forwarded when the log is complaining about the matching in OpenSSH.syslog.xml?


#6

Got me! I only know mine is working and yours isn’t :clown_face:


#7

So you have changed yours in past? Why? Was it not working? :joy:


#8

@dino2gnt I’ve tested your regex. Now I’m getting:
2019-03-08 20:27:06,310 ERROR [AggregatorFlush-Syslog] o.o.n.s.SyslogSinkConsumer: Unexpected exception while processing SyslogConnection java.lang.IndexOutOfBoundsException: No group 8

What I didn’t mention: The syslog client VM is an Ubuntu 18 running rsyslog. Not sure if that makes a difference. My OpenNMS version is 23.0.3.


#9

Show me the entire ueiMatch stanza for that message please.


#10
<ueiMatch>
  <process-match expression="^sshd$"/>
  <match type="regex" expression="^Failed (.*?) for invalid user (\S+) from (.*?) port (\d+) ssh(\d)$"/>
  <uei>uei.opennms.org/vendor/openssh/syslog/sshd/invalidUser</uei>
  <parameter-assignment matching-group="1" parameter-name="authMethod"/>
  <parameter-assignment matching-group="2" parameter-name="user"/>
  <parameter-assignment matching-group="3" parameter-name="remoteHost"/>
  <parameter-assignment matching-group="4" parameter-name="remotePort"/>
  <parameter-assignment matching-group="5" parameter-name="protocolVersion"/>
</ueiMatch>

#11

Marcel,

I think you want the RadixTreeSyslogParser, assuming you’re on a release that includes it. This parser is the default in Horizon 24.


#12

It works with the RadixTreeSyslogParser. Thanks @jeffg
Good to here, that this one is the default one in H24, since it seems to work out of the box.