Issue with sFlow

Error seeing the sFlow data:
I am trying to setup the Network analysis dashboard but unable to see any sFlow data in it
Expected outcome:
Visualize the sFlow data being sent from the Switches on the Grafana dashboard

OpenNMS version:
Version: 27.1.0
Other relevant data:
Karaf log

logs
2021-03-31T09:57:36,301 | ERROR | AggregatorFlush-Telemetry-SFlow | AbstractFlowAdapter              | 326 - org.opennms.features.telemetry.protocols.flows - 27.1.0 | Error while persisting flows: Failed to enrich one or more flows.

Caused by: org.bson.BsonInvalidOperationException: Value expected to be of type INT64 is of unexpected type DOCUMENT

I’m running sFlow in my lab (from an HP ProCurve switch) on 27.1.0 and it works fine and enriches as expected. What type of device is this sFlow coming from? Can you provide more details on your environment and setup?

Thanks for your response.
The sFlow is coming from HPE 5800-48G-PoE+ Switch, HPE Comware Platform, Software Version 5.20.105.

I am testing the sFlow - Flow Deep Dive with Opennms.

I did the following checks:

Telemetryd Availability

Nmap result:

Starting Nmap 7.60 ( https://nmap.org ) at 2021-03-25 14:23 PDT
Nmap scan report for localhost (127.0.0.1)
Host is up.
Other addresses for localhost (not scanned): ::1

PORT STATE SERVICE
6343/udp open|filtered sflow

Nmap done: 1 IP address (1 host up) scanned in 2.23 seconds

The flow packages are coming as well. I ran tcpdump and seeing the data coming through.

ONMS Health Check results in the following

admin@opennms> opennms:health-check
Verifying the health of the container

Verifying installed bundles [ Success ]
Connecting to ElasticSearch ReST API (Flows) [ Success ]

=> Everything is awesome

SNMP enabled and working fine. There is performance data for the Switch, but no Flow data.

Karaf log shows the error when Enriching the flows.

Please let me know if you need more details.
Thank you.

Can you provide a packet capture of some failing sflow data?

Here’s the tcpdump for the sFlow data received on the opennms server port 6343

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:14:44.928826 IP (tos 0x0, ttl 254, id 35900, offset 0, flags [none], proto UDP (17), length 276)
    15.1.253.63.3242 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 1, seqnum 9571, uptime 3239297244, samples 1, length 248
        expanded flow sample (3), length 212, seqnum 2405969, type 0, idx 53, rate 4000, pool 2847871565, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 99, src pri 0, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 136
              protocol Ethernet (1), length 122, stripped bytes 4, header_size 118
12:14:48.933167 IP (tos 0x0, ttl 254, id 35920, offset 0, flags [none], proto UDP (17), length 256)
    15.1.253.63.3242 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 1, seqnum 9572, uptime 3239301248, samples 1, length 228
        expanded flow sample (3), length 192, seqnum 2405970, type 0, idx 53, rate 4000, pool 2847871767, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 99, src pri 6, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 116
              protocol Ethernet (1), length 102, stripped bytes 4, header_size 98
12:16:56.363497 IP (tos 0x0, ttl 254, id 18037, offset 0, flags [none], proto UDP (17), length 284)
    15.1.253.63.50154 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 3, seqnum 11252, uptime 3239467652, samples 1, length 256
        expanded flow sample (3), length 220, seqnum 1849144, type 0, idx 159, rate 4000, pool 3563576748, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 99, src pri 6, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 144
              protocol Ethernet (1), length 278, stripped bytes 4, header_size 128
12:18:06.402097 IP (tos 0x0, ttl 254, id 18053, offset 0, flags [none], proto UDP (17), length 284)
    15.1.253.63.50154 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 3, seqnum 11253, uptime 3239537690, samples 1, length 256
        expanded flow sample (3), length 220, seqnum 1849145, type 0, idx 159, rate 4000, pool 3563580200, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 98, src pri 6, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 144
              protocol Ethernet (1), length 722, stripped bytes 4, header_size 128
12:18:56.027504 IP (tos 0x0, ttl 254, id 38619, offset 0, flags [none], proto UDP (17), length 284)
    15.1.253.63.3242 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 1, seqnum 9573, uptime 3239548340, samples 1, length 256
        expanded flow sample (3), length 220, seqnum 2405971, type 0, idx 53, rate 4000, pool 2847887450, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 99, src pri 6, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 144
              protocol Ethernet (1), length 278, stripped bytes 4, header_size 128
12:19:08.419785 IP (tos 0x0, ttl 254, id 18081, offset 0, flags [none], proto UDP (17), length 284)
    15.1.253.63.50154 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 3, seqnum 11254, uptime 3239599707, samples 1, length 256
        expanded flow sample (3), length 220, seqnum 1849146, type 0, idx 159, rate 4000, pool 3563583359, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 99, src pri 6, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 144
              protocol Ethernet (1), length 278, stripped bytes 4, header_size 128
12:24:06.139899 IP (tos 0x0, ttl 254, id 41569, offset 0, flags [none], proto UDP (17), length 284)
    15.1.253.63.3242 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 1, seqnum 9574, uptime 3239858450, samples 1, length 256
        expanded flow sample (3), length 220, seqnum 2405972, type 0, idx 53, rate 4000, pool 2847907154, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 98, src pri 6, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 144
              protocol Ethernet (1), length 178, stripped bytes 4, header_size 128
12:25:31.694847 IP (tos 0x0, ttl 254, id 18174, offset 0, flags [none], proto UDP (17), length 284)
    15.1.253.63.50154 > rt0086g.sdg.rd.hpicorp.net.6343: sFlowv5, IPv4 agent sdg0615sw01.sdg.rd.hpicorp.net, agent-id 3, seqnum 11255, uptime 3239982979, samples 1, length 256
        expanded flow sample (3), length 220, seqnum 1849147, type 0, idx 159, rate 4000, pool 3563600351, drops 0, records 2
            enterprise 0 Extended Switch data (1001) length 16
              src vlan 98, src pri 6, dst vlan 0, dst pri 0
            enterprise 0 Raw packet (1) length 144
              protocol Ethernet (1), length 178, stripped bytes 4, header_size 128
^C
8 packets captured
51 packets received by filter
0 packets dropped by kernel

No, an actual packet capture / pcap from e.g. wireshark or similar, please.

Here’s the packet capture for UDP port 6343. Not sure how to attach the pcap file here.
Thanks.

    Interface id: 0 (eth0)
        Interface name: eth0
    Encapsulation type: Ethernet (1)
    Arrival Time: Mar 31, 2021 14:52:06.214145794 Pacific Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1617227526.214145794 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 298 bytes (2384 bits)
    Capture Length: 298 bytes (2384 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:sflow:eth:ethertype:vlan:ethertype:ip:ospf]
    [Coloring Rule Name: TTL low or unexpected]
    [Coloring Rule String: ( ! ip.dst == 224.0.0.0/4 && ip.ttl < 5 && !pim && !ospf) || (ip.dst == 224.0.0.0/24 && ip.dst != 224.0.0.251 && ip.ttl != 1 && !(vrrp || carp))]
Ethernet II, Src: Fortinet_09:00:05 (00:09:0f:09:00:05), Dst: Vmware_99:cb:26 (00:50:56:99:cb:26)
    Destination: Vmware_99:cb:26 (00:50:56:99:cb:26)
        Address: Vmware_99:cb:26 (00:50:56:99:cb:26)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: Fortinet_09:00:05 (00:09:0f:09:00:05)
        Address: Fortinet_09:00:05 (00:09:0f:09:00:05)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 15.1.253.63, Dst: 15.1.251.31
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 284
    Identification: 0x5033 (20531)
    Flags: 0x0000
        0... .... .... .... = Reserved bit: Not set
        .0.. .... .... .... = Don't fragment: Not set
        ..0. .... .... .... = More fragments: Not set
        ...0 0000 0000 0000 = Fragment offset: 0
    Time to live: 254
    Protocol: UDP (17)
    Header checksum: 0x553c [validation disabled]
    [Header checksum status: Unverified]
    Source: 15.1.253.63
    Destination: 15.1.251.31
User Datagram Protocol, Src Port: 50154, Dst Port: 6343
    Source Port: 50154
    Destination Port: 6343
    Length: 264
    Checksum: 0x5e5c [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    [Timestamps]
        [Time since first frame: 0.000000000 seconds]
        [Time since previous frame: 0.000000000 seconds]
InMon sFlow
    Datagram version: 5
    Agent address type: IPv4 (1)
    Agent address: 15.1.255.63
    Sub-agent ID: 3
    Sequence number: 11363
    SysUptime: 37601 days, 14 hours, 11 minutes, 9 seconds (3248777469s)
    NumSamples: 1
    Expanded flow sample, seq 1849258
        0000 0000 0000 0000 0000 .... .... .... = Enterprise: standard sFlow (0)
        .... .... .... .... .... 0000 0000 0011 = sFlow sample type: Expanded flow sample (3)
        Sample length (byte): 220
        Sequence number: 1849258
        Source ID type: 0
        Source ID index: 159
        Sampling rate: 1 out of 4000 packets
        Sample pool: 3564048281 total packets
        Dropped packets: 0
        Input interface format: 0
        Input interface value: 159
        .000 0000 0000 0000 0000 0000 0000 0000 = Output interface format: 0
        Output interface value: 0
        Flow record: 2
        Extended switch data
            0000 0000 0000 0000 0000 .... .... .... = Enterprise: standard sFlow (0)
            Format: Extended switch data (1001)
            Flow data length (byte): 16
            Incoming 802.1Q VLAN: 99
            Incoming 802.1p priority: 6
            Outgoing 802.1Q VLAN: 0
            Outgoing 802.1p priority: 0
        Raw packet header
            0000 0000 0000 0000 0000 .... .... .... = Enterprise: standard sFlow (0)
            Format: Raw packet header (1)
            Flow data length (byte): 144
            Header protocol: Ethernet (1)
            Frame Length: 278
            Payload removed: 4
            Original packet length: 128
            Header of sampled packet: 01005e000005d07e286dc0c08100c063080045c00100cad5…
                Ethernet II, Src: HewlettP_6d:c0:c0 (d0:7e:28:6d:c0:c0), Dst: IPv4mcast_05 (01:00:5e:00:00:05)
                    Destination: IPv4mcast_05 (01:00:5e:00:00:05)
                        Address: IPv4mcast_05 (01:00:5e:00:00:05)
                        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
                        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
                    Source: HewlettP_6d:c0:c0 (d0:7e:28:6d:c0:c0)
                        Address: HewlettP_6d:c0:c0 (d0:7e:28:6d:c0:c0)
                        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
                        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
                    Type: 802.1Q Virtual LAN (0x8100)
                802.1Q Virtual LAN, PRI: 6, DEI: 0, ID: 99
                    110. .... .... .... = Priority: Internetwork Control (6)
                    ...0 .... .... .... = DEI: Ineligible
                    .... 0000 0110 0011 = ID: 99
                    Type: IPv4 (0x0800)
                Internet Protocol Version 4, Src: 15.1.253.22, Dst: 224.0.0.5
                    0100 .... = Version: 4
                    .... 0101 = Header Length: 20 bytes (5)
                    Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
                        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
                        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
                    Total Length: 256
                    Identification: 0xcad5 (51925)
                    Flags: 0x0000
                        0... .... .... .... = Reserved bit: Not set
                        .0.. .... .... .... = Don't fragment: Not set
                        ..0. .... .... .... = More fragments: Not set
                        ...0 0000 0000 0000 = Fragment offset: 0
                    Time to live: 1
                    Protocol: OSPF IGP (89)
                    Header checksum: 0x00f3 [validation disabled]
                    [Header checksum status: Unverified]
                    Source: 15.1.253.22
                    Destination: 224.0.0.5
                Open Shortest Path First
                    OSPF Header
                        Version: 2
                        Message Type: Hello Packet (1)
                        Packet Length: 220
                        Source OSPF Router: 15.1.255.22
                        Area ID: 0.0.0.0 (Backbone)
                        Checksum: 0x0000 (None)
                        Auth Type: Cryptographic (2)
                        Auth Crypt Key id: 99
                        Auth Crypt Data Length: 16
                        Auth Crypt Sequence Number: 35554086
                    OSPF Hello Packet
                        Network Mask: 255.255.255.128
                        Hello Interval [sec]: 3
                        Options: 0x02, (E) External Routing
                            0... .... = DN: Not set
                            .0.. .... = O: Not set
                            ..0. .... = (DC) Demand Circuits: Not supported
                            ...0 .... = (L) LLS Data block: Not Present
                            .... 0... = (N) NSSA: Not supported
                            .... .0.. = (MC) Multicast: Not capable
                            .... ..1. = (E) External Routing: Capable
                            .... ...0 = (MT) Multi-Topology Routing: No
                        Router Priority: 0
                        Router Dead Interval [sec]: 10
                        Designated Router: 15.1.253.1
                        Backup Designated Router: 0.0.0.0
                        Active Neighbor: 15.1.255.1
                        Active Neighbor: 15.1.255.7
                        Active Neighbor: 15.1.255.20
                        Active Neighbor: 15.1.255.21
                        Active Neighbor: 15.1.255.23
                        Active Neighbor: 15.1.255.24
                        Active Neighbor: 15.1.255.25
                        Active Neighbor: 15.1.255.30
                        Active Neighbor: 15.1.255.31
                        Active Neighbor: 15.1.255.32
                        Active Neighbor: 15.1.255.33

Huh. I guess you can’t. That seems rather… deficient. Sorry.

Is there any other information you need. Any pointers on how to troubleshoot the issue could help.

Not sure what else you could provide that would be helpful, beyond a pcap that shows sflow packets that fail.

What version of Elasticsearch and elasticsearch-drift-plugin are you running?

The version of Elasticsearch and elasticsearch-drift-plugin is 7.6.2.

I ran the “_ws.expert.severity != Ok” and “_ws.expert.severity == error” on the capture packet and no errors or malformed packets found. The Elastic Search version and the elasticsearch-drift-plugin matches.
Not sure what other test I can run. I followed the following Troubleshoot-Telemetryd and all the tests comes as success.
Is there any other steps or suggestions I can try.

Thank you.

Only thing I can suggest is trying InMon’s reference sflow receiver and seeing if it can process the sflow packets. InMon: sFlowTrend If HP is doing something nonstandard, I expect both implementations will have issues. If this issue is caused by OpenNMS failing to parse the packet correctly, sFlowTrend should work.

If we are failing to parse the packet correctly, open a bug report on issues.opennms.org and attach the packet captures of some failing packets there, so we can try to figure out what we’re missing.

Okay. I’ll deploy the InMon’s reference sflow receiver and post the results.

Thank you.

I tested the InMon’s reference sflow receiver and I can see the sFlow data populated on the web UI.
See below screenshot here. What’s the process in creating a bug report on issues.opennms.org ?

Thank you

Issue with parsing sFlow

Clearing the data directory and replacing the JAR provided in the Jira ticket did the trick. It’s working now.
I can now visualize the Flow data via OpenNMS & Grafana.

Thank you for helping!