Traffic Analyzer Missing Packets

Feb 17, 2015 at 3:25 PM
I am trying to analyze some network traffic that is ~60% bus load (100Mb) with ~172us from the start of one packet to the start of another packet.

I have tried to receive the packets using ReceivePacket(), ReceivePackets() and ReceiveSomePackets() but every method seems to miss some of the packets. I miss anywhere from 1-40 packets every minute or so no matter which method I use, but usually I only miss 1 or 2 packets. I can see the missing packets in Wireshark, and when I print the TotalStatistics.PacketsDroppedByDriver and the TotalStatistics.PacketsDroppedByInterface they are both zero. Also, the CPU usage of my program is only about 1%-5% so I do not understand why I am unable to process all of the received packets.

I tried tweaking the SetKernelBufferSize() and SetKernelMinimumBytesToCopy() and it seemed to help some but not that much.

I am currently at a loss on what to try next. I feel like it should be possible for me to process all of the packets if Wireshark is able to display them. Any suggestions?

Here is my code:
        // Function to initialize the Ethernet Adaptor
        public static int initEthAdaptor(PacketDevice selectedAdaptor, string ipAddress)
        {
            // Open the device in promiscuous mode with a 5 second timeout
            // 65536 guarantees that the whole packet will be captured on all the link layers
            try
            {
                communicator = selectedAdaptor.Open(65536, PacketDeviceOpenAttributes.Promiscuous, 5000);

                // Set the Ethernet filter
                status = setEthFilter(ipAddress, true, "57005");
                if (status != 1) return -1; // failure

                // Set the minimum number of bytes to be received by the kernel before they can be copied to the user
                // Set value to minimum of 1 byte to ensure the kernel copies the packets to the user as soon as possible
                communicator.SetKernelMinimumBytesToCopy(1);
            }
            catch (Exception e)
            {
                dispatcherGUI.Invoke(delGUIConsole, new object[] { "Failed to Open " + e.Message });
                return -1; // failure
            }

            dispatcherGUI.Invoke(delGUIConsole, new object[] { "Opened " + selectedAdaptor.Description + "" });

            return 1;
        }

        // Function to analyze the ethernet packet 
        public static int ethPacketAnalyzer(bool adaptor)
        {
            // Ensure an Ethernet adaptor is selected for packet analysis
            if (adaptor)
            {
                // Set the Ethernet filter
                status = setEthFilter("192.168.3.43", true, "2368");
                if (status != 1) return -1; // failure

                communicator.SetKernelBufferSize(8 * 1024 * 1024);
                communicator.SetKernelMinimumBytesToCopy(2 * 1024 * 1024);

                // METHOD #1
                while (true)
                {
                    Packet packet;

                    // Receive the Ethernet packet
                    PacketCommunicatorReceiveResult result = communicator.ReceivePacket(out packet);

                    // Check for a successful or failed read
                    if (result == PacketCommunicatorReceiveResult.Ok)
                    {
                        IpV4Datagram ip = packet.Ethernet.IpV4;
                        UdpDatagram udp = ip.Udp;

                        // Increment packet counter
                        numAnalyzePackets++;

                        // Shift the Unit Check byte array
                        unitCheck[4] = unitCheck[3];
                        unitCheck[3] = unitCheck[2];
                        unitCheck[2] = unitCheck[1];
                        unitCheck[1] = unitCheck[0];
                        //unitCheck[0] = lidar_packet[lidar_packet.Length - 1];
                        unitCheck[0] = udp.Payload.Last;

                        // Check for UNIT#
                        if (unitCheck[4] == 'U' && unitCheck[3] == 'N' && unitCheck[2] == 'I' && unitCheck[1] == 'T' && unitCheck[0] == '#')
                        {
                            if (numAnalyzePackets == 4160)
                            {
                                // No dropped packets
                                numAnalyzePackets = 0;
                            }
                            else
                            {
                                dispatcherGUI.Invoke(delGUIConsole, new object[] { "Packets were dropped.  Analyzed " + numAnalyzePackets });
                                dispatcherGUI.Invoke(delGUIConsole, new object[] { "     - Received " + communicator.TotalStatistics.PacketsReceived + ",    Captured " + communicator.TotalStatistics.PacketsCaptured + ",    Dropped by Driver " + communicator.TotalStatistics.PacketsDroppedByDriver + ",    Dropped by Interface " + communicator.TotalStatistics.PacketsDroppedByInterface });

                                numAnalyzePackets = 0;
                            }
                        }
                    }
                    else if (result == PacketCommunicatorReceiveResult.Timeout)
                    {
                        // Timeout elapsed
                        dispatcherGUI.Invoke(delGUIConsole, new object[] { "Failed to read Lidar response!  Timeout elapsed." });
                        return -1;
                    }
                    else
                    {
                        dispatcherGUI.Invoke(delGUIConsole, new object[] { "Error returned: Failed to read Lidar response!" });
                        return -2;
                    }
                }

                // METHOD #2
                // Start receiving Ethernet packets, create event for every single received packet
                //communicator.ReceivePackets(0, ethPacketHandlerAnalyzer);


                /*
                // METHOD #3
                while (true)
                {
                    // Start receiving Ethernet packets, create event for every X number of packets
                    communicator.ReceivePackets(2000, ethPacketHandlerAnalyzer);
                    //dispatcherGUI.Invoke(delGUIConsole, new object[] { "Loop again...  " + numRxPackets });
                }
                */

                /*
                // METHOD #4
                int numRxPackets;
                while (true)
                {
                    communicator.ReceiveSomePackets(out numRxPackets, 4160, ethPacketHandlerAnalyzer);
                    //dispatcherGUI.Invoke(delGUIConsole, new object[] { "Loop again...  " + numRxPackets });
                }
                */
            }
            else
            {
                dispatcherGUI.Invoke(delGUIConsole, new object[] { "Error:  An Ethernet Adaptor must be selected in order to analyze the packets." });
                return -1;
            }

            return 1; // pass
        }

        // Callback function invoked by Pcap.Net for every incoming packet
        private static void ethPacketHandlerAnalyzer(Packet packet)
        {
            IpV4Datagram ip = packet.Ethernet.IpV4;
            UdpDatagram udp = ip.Udp;

            // Save array of data bytes
            //byte[] lidar_packet = udp.Payload.ToArray();
            
            // Increment packet counter
            numAnalyzePackets++;

            // Shift the Unit Check byte array
            unitCheck[4] = unitCheck[3];
            unitCheck[3] = unitCheck[2];
            unitCheck[2] = unitCheck[1];
            unitCheck[1] = unitCheck[0];
            //unitCheck[0] = lidar_packet[lidar_packet.Length - 1];
            unitCheck[0] = udp.Payload.Last;

            // Check for UNIT#
            if (unitCheck[4] == 'U' && unitCheck[3] == 'N' && unitCheck[2] == 'I' && unitCheck[1] == 'T' && unitCheck[0] == '#')
            {
                if (numAnalyzePackets == 4160)
                {
                    // No dropped packets
                    numAnalyzePackets = 0;
                }
                else
                {
                    dispatcherGUI.Invoke(delGUIConsole, new object[] { "Packets were dropped.  Analyzed " + numAnalyzePackets });
                    dispatcherGUI.Invoke(delGUIConsole, new object[] { "     - Received " + communicator.TotalStatistics.PacketsReceived + ",    Captured " + communicator.TotalStatistics.PacketsCaptured + ",    Dropped by Driver " + communicator.TotalStatistics.PacketsDroppedByDriver + ",    Dropped by Interface " + communicator.TotalStatistics.PacketsDroppedByInterface });
                    
                    numAnalyzePackets = 0;
                }
            }
        }
Feb 19, 2015 at 10:50 AM
Hello!
your code is too much and i have some difficulties to read it. but your problem is at your filter. you make a filter to capture some specific packet you want. And that
limit the fact of capturing all packet in your network.
can you tell explicitly what you want to do?
Coordinator
Feb 20, 2015 at 5:19 AM
At what packet rate do you manage to not lose packets?
Feb 20, 2015 at 1:01 PM
@Barack1234
Yes sorry the code is not the cleanest, I copied it out of a larger project as is. I also left the 4 different code approaches in there with 3 of them commented out so that you could see what I had tried, but it made it even harder to follow.
I am trying to analyze ethernet packets from a sensor, so I am setting a filter to ensure I only see packets from a specific IP address and port number. The sensor cycles through a 4160 packet sequence and I look for specific bytes to find the string "UNIT#" so that I can frame out the sequence and count to see if I recieved all 4160 packets. Most of the time I do receive them all, but as I stated above every minute or so I miss between 1-40 packets of a cycle. I know its not the sensor because I can run this code on two different PCs at the same time and it is random which computer will drop packets and when but they have yet to drop them at the same time.

@Brickner
I have not tried any other packet rates because the sensors output rate cannot be adjusted. I have done some testing on a slower PC and packets are dropped more often. I have also tried testing using a USB to Ethernet adaptor and going straight to the NIC and both configurations drop packets, although I think straight to the NIC is slightly better.

Also, I forgot to clearly state before that these are UDP packets I am trying to receive. Is it possible that Windows could be dropping the packets at some layer above the kernel before it gets to my application? Because the PacketsDroppedByDriver is always zero yet I am missing some packets with a very low CPU usage.

Thanks for the help!
Coordinator
Feb 20, 2015 at 7:58 PM
According to https://www.winpcap.org/docs/docs_412/html/structpcap__stat.html, the number of packets dropped by the interface is not yet supported.
So I don't think you can count on that.
I'll update Pcap.Net documentation.

Can you test what is the average packet loss in percent?
Can you see if this is affected by the the Kernel buffer or other properties?