Merging Pcaps with PcapDotNet

Dec 18, 2012 at 5:27 AM

What is the best approach for merging the multiple Pcap files. 

Is there anything faster then rebuilding every packed with the PacketBuilder?

 

Txs

Dec 24, 2012 at 5:45 AM
Edited Dec 24, 2012 at 6:02 AM

u can build it yourself, it's structure very simple

you don't even need pcaplib, try this http://wiki.wireshark.org/Development/LibpcapFileFormat

however i doubt this would be faster if u are going to analyze packets

Dec 24, 2012 at 7:48 AM

No, it is not exactly what I was looking for.

You see, building PCap is not a challenge. And while your choice of native pcaplib instead of its managed wrapper (PcalDotNet) contradicts the nature of this discussion thread (PcalDotNet forum) it is technically a valid approach.  :o) But I was just looking for the most efficient way of doing this.

Anyway, I ended up merging the files manually by reading all packets from all files, sorting them by timestamp and dumping all of them to a new file. But I do it with PcapDotNet. Which is quite good for reading or building arbitrary packets (particularly in conjunction with LINQ).

However PcapDotNet API is badly designed for the packet modification: no public constructors tacking data buffers for Layers, no public modifiers for primitive Packet fields like Timstamp, no ability to create an arbitrary empty Pcap (dump file). Thus I had to use Reflection as in some other cases before. Yes PcapDotNet API is extremely conservative to say at least. 

Thus the algorithm is:

 - Identify the oldest pcap input file. 

- Open oldest pcap input file and use it to create derived output pcap file.

- Read all packets from all pcap files

- Sort all packets by Timestamp

- Reset private _timestamp field for all packets using Reflection. This is needed to be done in order to reset packets logical Timestemps, which are actually not timestamps but rather timeoffsets. PcapDotNet just converts them nicely in a DateTime type but under the hood it is still a timestamp.

- Dump all packets to the output pcap file.

 

Coordinator
Dec 28, 2012 at 2:37 PM

Hi taras_b,

 

If I understand correctly, the only issue you had was to change the Timestamp of a Packet instance.

Timestamps in pcap files are always offsets from January 1st, 1970, 00:00:00 UTC. For programming in .NET I think it's easier to use DateTime values.

Can you elaborate on why you needed to change the Timestamps?

Packets in Pcap.Net are immutable objects, and this is why there's no simple way of changing the timestamps, yet.

I'm considering adding an easier way to change a Packet timestamp, currently, if you know the packets are Ethernet packets, you can do it by doing the following:

 

Packet newPacket = PacketBuilder.Build(someTimestamp, packet.Ethernet.ExtractLayer(), packet.Ethernet.Payload.ExtractLayer());

 

So let me know what is the use case for changing the timestamp and I'll consider an easier way to change timestamps.

 

I hope this helps,

 

Boaz.

Dec 30, 2012 at 1:05 AM

Hi brickner,

Thank you for finding the time to look at it.

Yes you are correct about  the timestamp. It is indeed an absolute time offset but not a relative one as I incorrectly concluded initially. Thus resetting the packet timestamp is unnecessary.

>I'm considering adding an easier way to change a Packet timestamp...

Thank you. Yes it makes sense to allow one to changes the timestamp if required. Even if I do not need it for solving my problem. :o)

>if you know the packets are Ethernet packets, you can do it by doing the following:....

This is where I have a problem with the current API.

- First of all I have a heterogeneous pcaps and there is the warranty that not all of them are the Ethernet packets.

- Second problem is more "fundamental". I have a packet object. The object can be easily persisted thus I see no logical reason to disassemble this object into extracted layers and then reassemble it again with the builder. Unless of course you builder based proposal was specifically about solving the timestamp resetting problem.  

- The third problem is that the PacketDumpFile cannot be constructed independently and has to be "derived" from the existing pcap file. The resulting Pcap file also has a timestamp in the header. This timestamp is independent of the packets timestamps. However because this timestamp still logically depends on the first packet timestamp (packet timestamp must be equal or "newer" then the Pcap timestamp) it is desirable to be able to read and modify the Pcap timestamp.

Thus the constructor "new PacketDumpFile(string fileName, DateTime timestamp)" would solve the problem.

Anyway I ended up with the not the most elegant but an adequate solution based on using the input pcap file with the "oldest" packet for the creation of the output dump file:

    public class PcapMerger
    {
        public string Merge(string outFile, params string[] inFiles)
        {
            if (inFiles.Length == 0)
                return outFile; // To allow Fluent API

            PacketDumpFile dumpFile = null;
            var allPackets = new List<Packet>();

            foreach (string file in inFiles.OrderBy(x => GetFirstPacketTimestamp(x)))
            {
                using (PacketCommunicator comm = new OfflinePacketDevice(file).Open(-1, PacketDeviceOpenAttributes.Promiscuous, 700000))
                {
                    var packets = comm.ReceivePackets(-1).ToArray();

                    allPackets.AddRange(packets);

                    if (dumpFile == null)
                    {
                        dumpFile = comm.OpenDump(outFile);
                    }
                }
            }

            foreach (Packet item in allPackets.OrderBy(x => x.Timestamp))
            {
                dumpFile.Dump(item);
            }

            dumpFile.Dispose();
            return outFile;
        }

        private DateTime? GetFirstPacketTimestamp(string pcapFile)
        {
            using (PacketCommunicator comm = new OfflinePacketDevice(pcapFile).Open(-1, PacketDeviceOpenAttributes.Promiscuous, 700000))
            {
                Packet packet = comm.ReceivePackets(1).FirstOrDefault();
                if (packet == null)
                    return null;
                else
                    return packet.Timestamp;
            }
        }
    }

Thank you,

Taras