Capturing request body

Aug 11, 2011 at 11:22 PM

I am writing an application that will analyse data users enter on forms. Ive looked at using the FiddlerCore.dll but I think there are some issues with that that could mess up user's web browsers (it messed mine up twice on two different pcs) which would be unacceptble for a program I would like to sell commercially at some point. So I have started looking at pcap.net.

 

I need to capture the body of http requests to retrieve the data they have entered. I have stepped through the example programs and one of the things I did was (in the function PacketHandler(Packet packet) in InterpretingThePrograms) -

I output the packet.Buffer array to txt files. I tried this with a few encodings. For example when I visited google.com and google/news and output the data with - System.Text.Encoding.ASCII.GetString(packet.Buffer) I got the following text -

http://tinypic.com/r/dob4p5/7 (firefox wouldnt let me paste it in here, because of the binary data I think)

When I used System.Text.Encoding.ASCII.GetString(packet.Buffer) I got this text -

http://tinypic.com/r/2e1ejra/7



I have a few questions on my results -

1. What exactly is the packet.Buffer - Is it an IP packet header like this http://www.freesoft.org/CIE/Course/Section3/7.htm? Or is it an entire IP packet, both header and body?

2. Are the packets coming through the PacketHandler fucntion both requested and received packets?

3. I tried submitting a few forms on some pages and checking out the captured ascii and utf8 text and although it's nearly all binary I couldn't find any instances of the data I sent and the packets themselves seemed to small to contain all the data so it leads me to believe the packet.Buffer variable doesn't actually contain the posted data? Is this correct?

4. Any tips on how to retrieve http requests, not just the headers, the bodies too?

I appreciate any help anybody is able to offer me on this. Cheers.

Coordinator
Aug 12, 2011 at 7:10 AM

Hu nukenuts,

 

1. packet.Buffer gives you the entire packet buffer including Ethernet, IPv4, TCP and anything else you have in this packet.

Assuming it's an ASCII string is a mistake.

2. The packet you get are all the packets your device is handling. Use a filter to decide which packets you want to handle.

3. packet.Buffer contains everything, but you probably don't want everything, you want all the data. You probably want to reconstruct the TCP stream and converting only its data to a string.

4. This would be hard to do for requests that are not very small without TCP reconstruction. If the request is entirely in one packet, you can use packet.Ethernet.IpV4.Tcp.Http.Body.

 

I hope this helps,

 

Boaz.

Aug 13, 2011 at 11:51 AM

Hi mate, cheers for the information. I have a much better understanding of the code now, spent alot of time yesterday stepping through it and reading up on tcp/ip and ethernet layers.

I am now receiving only the packets I want through filtering however I have one or two follow questions if you wouldn't mind helping me with -

1. I need to capture packets going to 'www.examplesite.com' so I checked its IP address in an IP lookup and it was 123.456.abc.def. So I set my filter to BerkeleyPacketFilter filter = communicator.CreateFilter("dst host 123.456.abc.def");

But I then realised that www.examplesite.com can be accessed from multiple IP address so I would need to monitor them all. So does that mean I should set my filter to, for example, "dst host 123.456.abc.def or dst host 987.654.321.000 or dst host abc.def.ghi.jkl" or would just setting it too "dst host www.examplesite.com" automatically capture all those addresses?

 

2. When I use "dst host 123.456.abc.def" I am capturing all tcp traffic to that address whereas I only actually want the tcp traffic that contains http. Is there a way to set a filter to capture only http or do I just have to capture all tcp traffic and then filter it myself by checking the packet structure in the PacketHandler function to see if it contains http - if (packet.Ethernet.IpV4.Tcp.Http != null)

Coordinator
Aug 13, 2011 at 5:34 PM

Hi nukenuts,

 

1. Actually, that's a very good question.

I have always used the option with the different IPs.

However, the option with the domain name will also work.

The problem with the domain name option is that it is not exactly clear how does it work.

Remember that packets don't contain what domain they are sent to, only what IP they are sent to. In order to translate a domain to an IP, a DNS request is sent. The DNS request can return many IPs and all of them will be captured.

The problematic part is that DNS isn't static. What happens if an IP becomes invalid for a specific domain or if an IP is added to a specific domain.

Also, different DNS servers might give different results, so what happens if the filter uses one DNS server while the packets are based on another DNS server.

I don't know how often does the filter decide what DNS request to rely on. My guess is that it sends one request at the beginning to the default DNS server and uses the IPs it get, and then you should be careful because the packets you want to capture can be based on other DNS servers and also the IPs may change over time, and the filter will not be updated.

That is why I suggest to use the different IPs option over the domain name option.

2. You can use "dst host 123.123.13.123 and tcp dst port 80", assuming you want to capture HTTP requests to the server, and assuming your HTTP will work on port 80. Nothing guarantees that there are all tcp dst port 80 packets are actually HTTP but this will definitely reduce the number of packets that will be captured that are not HTTP.

 

I hope this helps,

 

Boaz.

Aug 14, 2011 at 1:51 AM

Cheers mate, some good info there. Im going to see if there is any way to find all possible ips for a certain site and if that's not possible I suppose I can just capture all outbound tcp port 80 traffic and then filter it myself by checking the HttpRequestDatagram.Uri.