Monitoring Http URLs as per User Click?

Sep 21, 2010 at 7:43 PM

Hi,

Is it Possible to capture http url Hit and click by the user By Using this FrameWork.?...

Many Thanks.

Sunny.

Coordinator
Sep 21, 2010 at 8:17 PM

Yes, it is possible.

You can capture all of the TCP port 80 traffic (using "tcp port 80" filter) and then look at the TCP payload and check the URL.

 

The next version of Pcap.Net will make parsing the HTTP part (from the TCP payload) much easier.

 

However, if the URL isn't fully in the first packet of the HTTP request, getting the URL will require some TCP reconstruction.

 

Boaz.

Sep 23, 2010 at 8:18 PM

When is the next version will be available? ;)

Coordinator
Sep 24, 2010 at 3:06 PM

That's a good question.

Probably after I'll return from New-Zealand (December).

Nov 17, 2010 at 3:46 PM

This is exactly what I need to do, sort data by which web app the packet was intended for or sent from.

I'm having some trouble parsing the TCP payload, however; none of the encodings I've tried have worked.  Could you point me in the right direction?

Also, I eventually need to record the amount of data sent for each request/response pair, meaning I will need to reconstruct TCP connections and associate the correct packets.  Do you have any suggestions or know of any articles that address this?

 

Amazing tool - thanks for the hard work brickner!

Nov 21, 2010 at 5:32 PM

Hello everyone,

I am new to this library but need it to see what website a user is visiting in a browser independent way.
I'm playing with the "interpreting the packets" example but do not know how to parse the packets into a readable form where I can see the url of the website that the user was requesting.
I think (as seen in this thread) that it must be very simple. Can anyone provide some example on this?

Thanks allot!

Nov 30, 2010 at 4:36 PM
Edited Nov 30, 2010 at 4:39 PM

I was using BinaryReader to get the data out

 

new BinaryReader(packet.Ethernet.IPv4.Tcp.Payload.ToMemoryStream()).ReadString()

 

 

It was too slow for live capturing though.

Also, that code is just a condensed example; I'm not getting a new reader and stream every time I need a string haha.

Dec 3, 2010 at 10:11 AM

Thanks for your reply. I tried it out but without any real success.
I now het all kinds of OutOfRangeExceptions when I run the code. Do you have some sample code that actually worked for you, so I can read the website's url?

Thanks!

Dec 3, 2010 at 3:19 PM
Edited Dec 3, 2010 at 3:19 PM

It's breaking because BinaryReader doesn't know when to stop.  There's also no safe 'HasNext' method, so I put it in a try catch and read until it broke.
Eventually I switched it to this:

if (packet.Ethernet.IpV4.Tcp.Payload != null)
{
    var text = Encoding.ASCII.GetString(packet.Ethernet.IpV4.Tcp.Payload.ToMemoryStream().ToArray());
    var match = Regex.Match(text.ToLower(), @"(?<=[get|post])\s(?/[a-z0-9\-/]*)\shttp/1\.1.*?host:\s(?[A-Za-z0-9\-\.]*)", RegexOptions.Singleline);
    if (match.Success)
    {
        Host = match.Groups["Host"].Value;
        Path = match.Groups["Path"].Value;
    }
}

If the request was made to somewebsite.com/foo/bar it will capture "somewebsite.com" and "/foo/bar"

Dec 3, 2010 at 3:50 PM

Alternatively, this regex expression will capture the entire path, including any parameters ("/foo/bar/index.html?x=5"):

@"(?<=[get|post])\s(?<Path>/.*)\shttp/1\.1.*?host:\s(?<Host>[A-Za-z0-9\-\.]*)"
And this one will capture the path, target file ("/foo/bar/index.html") but leave off any parameters:
@"(?<=[get|post])\s(?<Path>/.*?)[\?\s].*?http/1\.1.*?host:\s(?<Host>[A-Za-z0-9\-\.]*)"
 

Dec 3, 2010 at 7:54 PM

Cool. That does exactly what I need. Thanks a lot!

Dec 3, 2010 at 8:12 PM

Do you happen to know a way to select the right packet device automatically? I mean the one the user is using to browse the internet? In the example you must make a choice first and that is not what I want.

Dec 6, 2010 at 4:38 PM

No, I don't.  You could listen to each for a second and see which one gives you packets, but they could be using multiple interfaces (internet / intranet).
There's probably a better way to do it, or to ask Windows or something. 

Coordinator
Jan 5, 2011 at 8:38 PM

You are welcome to try the new HTTP Datagram to parse HTTP packets (available in Pcap.Net 0.8.0).

I hope this will make it easier to do what you need.

Apr 7, 2014 at 6:11 AM
Hi, does anyone know how to do it using the http datagram method?
Coordinator
Apr 12, 2014 at 8:10 AM
Hi javier_low,

Assuming you try to parse the first packet of an HTTP request, you can do:
((HttpRequestDatagram)packet.Ethernet.IpV4.Tcp.Http).Uri
To get the request Uri.

I hope this helps,

Boaz.