Router Based URL Logging - Want Complete Web History as a File

[commissarmo]

1. I am very intensive about web history. I visit c.500 websites/day and I like to have a complete, detailed, chronological history of my own web browsing.

2. To wit, I use addons to save full HTML copies of every page I visit.

3. But I would like to also have router-based URL logs which produce a file which has the URL of every single page I visit regardless of browser (I use 6 browsers simultaneously).

4. I am quiet confident this is possible, and read around it a bit, but I'm wondering what the 'best'/'easiest' way to accomplish this might be?

Some Keywords I've picked up reading about it: Tomato, Good Router (I have a terrible ISP issued one), WallWatcher, URLSnarf, DNS-level logging

5. I'm not sure how all these fit in, or fit together.

6. I would like to be able to have a DETAILED URL LOG (so if I visit www.tomshardware.com and then visit www.tomshardware.com/forum/42 AND then visit www.tomshardware.com/forum/43 I would like the URL log of every page.

I know some network logging information just lists the domain, that would NOT meet my objective.

7. Many thanks to any answers, you guys are always the best.

 

[bill001g]

Pretty much you have found the main tools, it is one of the many third party firmwares that log to a external server. It is not like the router has anyplace to save the data.

The bad/good news is most this stuff is very ineffective now. Thanks to edward snowden we now know the government was using similar logging methods. Almost every site is moving to HTTPS encrypted data streams. All you will see in the router is traffic to and from a ip on port 443. All the urls that you can look at in the http headers in unencrypted traffic are now gone. In many cases where multiple servers are hosted on a single ip you can not even be sure which site is being accessed.

You are going to have to do the logging on the end device before the encryption has taken place. I suspect we will see a migration to even more encryption which is why the government agencies are complaining so loud.

 

[commissarmo]

Thanks for the reply. Your message confuses me somewhat. Logging to an external server?

As to what you mean by encrypted data streams - are you suggesting that gathering the information at the router level (or the DNS level) won't work? It seems that people have done this before from what I have seen?

And as to my main objective, getting a complete URL/web history, I have to imagine that this is possible given that in principle... I could just copy and paste every URL every browser visits...?

 

[commissarmo]

I still want to learn more about this from either a router or DNS level logging perspective, but I'm wondering if this logging software is just an easy out?

http://www.pcpandora.com/monitored-activities/websites-visited/

Obviously, it runs on the local machine (which is ok) but I'd still like to get the information from the actual network traffic so I don't need to run additional programs on all my machines...

 

[bill001g]

Where do you expect a router to put the data it collects over a long period of time. Its not like it has a big disk drive, you might get it to write to a usb nas device but that is not a feature in most the tools. You could likely modify it. Still the router would never have the cpu power to produce any kind of report.

It "USED" to work but now that people have seen how this type of tool can be misused all the data is being encypted. The router will see nothing but IP addresses and encrypted data packets. You might see dns queries but a pc tend to cache this data so you only would see it when a new site was accesses.

Load wireshark and look at the difference you see on a site like this one that is not encrypted and even something simple like a google search that is encrypted.

There is huge pressure for all sites to encrypt their data because of government spying.

 

[commissarmo]

I think I understand what you're saying, but honestly my question has nothing to do with government spying or encryption, which you keep bringing up. I understand that you're saying that the IPs may not be readable at the router level (though I have read numerous reports of people doing exactly what I want, and then using software (LinkLogger I think is another one) to translate the IPs to URLs).

In either case, I think I'd like to let others weigh in.

 

[Alabalcho]

OK, let simplify it. When you access your bank, your bank does not send the info in clear text - it is encrypted, so the router (and the "bad" guys behind the router) sees only encrypted info.

Something similar happens with "regular" web sites as well - they tend to use https instead of http. So, the router will not see that you went to "http://mywebsite.com", it will see a connection over port 443 to IP address 123.234.56.78, and nothing else.

Edit: Read about pfsense - and find a device (a PC) to run it instead of your router.

 

[commissarmo]

Ok - so you're saying that basically what I want to do is impossible AT THE ROUTER LEVEL. (still I'm not sure why people report that this works then? http://www.howtogeek.com/68886/how-to-configure-your-router-for-network-wide-url-logging/)

That PFsense appliance looks very interesting. It seems to be a dedicated firewall/security/VPN server type box, if I'm understanding right? Presumably if I set up an old computer with the right software I could replicate that (seems the cheapest one is US$450, plus their softs and support could be worth it).

Would THAT allow me a hardware solution to log all of my own URLs? Again, I know this is possible in the trivial sense, and indeed, I've already found monitoring software that would create a URL log for me listed above amongst others). But I'd like to see if there's a simple, hardware method.

Any other possibilities out there?

 

[bill001g]

Look at the date on those posts. This is all old outdated information. More and more companies are announcing that all traffic will be encrypted to their web sites. Yahoo just announce even a better email encryption.

Before you run off chasing solution start with the easy option. Load wireshark on your PC and see what you are sending and receiving. Wireshark does a great job of decoding anything that is not encrypted. Run some encrypted and non encrypted web sites you will soon see what we are talking about. You can actually see the urls in non encrypted traffic.

You might get the site names by running reverse dns lookup but you will never see the actual URLS in encrypted data.

PFSENSE can collect and produce reports on unencypted data but it still can do nothing about encrypted data. The problem with using a firewall is now you must place a wireless AP in front of it so your wireless traffic does not go directly to the router bypassing the firewall. It really doesn't matter a lot you either put a dedicated server in as a firewall to collect the data or you use the router to collect the data and send it to a dedicated server. There are a bunch a ways to collect data and send it from the router one of the more common is called netflow.

Still what you will likely see from any tool you get is a report that says xxxx amount of data was send/received from ip xx.xx.xx.xx on port 443. If you are lucky and xx.xx.xx.xx resolves to a unique DNS name you might get the actual site but with akamai used most times you can not even tell what site it really is.

 

[commissarmo]

Using Fiddler (there are other services, though I found this one the most effective), I was able to get a log of visited URLs on my machine. It is able to read HTTPS sites (though not perfectly), and it creates a lot of extraneous information which can be filtered (though I haven't figured out how to do all this filtering yet), but in general it is the solution to the initial question of getting a log of URLs which is complete.

Again, it's hardly perfect, and indeed I have determined that getting a log of visited websites for me to keep track of my history is likely to be best accomplished by URL logs + web history + using addons that allow me to save complete webpages as a composite solution since there isn't a good solution to keeping complete records with just URL logging.

On the other hand, many of the comments above didn't even mention this as a potential solution, so I had to go to other forums before this came up...

 

[PT-115]

Registered to post on a 4 yo thread. Not often that I am so compelled. I was very annoyed to see how OP treated Titan who was obviously an expert giving free advice. And i was astonished that he had the patience and kind heartedness to let OPs petulance and entitlement go unchecked. Titan, you deserve a medal and i look forward to reading and learning from such a gifted teacher with 4 more years of knowledge.