Is there any way to find out how many hits you receive from bots, on an IIS6 site?

Hi folks,

Is there any way to figure out how many bots (even if it's just one type of bot) that hits our site, on a daily, weekly and monthly average?

we have the standard IIS6 web site logs in lots of files.

Any tricks or recommendations?

Update

Program needs to be .NET based. No php or extra ugly packages, please. Free would be a bonus.

From serverfault Pure.Krome

• Many reporting packages can report on this.

  For instance, Awstats says

  AWStats is able to detect robots visits.[...] "Robots visitors" are reported separately in the "Robots/Spiders visitors" chart.

  EDIT: AWStats is free, written in Perl, and works just fine on Windows and with IIS logs.

  Pure.Krome : I've updated my original post. Bascially, i can't use PHP. has to be .NET based ... and preferably free.

  WerkkreW : AWStats is based on Perl, not PHP.

  Jacco : AWstats has only very basic bot-detection. Anything that has a non-robot useragent and does not request robots.txt is not recognised as a robot.

  Pure.Krome : re: Php/Perl. opps. I ment Perl, but same thing.. i don't want to install extra stuff besides the app iteself.
  From Paul

• Here is a pretty good list of known bot user-agents. I am not familiar with any free web log analytic tools written in .NET, but you could easily parse the logs yourself if all you are looking for are bots.

  http://www.user-agents.org

  From WerkkreW

• I use WebLog Expert when I want to examine our logs. There is a free version (lite) but that doesn't include the scheduler or command line functionality. The software itself runs as a desktop app (either remotely with ability to FTP logs or can be installed on the server), and can output reports in CSV, PDF and HTML.

  From Dan

• Hello,

  why not outsourcing your web stats ? With solution like sitemeter, you just put an extra javascript, and they will give you stats on your site.

  Web stats may be a pain for admin, doesn't always worth it to implement in house solution

  From Mathieu Chateau

• Google Analytics anyone? Free, Javascript based, so no need for server side languages. In depth coverage of hits, bots too.

  Pure.Krome : I'm already using Google Analytics. It has stats on it's own robot hits?

  Gr33n3gg : Hm, sorry I misled you, I thought GA did bots too. The bots/crawlers can't execute Javascript, Awstats would work for you since it reads the server logs.

  Pure.Krome : I didn't know this, but Google Analytics actually records how many times it hits your site. That said, it doesn't count the OTHER bots, though.
  From Gr33n3gg