Miscellaneous ⇒ Search Engines :: Archives ⇒ RufusBot :: Archived ⇒ Community Forums ⇒ CPG Dragonfly™ CMS
Forum IndexSearch Engines

Archived ⇒ RufusBot

Agent: RufusBot (Rufus Web Miner;

OrgName: Abovenet Communications, Inc
Address: 150 S. 1st Street, Ste 289
City: San Jose
StateProv: CA
PostalCode: 95113
Country: US

I notice that most bots hit on robots.txt first but this thing didn't it went straight into coppermine.

This bot had about 20 concurrent connections. It was chewing through every single module I have. I just kept hitting refresh and I could see it cycling through all modules using all those connections. It's not listed as a bot in my who is where only in visitors log. Is this a good or bad bot? If good can we add it so the bots online can pick it up. If it's bad how can I ban it.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):

We've got it running now and it has been for hours, I'm going to block it although the IP we're getting is

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
FreeBSD 5.2.1-RELEASE, Apache 1.3.33, MySQL 4.0.16, PHP 4.3.9, DragnFly

yeah no one responded so i just banned it and added it as a bad bot for my robots file.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):

It is using several IPs and is one of the many developmental bots on the net. Always check the user agent definition in your logs as most good bots also provide an information or feedback link, like

Technically it is not a "bad" bot - it follows the rules of robots.txt so it is better to "Disallow" it there rather than ban the IP.

If you are having issues with too many concurrent connections, then I suggest you add "Crawl-delay: 5" where 5 is whatever digit in seconds you are comfortable with. This can be applied generally to all bots, or specifically to certain bots only. All "good" bots will respect this and it will ease pressure on your server - you will get searchen exposure on your terms.

Not all bots on the net are covered in DF definitions.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):

All times are UTC