OrgName: Abovenet Communications, Inc
Address: 150 S. 1st Street, Ste 289
City: San Jose
I notice that most bots hit on robots.txt first but this thing didn't it went straight into coppermine.
This bot had about 20 concurrent connections. It was chewing through every single module I have. I just kept hitting refresh and I could see it cycling through all modules using all those connections. It's not listed as a bot in my who is where only in visitors log. Is this a good or bad bot? If good can we add it so the bots online can pick it up. If it's bad how can I ban it.
Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Joined: Feb 23, 2005
It is using several IPs and is one of the many developmental bots on the net. Always check the user agent definition in your logs as most good bots also provide an information or feedback link, like 22.214.171.124/feedback.html
Technically it is not a "bad" bot - it follows the rules of robots.txt so it is better to "Disallow" it there rather than ban the IP.
If you are having issues with too many concurrent connections, then I suggest you add "Crawl-delay: 5" where 5 is whatever digit in seconds you are comfortable with. This can be applied generally to all bots, or specifically to certain bots only. All "good" bots will respect this and it will ease pressure on your server - you will get searchen exposure on your terms.
Not all bots on the net are covered in DF definitions.