over the last month or so both MSN and yahoo bots have been generating loads of SQL Error emails to my admin account.

Basically the SQL server is timing out before there queries complete.

EG:

On /index.php?name=coppermine&file=displayimage&meta=lastup&cat=5&pos=73
While executing query "SELECT COUNT(*) FROM cms_session WHERE uname='9ac603e5921b2947ffad4324b9712d86'"

the following error occured: MySQL server has gone away

In: header.php on line: 44

Guest information:
User id: 1
Username: Anonymous
Admin: No
IP: 207.46.195.240
Host: msnbot-207-46-195-240.search.msn.com

I have been ignoring it buts its only getting worse, is there a way to ban these bots?

I have tried adding them manually, but have no idea what their UA Detect String is, or how to find it.

Is there a way to find that info???

you can increase you crawl delay in yourdomain.com/robots.txt it is set to 20 right now

Honestly tho your site stats dont reflect being hit hard enough to cause this. I would be searching for another cause or contacting the host so they can check for a problem with the server.

Sure, I understand that the MySql shouldn't time out, but the only issues we have with the site is these pesky bots, all else runs fine, which is why I was checking options to quash them so to speak.

Have extended crawl delay to 30, will see if that helps thanks.

Since it is msn and yahoo, you can also block them using robots.txt but Diz is right - your real problem is your server restrictions/setup.

your only getting 3-6 hundred hits an hour. I have built sites for clients using dragonflycms that get 200,000+ hits an hour and have no issues at all. If you got flood control on, there is no problem. Even if it is off, your hosting provider have taken measures to prevent ddos attacks against there networks. Problem is with your server. Check your servers logs and resource usage. Besides msn and yahoo generally follow robots.txt and with the default delay set at 20 that means there only getting you site for 3 hits each per minute. Trust me it is not the bots, it is the server.


and with 800+ other websites that are on that server. it might not be your site crashing mysql if it was trust me you would know host would suspend your account until offending script was removed.

Edit: 872 other sites are hosted on that server. lol a whole lot of room for mistakes.

MySQL gone away it may be caused by too many different reasons and it may be server or dragonflycms related, not an easy one.

Are the errors coming from the same query, or the query changes?

lots of different server queries, always MSN and yahoo though. Not one email since changing the crawl-delay to 30 though.

Thanks for getting me looking at the stats. Its made me aware that site traffic has doubled in recent months, and in fact looks on track to triple.

I have no doubt this is all just bot traffic. I look at the site right now and see 25 "visitors". I have no doubt that all 25 are actually bots that DF just doesn't recognise to class them as bots.

Which leads me right back to my original question, can, and how, do we add new bots to the system?

Secifially, how to identify their UA String?

admin/ security/ bots there is a form on the right for adding new bots to the list


as for getting there info your guess is as good as mine

df firstly look for their ip range then (not sure) at their ua

basically you need their ip range and add it to the security table ... but I'll prefer someone more experienced then me to answer

BUT ... from one year msn bot uses session id and it also masquerade its UA!

Ok so pretty much as I thought then, that page is useless and bots cannot be added manually, as there is no way to ascertain there UA String, and its a "requirement" to add them.

@Kinsman: Which "page" are you talking about, can you explain?
It's also not a requirement to add them as they would simply treated as normal users browsing the site, and ... when was ever said it was a requirement.

in Security under add bots, their UA String IS a requirement, there is no way to add any without it, as the entry is not accepted.

If noone knows how to ascertain their UA String, then whats the point of even being able to add them?

Oh security page, not statistics.

Just because microsoft doesn't follow standards it doesn't make our feature obsolete or "useless".

whats M$ got to do with it, I can't add ANY bots currently, as I have no clue how to find their UA String.

there are plenty I would like to add besides MSN, like Yahoo for starters.