| Author |
Message |
farbird
Joined: Sep 19, 2005
Posts: 121
|
 Post subject: Bots browing my cpgnucalendar all the way to 2099
Posted: Tue Mar 28, 2006 10:36 am |
 |
I have many bots that keep wasting my precious bandwidth by crawling all over my cpgnucalendar
my cpg site is in a subdomain
www.domain.com/subdomain/index.php
in the subdomain, i already prevent bots from going into the modules directory and yet, in the who where block, i keep seeing bots and crawlers in the cpgnucalender..
how to stop them?
some are msn and yahoo bots...
at times i do get google bots going there as well.
farbird's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
9.0.6.1 |
|
| Back to top |
|
 |
farbird
Joined: Sep 19, 2005
Posts: 121
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Tue Mar 28, 2006 10:38 am |
|
using robots.txt
farbird's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
9.0.6.1 |
|
| Back to top |
|
|
sarah
Joined: Mar 25, 2005
Posts: 2131
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Tue Mar 28, 2006 12:42 pm |
|
They can't get to the modules directory anyway. Try this:
Disallow: /CPGNuCalendar*
_________________
Diagon Alley - Top Design
sarah's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux/1.3.37/4.1.21-standard/4.4.4/9.1.1 |
|
| Back to top |
|
|
farbird
Joined: Sep 19, 2005
Posts: 121
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Wed Mar 29, 2006 3:06 am |
|
not helping..
yahoo/intomi bots still in my cpgcalendar with the robots.txt updated above
farbird's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
9.0.6.1 |
|
| Back to top |
|
|
sarah
Joined: Mar 25, 2005
Posts: 2131
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Wed Mar 29, 2006 3:23 am |
|
I don't think they read it every time they visit, I think they cache it for a while or something.
_________________
Diagon Alley - Top Design
sarah's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux/1.3.37/4.1.21-standard/4.4.4/9.1.1 |
|
| Back to top |
|
|
farbird
Joined: Sep 19, 2005
Posts: 121
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Wed Mar 29, 2006 3:26 am |
|
my subdomain stats in cpanel of my webhost states that these bots take up 9-10 mb of bandwidth each...
farbird's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
9.0.6.1 |
|
| Back to top |
|
|
alva
Joined: May 31, 2005
Posts: 1150
Location: The Netherlands
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Wed Mar 29, 2006 6:18 pm |
|
Until the bots have digested your new robots.txt a solution could be to set the module to members only...?
alva's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux/Apache/5.0.24/5/9.1 CVS |
|
| Back to top |
|
|
djdevon3
Joined: Aug 05, 2004
Posts: 4365
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Thu Mar 30, 2006 2:44 am |
|
One thing I've noticed is that bots love calendars...more than anything else I've ever seen. I have no idea why. Maybe bots are designed to look for the most current information so when they see a calendar they chew through the info like it was candy.
djdevon3's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux/1.3.33/4.4/4.3.11 |
|
| Back to top |
|
|
farbird
Joined: Sep 19, 2005
Posts: 121
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Thu Mar 30, 2006 3:01 am |
|
setting calendar module to registered members only..
pity...
farbird's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
9.0.6.1 |
|
| Back to top |
|
|
alva
Joined: May 31, 2005
Posts: 1150
Location: The Netherlands
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Thu Mar 30, 2006 12:12 pm |
|
farbird wrote:
setting calendar module to registered members only..
pity...
You can still have the blocks show some info, only the links won't work now.
But a better solution would be this:
I'm too busy right now to dig in CPGNuCalendar's code, but shouldn't this be done by setting Dragonfly's $METATAGS['robots'] in parts of the CPGNuCalendar code just before require_once('header.php');?
alva's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux/Apache/5.0.24/5/9.1 CVS |
|
| Back to top |
|
|
farbird
Joined: Sep 19, 2005
Posts: 121
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Thu Mar 30, 2006 2:38 pm |
|
even when i restrict the block and modules to registered members only, the bots still crawled all over the calendar module...
exasperated....
farbird's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
9.0.6.1 |
|
| Back to top |
|
|
run0
Joined: Jun 28, 2004
Posts: 1559
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Thu Mar 30, 2006 5:22 pm |
|
theyre still following old links, be patient. if theyre using 10mb of bandwidth thats not so bad...
_________________
run0's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux/1.3.33 (Unix)/4.0.22-standard/4.3.9/DF 9.x |
|
| Back to top |
|
|
farbird
Joined: Sep 19, 2005
Posts: 121
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Fri Mar 31, 2006 12:35 am |
|
ok..
will wait out..
darn the bots.
hehe
can't live with them, can't live without them
farbird's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
9.0.6.1 |
|
| Back to top |
|
|
sarah
Joined: Mar 25, 2005
Posts: 2131
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Fri Mar 31, 2006 2:43 am |
|
They're just crawling a message that tells them they have to be registered to view that page. They love to do the same thing with for example, Private Messages until you disallow that too.
_________________
Diagon Alley - Top Design
sarah's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux/1.3.37/4.1.21-standard/4.4.4/9.1.1 |
|
| Back to top |
|
|
DaleHutch
Joined: Sep 05, 2004
Posts: 120
Location: 27°58'N 82°32'W
|
Post subject: Re: Bots browing my cpgnucalendar all the way to 2099
Posted: Wed Apr 25, 2007 2:30 am |
|
I'm getting slammed also. I've also tried everything.
_________________
A bug in the code is worth two in the documentation.
DaleHutch's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Ubuntu Linux 8.04.1/Apache 2.28/MySQL 5.0.51/ PHP 5/DF 9.2.x |
|
| Back to top |
|
|
Forum FAQ
Search
Log in to check your private messages
Login
