We forcefully moved to the new server and upgraded to Dragonfly CMS v10.
Please keep in mind we are working hard to get everything up and running, so please be patient.
Read/reply for issues in this forum topic /

Best regards, DJ Maze

Miscellaneous ⇒ Search Engines :: Archives ⇒ Bots Browsing Modules Sub-Folders :: Archived ⇒ Community Forums ⇒ CPG Dragonfly™ CMS

Archived :: Bots Browsing Modules Sub-Folders Revive this topic

Author Message
mazzgr
Newbie
Newbie

Offline Offline
Joined: Nov 28, 2005
Posts: 43
Post
Ever since I updated to 9.1.1 the search bots seem to be browsing the sub-folders of the modules folder.

I saw some of them searching the "Your Account" pages and then even member profile pages. They seem to like looking at everything in there including the calendar, etc They never did this in 9.0.* unless I never noticed or it was not clearly identified on the whoiswhere block.

My robots.txt is default except for the bot-trap directory I have added:User-agent: Baidu Disallow: / User-agent: *alexa* Disallow: / User-agent: Googlebot-Image Disallow: / User-agent: * Crawl-delay: 20 Disallow: /admin.php Disallow: /error.php Disallow: /admin/ Disallow: /blocks/ Disallow: /cache/ Disallow: /images/ Disallow: /includes/ Disallow: /language/ Disallow: /modules/ Disallow: /themes/ Disallow: /bot-trap/

So, how would I stop them looking into sub-directories (i.e. modules that I don't want them looking at and indexing)?

Thanks


mazzgr's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux OS / Apache 1.3.37 / MySQL 5.0.24-standard / PHP 4.4.2 / DF 9.1.1
View user's profile Visit poster's website
mazzgr
Newbie
Newbie

Offline Offline
Joined: Nov 28, 2005
Posts: 43
Post
Well, GoogleBot has been browsing the site now for 3 days as it is stuck looking at my calendar.

I assume he started back in 1901 and has now made his way up to 2025.

Will the bot ever stop and go away? Will this happen with every bot?


mazzgr's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux OS / Apache 1.3.37 / MySQL 5.0.24-standard / PHP 4.4.2 / DF 9.1.1
View user's profile Visit poster's website
Display posts from previous:
All times are UTC
Post new topic Revive this topic Forum IndexSearch Engines
Page 1 of 1

User Info

Welcome Anonymous



(Register)
Community

Support for DragonflyCMS in a other languages:

Deutsch
Español