Home Private Messages Search
CPG Dragonfly™ CMS
Toggle Content
 
Forums ⇒ Miscellaneous ⇒ Search Engines :: Archives ⇒ Bots Browsing Modules Sub-Folders :: Archived


Bots Browsing Modules Sub-Folders :: Archived
Questions and issues with search engines, SEO, bots, meta tags.
Post new topic    Revive this topic    Printer Friendly Page     Forum Index ⇒  Search Engines

Topic Archived View previous topic :: View next topic  
Author Message
mazzgr
Newbie
Newbie

Offline Offline
Joined: Nov 28, 2005
Posts: 43

PostPosted: Mon Feb 19, 2007 11:49 am
Post subject: Bots Browsing Modules Sub-Folders

Ever since I updated to 9.1.1 the search bots seem to be browsing the sub-folders of the modules folder.

I saw some of them searching the "Your Account" pages and then even member profile pages. They seem to like looking at everything in there including the calendar, etc They never did this in 9.0.* unless I never noticed or it was not clearly identified on the whoiswhere block.

My robots.txt is default except for the bot-trap directory I have added:
Code::
User-agent: Baidu
Disallow: /

User-agent: *alexa*
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: *
Crawl-delay: 20
Disallow: /admin.php
Disallow: /error.php
Disallow: /admin/
Disallow: /blocks/
Disallow: /cache/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /themes/
Disallow: /bot-trap/

So, how would I stop them looking into sub-directories (i.e. modules that I don't want them looking at and indexing)?

Thanks


mazzgr's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux OS / Apache 1.3.37 / MySQL 5.0.24-standard / PHP 4.4.2 / DF 9.1.1
Back to top
View user's profile Visit poster's website
mazzgr
Newbie
Newbie

Offline Offline
Joined: Nov 28, 2005
Posts: 43

PostPosted: Tue Feb 20, 2007 11:01 am
Post subject: Re: Bots Browsing Modules Sub-Folders

Well, GoogleBot has been browsing the site now for 3 days as it is stuck looking at my calendar.

I assume he started back in 1901 and has now made his way up to 2025.

Will the bot ever stop and go away? Will this happen with every bot?


mazzgr's server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS)
Linux OS / Apache 1.3.37 / MySQL 5.0.24-standard / PHP 4.4.2 / DF 9.1.1
Back to top
View user's profile Visit poster's website
Display posts from previous:   
Post new topic    Revive this topic    Printer Friendly Page    Forum Index ⇒  Search Engines
Page 1 of 1
All times are GMT

Archive Revive
Username:
This is an archived topic - your reply will not be appended here.
Instead, a new topic will be generated in the active forum.
The new topic will provide a reference link to this archived topic.
 
   Toggle Content User Info

Welcome Anonymous

Nickname
Password
(Register)

   Toggle Content Mercurial Commits
· Added: Security sessions management Bugfix: infinite loop in $leochanged check
· Bugfix: Poodle\Mail\SMTP failed sending Bugfix: Security system undefined data B
· Added: Poodle\HTTP\URLInfo Bugfix: Poodle\HTTP\Request headers parsing and body
· Bugfix: \Poodle::getKernel()->IDENTITY can be undefined in debugger Change: spli
· Changed another L10N->get() to L10N->dbget() for CFG->global->slogan and CFG->gl
· Changed L10N->get() to L10N->dbget() for CFG->global->slogan and CFG->global->si
· Improved L10N admin
· Update version due to L10N changes
· Bugfix: L10N date parsing Added: L10N dbget() check v9 constants
· Merge

read more...

   Toggle Content Community

Support for DragonflyCMS in a other languages:

Deutsch
Español

You are seeing squares or questionmarks on this page?

All content of this website is copyrighted by the Creative Commons NC-SA
The logos and trademarks used on this site are the property of their respective owners
We are not responsible for comments posted by our users, as they are the property of the poster.
Our server runs on a P3 1.2GHz with 512MB RAM with no accelerators
Support GoPHP5.org
Interactive software released under GNU GPL, Code Credits, Privacy Policy