Miscellaneous ⇒ Search Engines :: Archives ⇒ How can I restrict Bots to crawl Gallery module? :: Archived ⇒ Community Forums ⇒ CPG Dragonfly™ CMS
Forum IndexSearch Engines

Archived ⇒ How can I restrict Bots to crawl Gallery module?


Hello!

My hosting said me that the gallery is eating lots of resources, so I've decided to allow the gallery module only for registered users. After a few days of doing this, I continue seen in Who is where block bots crawling gallery pages, but not /coppermine.html page, are crawling /coppermine/displayimage/pid=****.html pages.

I need a way to restrict bots to crawl gallery pages that work.
How can I do this?

Many thanks and have a nice day!

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
FreeBSD/Apache 2.0.59/MySQL 5.0.33/PHP 5.2.3/CPGNuke 9.1.2.1


You can use a robots.txt file

You can find all the info on it here

Hope this helps Very Happy

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/Apache2/MySQL 4.1.15-Debian/PHP4 4.4.2-1build1/9.1.1


Of course the bots are still crawling the links - they are still indexed, but they don't get a result since they cannot access the module, so the server impact will be minimal, and eventually they will drop the links from their index.

Follow Wide's advice, but remember, this will still take some time to take effect since the bots do not check robots.txt every visit, and it will only be followed by genuine bots that obey the rules - rogue bots ignore robots.txt and you probably have many of those.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):


Thank you all.

Yesterday I've made the robots.txt solution and it seems that the activity decreased.

Thank you again.
Have a nice day!

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
FreeBSD/Apache 2.0.59/MySQL 5.0.33/PHP 5.2.3/CPGNuke 9.1.2.1

All times are UTC