Miscellaneous ⇒ Search Engines :: Archives ⇒ Robots.txt and LEO :: Archived ⇒ Community Forums ⇒ CPG Dragonfly™ CMS

FAQ

Moderators

Forum Index ⇒ Search Engines

Archived ⇒ Robots.txt and LEO

MajorHeadache

Offline
Joined: Mar 05, 2005
Posts: 263
Location: My Little Pony Arena

0 👍 / 0 👎

Tue Feb 27, 2007 8:08 pm

I've noticed that my robots.txt file, which I've never edited, says Disallow: /modules/ for Agent: *

To me this seems like it would keep bots out of the Forums folder, since it's a sub-folder of modules. But I'm pretty sure that my forums are crawled quite regularly. Am I mistaken in how it works? Does LEO have anything to do with this?

If I wanted to keep robots out of my forums for awhile, would I disallow: /modules/Forums, or just /Forums/, or something else?

How about my integrated G2 gallery, which is installed in public_html/gallery2, while the integrating module is installed in /modules/gallery/ but with LEO the URLs are like:

www.mlparena.com/gallery2/v/Art/

Thanks

My Little Pony Arena

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Redhat Enterprise 4/Apache 2.2/MySQL 4.1.22/PHP 5/CPG 9.1.2.1

DJ Maze

Offline
Joined: Apr 19, 2004
Posts: 6367
Location: http://tinyurl.com/5z8dmv

0 👍 / 0 👎

Tue Feb 27, 2007 11:22 pm

Disallow: /Forums/
Disallow: /gallery2/

For non-leo websites it's impossible Very Happy

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Fedora 25 / Apache 2.4.27 / MariaDB 10.1.26 / PHP 7.1.10 / Mercurial

All times are UTC

Post new topic Forum Index ⇒ Search Engines
Page 1 of 1