Miscellaneous ⇒ Search Engines :: Archives ⇒ Robots.txt and LEO :: Archived ⇒ Community Forums ⇒ CPG Dragonfly™ CMS
Forum IndexSearch Engines

Archived ⇒ Robots.txt and LEO

I've noticed that my robots.txt file, which I've never edited, says Disallow: /modules/ for Agent: *

To me this seems like it would keep bots out of the Forums folder, since it's a sub-folder of modules. But I'm pretty sure that my forums are crawled quite regularly. Am I mistaken in how it works? Does LEO have anything to do with this?

If I wanted to keep robots out of my forums for awhile, would I disallow: /modules/Forums, or just /Forums/, or something else?

How about my integrated G2 gallery, which is installed in public_html/gallery2, while the integrating module is installed in /modules/gallery/ but with LEO the URLs are like:



My Little Pony Arena

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Redhat Enterprise 4/Apache 2.2/MySQL 4.1.22/PHP 5/CPG

Disallow: /Forums/
Disallow: /gallery2/

For non-leo websites it's impossible Very Happy

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Fedora 25 / Apache 2.4.27 / MariaDB 10.1.26 / PHP 7.1.10 / Mercurial

All times are UTC