Ever since I updated to 9.1.1 the search bots seem to be browsing the sub-folders of the modules folder.
I saw some of them searching the "Your Account" pages and then even member profile pages. They seem to like looking at everything in there including the calendar, etc They never did this in 9.0.* unless I never noticed or it was not clearly identified on the whoiswhere block.
My robots.txt is default except for the bot-trap directory I have added:User-agent: Baidu
So, how would I stop them looking into sub-directories (i.e. modules that I don't want them looking at and indexing)?
Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Linux OS / Apache 1.3.37 / MySQL 5.0.24-standard / PHP 4.4.2 / DF 9.1.1