Ever since I updated to 9.1.1 the search bots seem to be browsing the sub-folders of the modules folder.

I saw some of them searching the "Your Account" pages and then even member profile pages. They seem to like looking at everything in there including the calendar, etc They never did this in 9.0.* unless I never noticed or it was not clearly identified on the whoiswhere block.

My robots.txt is default except for the bot-trap directory I have added:

Code::

User-agent: Baidu Disallow: / User-agent: *alexa* Disallow: / User-agent: Googlebot-Image Disallow: / User-agent: * Crawl-delay: 20 Disallow: /admin.php Disallow: /error.php Disallow: /admin/ Disallow: /blocks/ Disallow: /cache/ Disallow: /images/ Disallow: /includes/ Disallow: /language/ Disallow: /modules/ Disallow: /themes/ Disallow: /bot-trap/

So, how would I stop them looking into sub-directories (i.e. modules that I don't want them looking at and indexing)?

Thanks

Well, GoogleBot has been browsing the site now for 3 days as it is stuck looking at my calendar.

I assume he started back in 1901 and has now made his way up to 2025.

Will the bot ever stop and go away? Will this happen with every bot?