General ⇒ Announcements :: Archives ⇒ HTTrack banned :: Archived ⇒ Community Forums ⇒ CPG Dragonfly™ CMS
Forum IndexAnnouncements

Archived ⇒ HTTrack banned


If you were thinking why our website was slow or down for the last weeks, i now have solved a part of it.

Someone in Germany using t-dialin.net (T Online) is harvesting our website using HTTrack webcopier.
Due to HTTrack we also found some sort of bug in PHP that is marked as bogus. The issue here is the HTTP HEAD method.
Normaly we only reply on GET and POST since HEAD requests are not used much. However HTTrack is requesting HEAD before each pageview.
Since HEAD is passed the PHP script is executed two times for one page.
That in combination with a harvester has seased the webserver rapidly.

Therefore i advice you to ban HTTrack using .htaccess
RewriteCond %{HTTP_USER_AGENT} "HTTrack" [NC] RewriteRule ^.*$ - [F]

I have added a fix in cvs to exit() on a HEAD request inside cmsinit.inc as well.
dragonflycms.org/cvs/h...9.100-9.99

As a side note this is HTTrack's UA
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Fedora 25 / Apache 2.4.27 / MariaDB 10.1.26 / PHP 7.1.10 / Mercurial


Nice catch. Cool So the cvs updated, csminit.inc takes care of it all or does .htaccess still need to be modified? Based on what I am seeing .htaccess will need to be modified, but just double checking.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
[CentOS release 4.6 (Final)] | [Apache 1.3.37] | [MySQL 4.1.21-standard-log (client: 4.1.21) | [PHP 4.4.7] | [DF 9.2.1] | [FPro 2.0.2]


cmsinit.inc only exits on a HEAD request so that the script doesn't get executed for nothing.

The .htaccess bans the HTTrack bot.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Fedora 25 / Apache 2.4.27 / MariaDB 10.1.26 / PHP 7.1.10 / Mercurial


DJ Maze wrote

As a side note this is HTTrack's UA
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)


I have enabled blocking of unidentified useragents. Am I correct that HTTrack qualifies and that my bandwith will be saved?

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Linux/Apache/5.0.24/5/9.1 CVS


if any UA that are not verify in the security Class when you turn on not known UA, its block them.

Firefox is my Favorite Browser

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Apache/1.3.34 (Unix)/4.0.25-standard/4.4.1/CVS


Awesome, I am glad we have such a dedicated team!

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Red Hat Linux 7.3/Apache 1.3.33 /MySQL4.0.22-standard/PHP4.3.11/CPGNuke 9.0.6.1

All times are UTC