Due to all the detected bad bots and harvesters i've decided to open up our current list to all visitors of this website for reference and usage.

Do keep in mind that ip's change and that it might be possible that you ban future clients so use the list carefully.

Most of them are either detected due to rotating UA or flooding.
Entries WITHOUT comment identify themselves as legal browser which means they FAKE it.

Code::

60.229.210.127 61.152.169.27 64.136.49.226 66.55.138.98 66.111.225.95 # Hurricane Electric 66.220.7.137 66.220.7.150 66.220.7.153 66.220.7.163 66.220.7.184 66.220.7.190 66.220.7.244 66.220.20.6 66.220.20.25 66.220.20.26 66.230.177.58 # MSIE 6.0 66.246.218.107 66.249.66.229 68.41.127.17 # HTTrack 68.96.251.167 72.150.30.116 80.77.86. 82.165.181.61 82.248.208.129 132.199.145.148 203.144.143.3 # Medusa havester 203.144.160.246 # Medusa havester 203.162.3.147 # hackers proxy 203.210.241.140 207.44.242.75 209.66.122.10 211.206.89.94 221.149.161.5 222.252.232.224 222.252.234.

Feel free to add yours to this list

Code::

193.77.185.137 Java/1.5.0_04 213.93.106.254 Mozilla/4.0 24.118.119.182 Java/1.5.0_04 24.67.223.55 Mozilla/4.0 61.120.98.99 Java/1.4.1_04 62.145.145.6 Mozilla/4.0 62.163.32.148 Java/1.5.0 62.163.48.149 Mozilla/4.0 62.194.14.170 Mozilla/4.0 62.194.17.125 Mozilla/4.0 62.194.7.33 Mozilla/4.0 65.6.15.98 Java/1.5.0_04 66.103.145.34 Java/1.4.1_04 66.252.133.170 Mozilla/4.0 68.231.30.238 Mozilla/4.0 69.163.154.50 Java/1.4.1_04 70.148.166.166 Java/1.5.0_04 70.85.116.229 http://www.jaja-jak-globusy.com/ 70.86.244.194 Mozilla/4.0 81.208.60.207 Mozilla/4.0 81.73.137.226 Mozilla/4.0 83.45.133.255 Mozilla/4.0

This whole list, with the duplicates from DJMaze's list removed is from my list of bots that ignore robots.txt

Here's a post spammer that promotes "enlargements" thru forums and "contact us"

203.160.1.38

Here are a few that like to harvest that gave me issues



DROP all -- 66.191.37.0/24 anywhere
DROP all -- 67.150.0.0/16 anywhere
DROP all -- 66.52.0.0/16 anywhere
DROP all -- 65.148.0.0/16 anywhere




This list is a great resource

That 207.44.242.75 IP on DJMaze's list is some hosting company. Why are they flooding my site?

either a bot or a virus but i think the last one

got this mail to day:

Code::

A bad robot hit /bot-trap/ 2006-04-27 (Thu) 05:24:58 address is 216.255.189.226, agent is Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

dont know who owns it.

OrgName: InterCage, Inc.
OrgID: INTER-359
Address: 1955 Monument Blvd.
Address: #236
City: Concord
StateProv: CA
PostalCode: 94520
Country: US

ReferralServer: rwhois.intercage.com:4321/

NetRange: 216.255.176.0 - 216.255.191.255
CIDR: 216.255.176.0/20
NetName: INTERCAGE-NETWORK-GROUP2
NetHandle: NET-216-255-176-0-1
Parent: NET-216-0-0-0-0
NetType: Direct Allocation
NameServer: NS10.INTERCAGE.COM
NameServer: NS11.INTERCAGE.COM
Comment:
RegDate: 2005-09-20
Updated: 2005-09-20

OrgAbuseHandle: ABUSE735-ARIN
OrgAbuseName: Abuse Department
OrgAbusePhone: +1-925-550-3947
OrgAbuseEmail: abuse @ intercage.com

OrgNOCHandle: NETWO670-ARIN
OrgNOCName: Network Operations
OrgNOCPhone: +1-925-550-3947
OrgNOCEmail: noc @ intercage.com

OrgTechHandle: INE4-ARIN
OrgTechName: IP Network Engineering
OrgTechPhone: +1-925-550-3947
OrgTechEmail: ipeng @ intercage.com

# ARIN WHOIS database, last updated 2006-04-26 19:10
# Enter ? for additional hints on searching ARIN's WHOIS database.

OrgName: InterCage, Inc.
OrgID: INTER-359
Address: 1955 Monument Blvd.
Address: #236
City: Concord
StateProv: CA
PostalCode: 94520
Country: US
Comment: InterCage, Inc. Network Group
RegDate: 2004-10-12
Updated: 2005-04-14

ReferralServer: rwhois.intercage.com:4321/

AbuseHandle: ABUSE735-ARIN
AbuseName: Abuse Department
AbusePhone: +1-925-550-3947
AbuseEmail: abuse @ intercage.com

AdminHandle: IPADM190-ARIN
AdminName: IP Administration
AdminPhone: +1-925-550-3947
AdminEmail: ipadmin @ intercage.com

NOCHandle: NETWO670-ARIN
NOCName: Network Operations
NOCPhone: +1-925-550-3947
NOCEmail: noc @ intercage.com

TechHandle: INE4-ARIN
TechName: IP Network Engineering
TechPhone: +1-925-550-3947
TechEmail: ipeng @ intercage.com

# ARIN WHOIS database, last updated 2006-04-26 19:10
# Enter ? for additional hints on searching ARIN's WHOIS database.

and two new ones:

Code::

deny from 66.55.138.98 deny from 212.62.48.152

These guys had about 200 ips crawling me today:

66.90.110.192/22
66.90.95.0/24

It's a hosting company, but not a very reputable one from what I can glean.

jzky wrote:

Looks like Picsearch dont read robot.txt. Added a Disallow: /bot-trap/ in the beginning of january. and now this email arrived: Code:: A bad robot hit /bot-trap/ 2006-05-03 (Wed) 00:15:27 address is 68.178.174.79, agent is psbot/0.1 (+http://www.picsearch.com/bot.html)

Got a reply from picbot:

Code::

Our bot does read robots.txt and follow it; however the above request did not come from our IP address. Someone is impersonating psbot. Unfortunately this means we cannot help you directly as we have no control over what other people do (but we will persue the issue). Thank you for bringing this to our attention. -- psbot support

ws.arin.net/cgi-bin/wh...178.174.79 = godaddy

I'm getting hit from 80.229.157.13

Can I add this to the robots.txt anyway and make it deny access?

How else can I ban this IP from the entire site? \got to about 250 before it slowly started going down again? What do these 'bots' actually do (assuming this was one?)

maybe we could use a module like sentinel for DragonFly to stop these nasty little buggers

robots.txt only works if it's a good bot that obeys the rules and even with good bots, it doesn't kick in until the bot re-checks your robots file (they don't check every visit).

dragonflycms.org/Forum...17268.html

CVS has an in-built security system already, and can ban an IP address/range for IPv4 and IPv6 (based on MAC).

No need for sentinel.