High Server Load

[MajorHeadache]

I know I've been here before, but I'm hoping to get a thorough and detailed analysis, hopefully leading to a permanent solution. This is also a different server than I had in the past, better CPU etc. I'll try to make this as detailed and thorough as possible, but if I can provide any more info, let me know.

Here is the situation: Most of the time, the server load (viewing top) is reasonable -- somewhere between 0.8 and 2.5. Sites are responsive, things are good. But periodically, at random times, the load will shoot up to 6, 10, 20, I even saw 74 once. Tonight it has been spiking above 5 quite a lot. Usually I will restart Apache from WHM and it will calm down. More often than not, the wait-states will be very high, near 100%, and this is when the server is basically unresponsive. Tonight I simply put my big DF site into maintenance mode and the load dropped below 0.5 within a minute or so. Ten minutes later, I turned it back on, and quite soon the load was above 3.

The site: My site is www.mlparena.com, a fairly popular site dedicated to My Little Ponies. I administer it for my wife. You wouldn't think it would be too popular, but it seems to be. I'm running DF 9.0.6.1. I plan to upgrade, but unless someone says it's much faster, I'm not exactly keen to open that can of worms, especially considering the amount of tweaks, additional modules, etc that it has.

The extras: I have integrated Menalto Gallery 2.02, via the dragonflied module. The database for this is about 66MB at this point. Sometimes I turn off this module but I don't notice a significant reduction in the load. I would like to run IPTracker, but haven't had it on in a year for fear it was increasing the MySQL load. I also have the following modules up and running:
dfmaps
daily babe
flash games
forums centre block
most posts
most online users individual
reviews
shoutblock &
treasury module

Another issue that concerns me is the size of my database. I've never culled old posts and it goes back to 2003. The main db for the site is 910MB. In the past when I've tried doing maintenance, it never ever completes, just quits after awhile with the old posts still there. Maybe this isn't a big deal, maybe it's huge, I have no idea.

We also have a lot of forums, and a lot of ranks, including many unique ranks for individuals. I've often wondered if doing so added to the load in some way.

The server. I'm using a Planet single Xeon 2GB ram server.
1 Intel \ 2.0 GHz 1333FSB - Woodcrest \ Xeon 5130 (Dual Core)
2 Generic \ 1024 MB \ DDR2 667 FB DIMM
1 Dell \ 9G Drive Controller - SAS/SATA \ SAS 5/i
1 Western Digital \ 250GB:SATA2:7200RPM \ WD2500JS

WHM, cpanel, redhat, the rest of my server specs are in my sig.

I had configserver go over it right after I got it. They did their cpanel special. I also asked them to tune my.cnf. This is what that looks like today:

Code::

[client] port socket [mysqld] port socket skip-locking key_buffer = 256M max_allowed_packet = 1M table_cache = 256 sort_buffer_size = 1M read_buffer_size = 1M read_rnd_buffer_size = 4M myisam_sort_buffer_size = 64M thread_cache_size = 8 query_cache_size= 16M thread_concurrency = 2 max_connections = 500 skip-innodb log-slow-queries = /var/log/mysql-slow.log <- I just looked, this log file is now 65MB! long_query_time = 5 server-id = 1 [mysqldump] quick max_allowed_packet = 16M [mysql] no-auto-rehash [isamchk] key_buffer = 128M sort_buffer_size = 128M read_buffer = 2M write_buffer = 2M [myisamchk] key_buffer = 128M sort_buffer_size = 128M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] interactive-timeout

Backups are turned off (unfortunately!) Stats are off on most sites, including this one.

My current theory is that maybe I am getting slammed by bots. This would explain the random timing. Either that or something is going on with the server that I am not aware of.

Here is my robots.txt file:
User-agent: *
Disallow: notmyrealadmin.php
Disallow: error.php
Disallow: /admin/
Disallow: /blocks/
Disallow: /cache/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /themes/
User-agent: Baidu
Disallow: /
User-agent: *alexa*
Disallow: /
User-agent: Googlebot-Image
Disallow: /

I am including some screenshots of top. These were all taken tonight. I don't claim to understand them much.

www.mlparena.com/top2.gif
www.mlparena.com/top3.gif
www.mlparena.com/top4.gif
www.mlparena.com/top5.gif
www.mlparena.com/top6.gif
www.mlparena.com/top7.gif
www.mlparena.com/top8.gif
www.mlparena.com/top9.gif
www.mlparena.com/top10.gif

top9 is the nice status after I turned off the pony site for 10 minutes, then turned it back on. top10 was an hour later.

I also get an email if the server load is high for 5 minutes or more. Here's one from tonight. Since it's longer than what I see in top, it shows more processes.

www.mlparena.com/highload.txt

it's been particularly bad tonight.

I am pretty desperate to get this sorted out. Besides this site, I have a few paying clients who don't appreciate the server being unreachable ever, much less this often.

I am also hoping I can find the solution here because everyone else simply says DF is to blame and I should look for a different script. If I should pay for a particular service, I will.

I don't know if there is going to be a magic bullet, or a series of minor things that add up, but I'm sure hoping this can be fixed

[DJ Maze]

You ever investigated the netstats?

Keep in mind that one single PC can't have hundreds of simulatinous connection. It's better to have multiple cheaper slower machines to handle many connections.
Think about traffic and cars and jams

Dragonfly 9.1 has a security system with anti-flooding, since that is active we have had no real high peaks anymore.
So it seems many obscure visitors eat servers.

[Phoenix]

You should also add a crawl-delay (5) to robots.txt - Yahoo, and others that obey crawl-delay, can destroy you without it. Just blocking them in htaccess can do you more harm than good.

However, there are numerous wannabes and spammers that are blocked, and others that you can block in the security system (9.1.1) that DJ refers to.

Google ignores it , but they do state you can contact them for a reduced crawl rate. Beware of this, as their idea of reduced rate can be "switched off", and you end up with no indexing. Like MS, Google does its own thing, regardless of what users want - come to think of it, Google takes the lead these days - MS has improved.

[MajorHeadache]

DJ Maze wrote:

You ever investigated the netstats?

Do you mean like web stats? Anything I should be looking for?

DJ Maze wrote:

Dragonfly 9.1 has a security system with anti-flooding, since that is active we have had no real high peaks anymore. So it seems many obscure visitors eat servers.

Is 9.1 compatible with the modules I have installed?

[DJ Maze]

Go into SSH and execute: netstat -n

As for the add-ons: i never can say

[MajorHeadache]

this is what netstat reported. I wouldn't know if it was good bad or indifferent.

www.mlparena.com/netstat1.txt <-load around 0.3

www.mlparena.com/netstat2.txt <-load around 3.3

www.mlparena.com/netstat3.txt <-load around 8

I have noticed that I get a number of emails from the server saying that this or that IP was blocked temporarily for too many connections. Usually there are around 320 connections on one IP. The few IPs have checked have been on my DF site.

I always attributed this to the server being slow, so maybe the user trying to access the site repeatedly. But maybe something else is happening?

Is it normal to have so many sleeping processes? Seems like 200 to 250 is the norm for me.

[NanoCaiordo]

if you even get an email then you have Dos_Deflate installed and your server is under a D-Dos attack.

[MajorHeadache]

NanoCaiordo wrote:

if you even get an email then you have Dos_Deflate installed and your server is under a D-Dos attack.

I don't think I have that installed. I've looked at the list of things configserver did, and I didn't see anything like that. The only thing I saw regarding DOS was this line:

'Enable "Attempt to prevent pop3 connection floods" - this stops people trying to attack your server by DOSing the POP3 service'

Here is a typical email I get:

www.mlparena.com/toomany.txt

Sometimes the first one to maybe 20 lines will say SYN_RECV instead of TIME_WAIT.

I may get 5 to 10 of these a day.

And just for reference, here's another netstat output from this morning, server load was low:

www.mlparena.com/netstat4.txt

Can anyone looking at these tell me whether they look normal or not? Does anything stand out as noteworthy? Is there a way to run netstat that determines which of the connections belong to which site?

[Marcus81]

Do you have only this site on server?

I am having the same problem.. But on my server is 200 web sites more. But my is the biggest.
So I am looking for the reaseon of over load.

Do you try to update DF to 9.1.1 ?

Maybe the MySQL 4.1.21 and PHP 4.4.5 did not work corectly with DF in this case? Because I have the same version of mysql and php on my server.

[Marcus81]

Also,
I have notice that TIME+ in top going up to 300:0:0 for mysql process..

And have noticed that free RAM memory is about 16 MB when it comes to overloading.

[MajorHeadache]

Marcus, I would love to continue this topic but it will have to wait a week as I am just heading out of town for a family vacation. Hope to continue upon my return.

[hqarrse]

I can't comment on the possible DOS but may be able to help with the DB config.

I've been through this a few times now as we've grown, and this is one of the best tools I've found for tackling database performance:

www.webyog.com/en/downloads.php

This will monitor mysql for short while then give you suggestions for improving performance.

As an example on my server it suggested the surprising but successful recommendation to turn off query caching - something I had previously been increasing to take advantage of more memory. In our case the writes to the DB were frequent enough that caching was actually reducing performance.

Another very helpful figure that monyog produces would have saved me a lot of pain in the past is 'worst case memory usage'. If that is more than the RAM you have (less some for Apache - approx 10kB per thread) then it would explain your spikes problem for instance. I had this as I was allowing too many connections, the server started using swap space, it therefore responded more slowly, more connections queued and so on until it died.

That said you don't need a tool to see that if all your DB connections are used then you will use way more RAM than you actually have - your key cache (a global mysql cache) plus 500 * the dedicated per thread buffers gives well over 4GB, just for MySQL.

I think it is very likely that you can stop your spikes just by reducing the number of MySQL connections you allow. The go from there with monyog and see how you get on.

[hqarrse]

One more: I can't see eaccelerator on the spec list. Installing this gave us a huge performance increase on our apache server. Server loads of 3 - 4 dropped to 1.5 straight after install.

Of course I know that none of this cures a possible DOS problem!

[DJ Maze]

DOS needs a hardware firewall in front of the server

[MajorHeadache]

I'm back from my vacation (when you take your family, it's hardly a vacation!) and ready to dive in. Unfortunately, I feel I have more questions than answers when it comes to this subject. I'm just a little glad to know I'm not the only one.

Like marcus81, I also have a number of sites on my server, but more like 100, not 200. And also like him, my own site, a cpgnuke site, is by far the heaviest hitter. I am anxious to upgrade to 9.1 but haven't had the energy or time to undertake it.

I liked what hqarrse had to say. I'd like to try MONyog, and I probably have to reduce my number of connections as well. I htink mine is set to 800! What happens after the max number is reached, from the users persepctive? I installed MONyog but so far can't get it to connect to mysql. But I did get it to connect to SSH, whatever thats worth. If you have any suggestions I sure would appreciate any help on that.

This also brings up another issue. I have a security package installed by config server. One thing it does is block users temporarily if they have too many connections. This happens maybe once a day and happened just now. I checked their IP and sure enough, they are a member of my cphnuke site. It claimed they had 340 connections open. Basically the email looked like this:

Connections:
tcp 0 0 74.52.167.82:80 66.82.9.xx:7368 TIME_WAIT
tcp 0 0 74.52.167.82:80 66.82.9.xx:6857 TIME_WAIT
tcp 0 0 74.52.167.82:80 66.82.9.xx:6858 TIME_WAIT
tcp 0 0 74.52.167.82:80 66.82.9.xx:7627 TIME_WAIT
with 336 more lines below that.

So I checked with IP Tracker on the site to see where they'd been. They had only surfed to 17 pages in the course of 11 minutes. What's up with that?