Miscellaneous ⇒ User & Newbie Resources :: Archives ⇒ Incident Reporting - Outline for dealing with Security issue :: Archived ⇒ Community Forums ⇒ CPG Dragonfly™ CMS
Forum IndexUser & Newbie Resources

Archived ⇒ Incident Reporting - Outline for dealing with Security issue


Note: this is a copy of the announcement you can find in the security forum
Help! I've been hacked and I can't get back up!

Did that get your attention? This is typically what most of us see when discussing incidents these days, and unfortunately it's not very clear or helpful. I post this in order for you to understand what makes a good incident report and hope it also helps and assists you if something like this should happen to you down the road.


Incident Reporting for Dragonfly CMS




Introduction:

I want to state the steps and methods I'll use so I'll outline in this post what can be done on just about any web site, not necessarily just ones running Dragonfly CMS, but I will assume you are.

Since most of you will experience or discover your incidents by trying to login or load your web page, I will stick to this approach. I won't get you to download and install a bunch of software, you should be able to do everything suggested with the same tools you use now.

Now before you go and post that message here about your hack let's run through this text first. You want to remain calm and logical through the data collection and not start posting on the forums anyplace....yet.

Background:

Before anyone can help or offer suggestions you need to understand and provide a few bits of information first.


1. What type of server are you using?

This is important as not all servers or hosts are equal or similar. First of all is whether the platform is WINDOWS BASED or LINUX BASED or OTHER. The first two account for about 99% of the market so it's very likely one of these two. But tell us about it. Is it Windows 2000 sp1 or 2003 RC2? Is it RHEL or Debian? Is it a stable or unstable release? Is it running an entire web site, or an entire company's information. Is this your laptop your parents bought you?

Important to know if you own or rent the server directly, meaning do you have physical access to the machine. Is this a pre-built server that your hosting company provided or did you roll your own machine. Or, are you simply renting space on virtual-hosting systems? These are the ones that offer a control panel to admin your site, perhaps even have a reseller panel also. Virtual hosting is very, very common. If you have one of these you'll want to notify them very quickly, but lets still gather the data we need.

With virtual hosting your choices will be more clear, but not necessarily quick and easy. On dedicated systems or systems you run yourself will be more beneficial but you will have to get all that data and fixing yourself, at least I'd recommend it. If you have physical access to the system you can go image the system live, wipe it and refresh in minutes.


2. What software is the server running on your site?

This is very important so we can see what versions of web server PHP and MySQL you are using. There are numerous bugs in web servers, PHP, and mysql that can cause incidents to occur not just Dragonfly CMS. What modules, themes or extra's are installed? Any additional stuff running in your site, like blogging software or adsense-type additions? Did you just try to install some php-nuke module? These details will be important to know about trying to trace the vulnerability.

Is the server running SSH (which is a big exploit for many sites running it)? These are just a couple of examples of software and versions that can be important to keeping track of.

In addition keeping track of changes and upgrades and patches will be as important since if it could be determined that your provider just upgraded some software and then a hacker found it vulnerable that will be needed for investigation. This is very rare in the real world but my point is not to assume anything like this isn't important. It could be vital.


3. What is your provider using for a control panel to administer your site?

This is less critical, but again some panels have complete package solutions which may result in old problems being an issue.

CPanel is one of the more common ones you'll see around and is actively developed. There are many others, knowing existing problems with these packagaes could inform you if your vulnerable. The versions are found on the admin login for CPanel to your applicable site. These upgrades however are not free so some providers may avoid smaller 'patches' which could be critical to your problem, it's important to know where you are terms of these updates.

Is your provider using a custom panel? If so, you want to get details on when it was updated, if there are anywhere to get support and/or discuss problems with this software. Usually it's supported much like normal problems. But find out the details!


4. How often do you backup and inspect your site for unauthorized changes?

In order to determine something actually happened you need a means to detect this. You need to have periodic backups of your 'running' site to ensure you limit the time frame between changes. This is something most people don't even bother with, but I would suggest you inspect your code every 1000 page views (high traffic sites may do this daily or more frequently). However in my experience, almost no one does this.
You need to know that if something happened an hour ago, you can compare that to last months backup and see the differences. Did some bad code get onto your site? Did you change something that's now causing improper activity? Did someone 'really' hack your site, or did your last update let them? This is why a time reference can be vital and it needs to be as current to the event as possible to detect the occurrence to a reasonable time frame, otherwise you never know.

With constant backups you don't have a good point of reference. Remember when a crime is committed (lets say your car gets broken into and they steal your stereo) what will be most important question for the police is 'WHEN did this occur?' If you know you parked it at 10pm and you went out at 5 am and discovered it, you know it occurred between those times.

If you parked the car in long term storage (say for several months) and never checked it then you'd have no idea when it occurred exactly which would limit the chance of determining any further information, like who did it, why, etc.

So with backups you want to keep at least the last two or three, again just for comparison. If they are zipped you could probably archive a years worth without wasting a lot of space.

To recap; KNOW Your Systems Hardware/OS Details, KNOW what software runs on the web server, especially specific details about the versions, KNOW what your hosting provider uses for software, KNOW when your last backup was done.

So these 4 points are clearly fundamental to outlining any incident handling, and to assist in investigating further what may have happened and will assist in narrowing down the time frame when it occurred.

Be Prepared:

Above we discussed what we need to know or understand, now we want to assume the worst and protect ourselves (and our time and stress levels) by minimizing the impact that an attack on a web site can have. In the old days index.html defacements were all the rage.

Today however thanks to PHP and dynamic content this has changed the rules. Now Cross Site Scripting (XSS) sql injection and PHP poisoning are becoming a more common and the damage that can result is also more deadly than ever before. Theft of admin passwords or other sensitive data can result in you (or your company) facing legal issues as a consequence, so you need to be prepared.

1. Setup a web site backup policy and ENFORCE IT!

Critical. Without a backup policy you get stuck with rebuilding which can cause other problems that will further stress out the poor webmaster who just got hacked.

Make sure all the scripts you use on your site are kept offsite. Some people just keep the ZIP file. But what if you make changes to your scripts? Keep a working copy off the web site where no one can get at it.

2. Backup your database several times a day.

MySQL is so easy, and with Dragonfly CMS it's even easier so there is no excuse not to keep multiple backups of your database. Keep several copies. Here's my solution:

Create a directory for the current year. Inside create a directory for the current month.

Now create daily backups and name them 'backup-01' with the number representing the day.

If you do 3-50 backups a day, then add another number to the backup such as 'backup-01-33' to indicate that this is the 33rd backup of the day.

Some sites do hourly backups, some only do it once a month. You decide what volume works best for you.

Tank brought up a valid point about very large backups (~500MB+) can be heavy on the server. In these cases it would make more sense to mirror the database onto the same server or even perform incremental backups based on changes. You need to be the judge and 'be smart about it'.

2.1 Always work from originals, never backups.

It's good to never have to use your backups for any purpose other than recovery or as a comparison to what is currently there to note any changes since that backup. Some folks will copy their backups into their working directory on top of the originals or previous backups. THEN they work with them! If someone managed to alter one of your scripts and you copied back into your work you'll never be able to find this problem without a lot of work. Don't waste important time, ensure your originals are always your originals.


3. Backup your server logs.

This is a waste of space for most people, until they get hacked. Then they go to the current web logs and do a review. It's impossible to keep up to the second current logs, but regularly backing them up ensures you have limited the time from when damage could be detected.

But you need to be sure the log wasn't edited by the hacker? Did they delete the logs to prevent discovery? If so then you cannot trust what you find AFTER the fact, you need more evidence BEFORE and the only way to do this is to keep backups of the server logs. Much like you keep backups of Dragonfly CMS, logs are also very important.

Some providers will only allow you limited logs, so you may need to work with them to see all the access attempts directly to their server as opposed to just your site. If you run your own server you will need to familiarize yourself with the locations of all the logs and decide how to backup these files regularly. There really isn't anything called 'too frequently' unless thats all your site does.


4. Stay on top of patches and security notes that are released.

This is very important. With full-disclosure it's taking hours instead of days for vulnerabilities to hit the mainstream once a disclosure has taken place. I'm a fan of full-disclosure, but for many people it's where the problems come from. If you hear of something becoming a problem do something about it. Even if it turns out to be nothing, you were prepared and ready, not surprised. Sure you may waste some time, but after a while you will get more experience and gain insight into what you should spend time on and what not to.

There are several other steps one could take, and are a bit outside the scope of this text. If you keep good backups it's very likely if something does happen even if you cannot find anything to warrant an investigation, you'll be back up and running in no time without worrying about your data being affected or not. Remember if you don't practice fire-drills until there actually is a fire, you may not be around for a second try. Don't wait for your web site to get hacked before you practice your safety and security.

Getting hacked is a real potential even with a product as secure as Dragonfly CMS, and no one can prevent every possibility. Covering your back with backups, and numerous copies of the server logs can allow you to determine if and when someone did something and how it was accomplished.


It happened, I've been hacked/attacked! Help!!


1. First thing is.......stay calm.

Don't get irritable, upset, slamming your fists, sending scathing emails or making outrageous forum posts. This just gets you more upset since others will want to avoid you as opposed to stepping in to help.


2. Turn the site off.

In Dragonfly CMS this means putting the site BACK into MAINTENANCE MODE. If you forgot your site is installed in this mode be default. Go to Admin->System->Main Settings. Under the title are several links to the various components of this section. Click on 'Maintenance'. Then toggle the drop menu 'Active' to YES and then click the 'Save Changes' button below it.

This will allow you to access and use the site (admin) but for visitors and bots (or further hackers) will get this site is down. It also prevents valid users from doing things that may have to be removed or will be lost during the restoral process.

Yes, losing data or user settings is real, so be prepared to disclose to your users who are affected. I would not post a general message saying 'http://www.mysite.com was hacked yesterday' as you are now creating news or publicity for the hacker, and this is EXACTLY what they are looking for....attention. Do not give it to them. Instead either state that you were doing 'site upgrades/improvements' or send PM to the individuals directly affected. OR if you have a member forum only (not visitor viewable) then you could use this to announce it. As a last resort I would use the newsletter feature to send a mass mail to all your users. Keep in mind your hacker may be a valid user also...


3. Next, start making backups.

Yes you need to make a backup (or a snapshot or image) of the site. If you can I recommend you make an image of the site to disk or an image of the directory affected to disk. Best to use a large enough disk so you can keep in all intact rather than backing up to CDROM/DVD over multiple volumes. If it fits on a single disk then your much better off so usually a hard disk is preferred. For smaller sites a CDROM or USB drive may do the trick.


4. Make sure you get all the logs off the machine and back them up.

Not much to say but if you have a successful image that includes the logs your already done this step. Some sites may not keep the logs in the same tree or in the site itself so you may need to locate and back them up. If you own the box, then stick to making a complete hard disk image, perhaps in sections rather than all of them.


5. Repair the site.

It would be prudent to completely delete all the files, but you'll want to make sure it's a freshly created and empty directory.

Then you can restore your data from backup, however in most cases simply copying your files over from your private backup or working directory is the best and most secure way to ensure all your files are back to what you left them as.

If not you could audit the files and look for the ones that were modified or missing (or added) and then fix accordingly. For most people doing a bulk erase and bulk copy/replace will be easier and faster.


6. Review the logs.

I assume you know how to read an apache log or an IIS log so you will need to investigate these to find the potential point of entry. You may have to compare the results to previous logs so those backups will become quite handy.


7. Report the incident to your host/provider/law enforcement/other agency.

You should follow up after a certain period of time, but don't keep bothering them about it over and over. One or maybe two follow-ups will suffice for most incidents. If it involves substantial loss then this may not apply to you.

Eventually you will have determined as much as you could about what happened, where it happened and what occurred as a result. You should report it FIRST and FOREMOST to your host or provider and work with them to see if it can be confirmed. In some cases you may need to work with them in the above steps to either get the logs or to investigate further, especially in the case of a shared host compromise, which may affect tens or even hundreds of other web sites. In most cases you will simply get an email reply stating they will look into the matter. It's very impractical it will go any further than this, but do find out what the result of this report was.

In some cases it may warrant reporting the incident to the police or other law enforcement for investigation. This is rare, and in some cases a waste of time. Most law enforcement agencies (at least around here) are so overwhelmed with 'little stuff' that they cannot possibly do anything about it (why do you think spam is such a problem, they are just millions of little problems) so unless you have substantial losses as a result, I wouldn't consider this just to 'go after the bad guy'.

I would work as close as possible with your host to ensure that you put everything together that as been accumulated above so that your report gets handled as smoothly as possible. This will greatly improve the chance that something gets done with it. But having said this, don't count on it. In many cases your incident becomes another in a long list of incidents, and it may take hundreds or thousands of additional reports to develop into anything further. So once you've reported it forget about it and move on.

In some cases you may want (or desire) to seek revenge or retribution against the hacker or perceived hacker. This isn't wise. Not only do you then become a problem, you also can't be sure of whether your returning the favor against the hacker, or a innocent bystander, who may report you as a hacker. Then your case is invalid.

Conclusion

At this stage if you can determine that something in Dragonfly CMS has allowed something to happen that should not have, or your host/ISP feels something in Dragonfly CMS was at fault, then it's time to come here and provide some details.

In other words if you think you have discovered an exploit against Dragonfly CMS then a post here with the relevant details collected above (we don't need to know everything, just how it happened and a log entry to confirm) in the security forum will allow us to review the potential exploit and patch it if needed. If you feel this may contain sensitive information you don't want to post publically, then you may PM to myself or anyone on the developer team with your information.

We can then confirm or deny the incident as 'affecting' or 'non-affecting' the Dragonfly CMS distribution, and provide a patch if needed.

If you see something your not sure of, or have a question about then please use the security forum to ask your questions or check with others about it. I would ask that you don't post any topics however that indicate you (or someone else) have been 'ATTACKED' since this does not help, it only hinders any help for you or others that come here. Your also serving
notice for the potential hacker and providing ego boosts which is their payback.

I hope this serves as a template for others and I welcome any feedback, suggestions, complaints, etc. to me directly.


Edit Notes: Cleaned up formatting of article.
Edit Notes: Cleaned up formatting, outdated information, changed CPG-Nuke to Dragonfly though it still applies to CPG-Nuke, added a couple minor sections, fixed typo's not claiming to have found them all, in the article.

J.
j e r u v y a t y a h o o d o t c o m

Need help? Look here: www.dragonflycms.org/W...d=112.html
Need to chat? Look for me on irc.freenode.net

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Ubuntu7.10/Debian3.1 - 2.2.3/1.3.37 - 5.0.38/4.0.27 - 5.2.1/4.4.7 - CVS/9.1.2}

All times are UTC