I have recently been approached following a successful defacement of a clients websites.  These sites were not ecommerce related, and they didn’t store any information regarding individuals, customers, staff, etc.  They were just examples of ‘normal’ business websites for trying to do business in the 21st Century.

Their websites had been targeted and defaced.  When we first arrived we were not sure how it had been compromised or how the compromise was effecting the site, we just knew it had been.  The important thing to bear in mind is that you can be bringing the site back online while at the same time keeping the defaced website for investigation.

We were amazed when we couldn’t find any specific details for how to deal with website defacements, so we had to come up with our own best practice.  We did found plenty about “deface website” however very little of it was useful for our requirements.

Step 1 – Take Website Offline

The very first thing we did was to take the websites offline.  The internet and your customers can cope quite easily with your website being unavailable for a few hours.  At worst it could be that your website may be distributing malware, participating in fraud attempts or hosting content that may be illegal.  At best the defacement will not be giving you a very good impression about your business.  Either way you don’t want to take the risk.

If you’re running an eCommerce site and your website is your main revenue stream this step may be more difficult, again check with your stakeholders and you may have to play it slightly differently.

We are also interested in preserving any data that may be on your site which could help in a success prosecution, so turn off the site.  This may seem to be a knee-jerk reaction (especially if you are considering doing a full investigation), but you now have to protect the company and other innocent third parties from additional risk of a defaced site.  With the right technical skills you can make defaced site still accessible from those that need access.

Make sure that you do not remove any of the content from the existing host. this includes making sure you retain a copy of all the files, the database and very importantly the access logs too.  If you can set your SQL database to ‘read only’ (we did this by giving the appropiate account only the ‘select’ permission).

At this stage if your attacker is monitoring the site then they’ll probably know that someone is looking at the problem.

Step 2 – Ascertain Resources

When we got to the customer site was to ask who the main players were in the company, and who we may need to ask help for externally. We did some stakeholder analysis and got the business development manager, marketing manager, IT manager and the MD.  From outside the company we got details for the web development company, the hosting company and the e-crime unit of the local police force.

From a technical perspective the likes of Google Webmaster Tools and Bing Webmaster Tools and any Google Analytics services can be very useful in finding out what happened or offering dates as to when they first saw a problem.  Find out who has access to these records and make sure you can have access too.  You will also need any logon details for the various services, including FTP, database logon, CMS details, hosting control panels, domain control panels or any other details of this nature.  If you can’t have the details you’ll need to know who does and liase with them accordingly.

At this stage you’ll probably want to check on if there are any backups of the site, if there are any internal procedures, change control or procedures relating to the site.  Ask the questions, and you may have to ask the companies discovered above for the details.  We also found that taking a screenshot of all the pages for our site in Googles cache gave us a quick way to check what had changed.

When we had our first sit down we started building up a fuller picture of the problem.  We found that the attackers had removed pages from the site using the CMS, had added additional pages (with blank content) and had removed the Analytics code about 5 days previous.  We also found a malware warning from Google from the day before.

We still did not have any information about how the site had been compromised, but we knew we had a historic backup complete with all the sites pages and we did have a timeframe in which this attack had been implemented.  We also had the full compromised site sat offline from the previous step.

Step 3 – Business Continuity

It may be a requirement that your business needs to have something online giving people phone numbers and addresses for your business.  Very often a page not loading will put people off your brand.  A simple HTML page with your logo, a brief “sorry we’re offline at the moment” message with your address and phone number could help reduce customer worries.

We put together a very small and basic HTML page and uploaded it to a completely different web host and made sure all the error pages were directed to this page with a temporary unavailable message.  It’s up to you if you want to give your visitors more information than just a generic ‘try back again later’ message, check with Marketing how they want to play it.

We then changed the DNS and made sure it pointed to our new host.  You may want to ask your Web Development team for this one, and make sure they know that getting it quickly is very important at this stage.  Keep it simple at this stage.  If at a later stage you need more detail add it on.

Step 4 – Obtain a Full Backup

In order to aid our investigation and to improve the time to get the site back up and running we made a list of things we had to have from the existing site:

• Full database backup
• Full directory structure complete with all the files
• Access logs from the server
• Apache or Nginx site configuration files

We thought it very important to make sure we had a full and complete copy of the entire database, since this was where the vulnerability appeared to be residing.  We used our MySQL admin control panel and created a dump of the entire database.  We logged on to the websites via secured FTP and downloaded all the files detailed above.

Just to maintain an accurate copy of all the evidence we then burnt two copies of the files above onto DVD.  One we put in the safe, the other we used for a source for the rest of the project.

Coming back to a point raised in Step 2 regarding checking the Google cache for pages.  We saved a screenshot of each of the pages in Googles cache.  We found this was necessary incase we didn’t have any other way of recovering the data.  We managed to get 80% of all the content of the site from Google (thank you Google).

Step 5 – Risk Analysis

We assumed that everything we copied from the website was a source of risk.  We assumed that the PHP files had been compromised, the JPG files were harbouring nasty images and the SQL data was full of evil problems.  We then worked on lowering each risk accordingly.

Luckily enough we had an original copy of the files and the SQL data from when the site was very first set up (about two years previous to the time we got there), so that gave us a starting point.  We also got confirmation that at no stage in those last couple of years should the PHP files have changed, all changes had been made on the web interface and should have been SQL changes only.

So we compared hashes of the original files with the files we downloaded.  All the PHP files were correct, and so too were the images.  We felt confident that these required no further investigation and were safe.  So we marked these as safe and carried on.

We didn’t see a great deal of point in doing the same hashing with the SQL database.  So instead we loaded it up onto am isolated MySQL environment and looked at the database manually.  It became apparent very quickly that content had been physically deleted out of the database.  At this point we made a note to check what actual security permissions the CMS used, and to make sure they were of the right level to give us the security we needed.

We also noticed that links had been added into the pages which were nothing to do with the client.  So again we made a note of these links to warrant further information.

At this end of this stage we could say that the files were safe, but the database wasn’t.

Step 6 – Bring it Back Online

Bringing a website back online is entirely dependant on how safe you as an Information Security Professional think it is to do.  You obviously have options for business continuity, if you are confident you can clean the data do you bring the site back online pretty much as is?  Do you revert to a last known good state if that state is acceptable to the client?  If you can’t trust any of the data do you start again from scratch?

Without meaning to state the obvious, the worst thing you can do is bring the site back online and then have to defaced again.  Have you discovered the source of the vulnerability and are you confident it can be protected or mitigated?

In our case as the previous step indicated we believed that the integrity of the files was intact, but the database wasn’t  We had a brief meeting with the various stakeholders and agreed that getting the site back online safely was key to a success project.

We ran a check on the database similar to this one to spot where we had problems:

select * from pages
where page_content like ‘%http%’
and page_content not like ‘%gypthecat.com%’

We then checked every section and made sure that it was right.  At this stage we did spot attempts to distribute what we imagined to be malware via URLs.  From this we outputted a manually sanitised table that we were happy with.

From our steps previously we had a suspicion how the site was attacked, but didn’t know for sure, so there was still a risk that we may have had a vulnerably site with what we were uploading.

So what we did to get our site back up and running was to set up an incredibly hardened web server, with the absolute bare minimum of permissions (including ownership of files and giving the SQL user only ‘select’ access to the data) to make sure we were confident the vulnerability wouldn’t take place again.  Yes we would loose functionality but it would keep the stakeholders happy and get the site back online.

We then went on Google Webmaster and asked Google to remove the warning from our website by doing a rescan.

Step 7 – Investigation, Mitigation and Protection

This is not the end of the project for the website defacement by any means.  We spoke to many knowledgeable people, raised many support requests, raised vulnerability reports and generally worked towards giving our clients a secure yet useable website. At the moment I can’t really post much more about what happened, but I may follow this post sometime soon with a description on some generic steps you can take to make your CMS more secure.

We worked with a number of vendors to figure out how the site was attacked.  I can’t give too much a way but it appeared to have been a zero day exploit penetrated via a third party CMW add-on, and then further exploited via weak passwords for a number of CMS users.  My client may have been targeted since they had high visitor numbers across multiple corporate sites, but that is just speculation.