In accordance with Finagle’s corollary to Murphy’s Law, the website broke the day our sysadmin went on a 2 week vacation on another continent. What’s most surprising about this is how little it surprised me. First, what happened:
Our library website is using a very old installation of the Joomla content management system (1.5.7 I believe.) Our implementation, for whatever reason, is insecure. I know very little about Joomla or server security, other than to nod sagely and say, “could be an SQL injection attack” (much in the same way dudes will surround an open car hood, though they know nothing about fixing cars, and say, “it’s probably the transmission.”)
So last Friday, sometime around 11am, our website stopped being an actual website, and started being just a page that displayed the site title. Not so useful for users, I’d imagine. My first instinct is to look at the main index.php page to see if it’s been replaced with a different one. I’ve dealt with this hack in the past, just the result of some asshole saying “LOOK WHAT I CAN DO!” You just delete the new index file they put in, and put yours back in.
When I checked our index file, it was present, not renamed, and all the content was accounted for. At the end of the file there was a php command that was trying to redirect the site to some .biz.tr website, so I took out that code and figured the problem was fixed. Nope, site was still b0rked. I went into all the sub-folders’ index files, and found some malicious code in them too, so I decided to just replace all of them with clean backup versions. Still. B0rked. On to the config file. Everything looks fine there, but I replace it with a back up version anyway.
Also it was about this time I sent out an email to the staff that basically said YES I KNOW THE SITE IS DOWN YOU CAN ALL STOP CALLING ME ABOUT IT.
At this point I’m stumped, so I call the head of Media Services, who maintains the servers. He goes in to check which files were accessed at 11am that day. None. Uh, ok. He has me go into the database, to see if the content looks ok, and it does. It occurs to me that I’m able to get into the site from the admin panel, which is a subfolder in the site root, so it’s not that the whole site directory is corrupt. Subpages of the actual site, however, are not loading.
We finally realize that this is not going to be an easy fix, so I put up a temporary webpage linking to common services, most of which are on different servers, so they’re fine (catalog, database list, LibGuides, and Google forms.)
The head of Media Services then spent his weekend picking through all the myriad of folders on the server to find workable backups of pretty much all the pieces of the site (which, in a content management system, are many.) He then pieced the site back together, file by file. I honestly don’t know the details of how he made this happen, because whenever I asked him about it, he sounded like he was going to cry or murder a baby panda, so I’m just gonna let that go. He obviously has some sort of PTSD, and I don’t want to poke the painful memories of “the incident.” He did mention something about finding out that the site was actually hacked in June, and was only taken down just now by a remotely-issued command that activated the previously-inserted code. Insidious bastards.
I did a Google search for the spam url I found in the main index page, and it’s been injected into tons of insecure Joomla installs. I only mention this because people keep asking what kind of douchebag hacker makes it his life work to take down crappy college library websites. It was just a bot that looked for vulnerable targets. Nothing personal, my friends.
So say your site gets hacked, and you try fixing the index and config files, as I mentioned in the last post. And you try checking the server logs to see what files were messed with so you can replace them with backups. And you turn on error-reporting in your CMS to try to see what’s going wrong. And you Google some of the malicious code you found in your files. And say none of these fixes work, or yield any useful information. What now?
Well next you’ll want to search for common hacks to your specific CMS and version, to see if anyone can walk you through fixing them. Here’s a pro-tip though: in the end, most fixes will just tell you to install a fresh version of the software, and if you’re in my situation that’s not an option, so learn from our fail. Set up your website in such a way that disaster recovery is a relatively easy job, or at LEAST a viable option.
I will admit that much of this advice is based on my experience with Joomla and WordPress. I have much less experience with Drupal, so some of it may not apply there. If you’re a Drupal person, and have advice for keeping your site safe from hackers, please post it in the comments!
- Keep your site software up to date.
Also your plug-ins. Also your themes. Because chances are, if they’re all from reputable sources, the developers will be addressing vulnerabilities as they pop up. The world of hacking is a shifting landscape, and what’s secure today is not necessarily secure tomorrow.
- Keep your customization modular.
In WordPress, this means using a child theme, rather than making changes to the main theme. When you update a theme, it will override any changes you made to those files. Now you’re in a situation where you have stop updating your theme, and are thus breaking RULE NUMBER ONE. You will regret this.
- Keep your site root clean.
Actually, not just the site root, but all its sub-directories. Part of the problem with our site is that the root folder is cluttered up with custom includes, images, project folders, etc. If you’re not the one who put them there (as in my case, where I’m taking over a site from someone who is no longer here) it’s hard to know what folders are part of the CMS’s software, and which ones are not. In general, if you re-install the software, it should just ignore these unrelated files and folders, but if the software contains new files and folders that have the same name as yours, you can accidentally overwrite your files. I’d say either place these files one level up, OR, if you want them to have the site root’s url, create onefolder in the site root, and put all of it in there. Clearly mark that that folder is NOT part of the CMS’s file structure.
Srsly. Updating or re-installing your CMS may not be a difficult process, but YOU may not be around when it needs to be done. YOU may be on another continent, or at another job. YOU may have gotten hit on the head or killed those particular brain cells with alcohol. There are so many pieces to a CMS (plugins, templates, images, forms, database(s), etc,) it’s supremely helpful to know which of these need to be backed up in like six places before you re-install, so you don’t lose the hours and hours of work you put into customizing them. Which leads to…
- Keep backups of important files and folders.
Yes, I know you’re backing up your entire site on a regular basis, because to not do so would be INSANE, but even so, keep an extra copy of important stuff, JUST. IN. CASE. I have a folder on my desktop with my config file, my entire child theme folder, and my custom plugin folders. WordPress is smart, and names the blank config file something else, so when you update, that default config file doesn’t overwrite yours, but still. (Remember to update these backups every time you make a change. I got in that habit anyway, because I keep an entirely local copy of the site to make changes to before making them live, so it’s kind of a reflex at this point that when I make a change in one place, I update those files everywhere else.)
- Minimize the use of 3rd party modules or vulnerable code.
Wherever possible on our WordPress site, I used CSS/jQuery to create my own custom features (like our tabbed search box) rather than install another plugins. Plugins can increase the vulnerability of your site, so use them with caution (and, not to drill it into your head or anything, but keep them updated!) We’ve also made the switch to Google forms for all our forms, so we have the benefit of their security features (and so the forms are connected to off-site databases, rather than databases on our servers.)
- Create a simple html backup site ready to go at all times.
Honestly, I never even thought of this until the head of Media Services suggested it. Libraries subscribe to many services that are hosted off-site (such as the catalog, research guides, databases, resource managers, and discovery services.) These services are the core of our business, and are still available even when your site is down. Create a simple site that links to whatever services and resources are still available, as well as basic information like hours and contact info. I just downloaded a free CSS template and created a quick and dirty 2 page website that can be put up during downtime (unless the entire server is down. Then I guess you have to put them elsewhere and do a redirect? ACK! SERVER STUFF FRIGHTENS AND CONFUSES ME.)
So ok, there you have it. I am by no means an expert on the topic of hacking, or disaster recovery, or even web development for that matter. This is just an attempt to learn from my own experiences, and to put what I learned out there, just in case it can help someone else in a similar situation. If anyone else has some advice on these matters, please share in the comments!