Update July 11, 2008: At the end of this article is a real life issue one of our readers experienced. So if you think this is just a nice read, you will want to read his post to me and what happened to him. Take this to heart, because it is never a matter of if, but when, over and over again. Trojans and malware are as problematic as any medical condition we have in our society. No one is immune.
———– Original Article ———–
I am going through the tail end of a intensive two weeks on site and server security with content management systems. Around early January this year there was some kind of hacker blitz with alot of old trojans being wrapped in torrent downloads. Alot of worms as well. I never encountered so many from all directions before. Usually, I would see on trojan every three to six months on our system. Last month and this month I was seeing daily hacks and trojans being created in our systems after the firewall of course.
This is the latest method of hacker attacks. They come across as a Micro*soft file. They are initiated based upon an action in your browser usually. Therefore until a certain action occurs, they lay dormant and sublime. No anti-virus software will help you on this one, because they are not trojans or worms or viruses. But once that certain action kicks in and uses one of the morphed and usually overwritten Micro_soft file, the files calls up other files that were entered usually through your browser’s cache from giving permission to download or view some media file on a website.
Or they come in as a complete one step install of another software package after the hackers hack the innocent software package you were wanting, but not before they include a unique set of files that get dropped all over the place. Usually the files are kept in parts of the OS you do not have access to. You need to be outside the OS shell to see them.
Anyways, it took me 25 days to rebuild our network and systems from the ground up. There must have been at least 150 packages to install. It was very exhausting and wasteful. The only great event was building a stronger system including updating all software packages including routers. And with it we built a much stronger defense and disaster recovery plan.
For example we also suffered script injections. Now we don’t, but I guarantee we will in the future. The focus I can pass on to you, is always expect to be hacked every day. The strategy is to be able to recover in the quickest period of time. As I write this I have spent two days on secured connection protocol (SCP) and secured file transfer protocol (SFTP). It’s worth taking a review course in this area, as there is alot of different key generation programs out there. I can tell you I could have saved myself an entire day by talking to our upstream IT Team. I found out they had hard-coded the servers to accept only SCP connections and not SFTP connections. Nice!
For the past three days we have been uploading using SCP. You need generated key pairs for this, one uploaded to the server and one locally on each machine you use to file transfer on. What we have done is create a pristine and stringent with all functioning and tested applications. Man! - there were alot of software packages we use. Everything from video and graphic production to Internet marketing to system management. It took me two weeks alone just to reinstall everything with all the updates and upgrades. We are currently on day three of a compressed 23GB upload of a disaster recovery system. It is encrypted three different ways for the web. This is so if The Planet Servers get hacked, (like trying to hack Google or Microsoft - fat chance!) we are still protected.
One other thing, the SCP runs at 66% the rate of regular FTP. This is at 1,024 bit encryption. Banks use 128 bit encryption to give you an idea of how paranoid we are now. With hackers you can’t be paranoid enough. All our new sites this year all use 256 bit encryption. This upload is used only in the event of physical theft, fire, or acts of God. The turn around time for local systems to be up and running is now 8 hours. This is from a complete fire or theft of everything including the building, LOL!
Now all we can do is improve on this by keeping a external hard drive at our banks vault. This will only work when they are open of course. Otherwise we need a third physical location after hours. This is for acts of God though. This will keep us recovered in about two hours. If it is just hacker related, we can be up and running again in about 40 minutes from local backups not on the network. This brings in another tip. Keep a complete and fully functional backup off the grid.
So you want to definitely check with who ever manages your dedicated servers or hosting accounts and ask them first what they are using for secured FTP. This will save you hours of frustration trying dozens of configurations and finally figuring out nothing will work, because it was not meant to work in the first place. Now we are able to recover from any hacker attack locally and be able to reappear anywhere in the world within 40 minutes to 8 hours.
As far as server attacks go, we can recover within ten minutes. That’s because I have been around servers and smarter people for the past 13 years and have seen and learned all about these type of hacker attacks. You are best advised to have a outsourced or internal IT staff with a top notch security specialist. These are like specialized physicians who do nothing but read and go to security conferences and such. It’s a full-time career now. You need this if you are going to run a high profile website as we are doing. That’s 250K unique visitors or better a month.
————— What happens when you don’t make a backup? ————-
Nick’s Story
Hey Robert,
This is Nick, (from the forum where you are known as seoagent), who you sent some info about downloading a new virus program after I asked you for some help with getting rid of hijacking software.
I am now having to come to the local library just to make contact, I really need your help man, so if you could email me? I would really appreciate it. Please get back to me asap. And sorry i couldn’t get in contact any other way.
Now I have done exactly what you said and downloaded Iolo’s Software Mechanic Professional 8.0, i installed it and when i restarted my computer I no longer can get into windows. It says “the user profile service failed the login” and “the remote procedure call failed and did not execute”
This is really ң$$ӣ$ up man, what am i supposed to do now, i have no backup and no password backup.
I really need your help now man, and I really hope you can help me out. There must be some way to get back access to windows, or at least reinstall it and then get the stuff back, seeing as its on my hard drive.
————————-
Take this to heart. Here is my reply:
Nick: What a mess hey? Re: “the user profile service failed the login” and “the remote procedure call failed and did not execute.”
This is because you had a trojan which emulated itself as the driver file which calls up the remote procedure has been removed by System Mechanic. Your MS driver file was created in part of the shell before it starts ups. this is how trojans work as well as malware programs.
NOTE: It bites I know. I just got back to work after pulling some twenty hours. I went to bed at 5:30 a.m. and I am now working. I am also stepping out for a few hours. I thought I would let you know. So lets deal with this.
—————————————————————— INSTRUCTIONS ——————————————————————–
—————————————————————- END OF INSTRUCTIONS ————————————————————
That’s your solution. Like it or hate it, this is what you need to do for now and the future. You must plan to have your system hacked, wrecked, and burned in a fire. When you make a plan based upon that, you will survive to see the next day in spite of all things that go wrong. I can not stress this enough. This exercise was your wakeup call Nick, sorry about that. But neither Iolo nor myself can advise you on any systems that are already infected. But I will make note as others have: ALWAYS MAKE A BACKUP BEFORE DOING ANY INSTALL OF ANYTHING AT ANYTIME. This is the lesson today.
Make no mistake about it. Those people who do not invest in a disaster recovery system which is a combination of hardware and software outside of their current configuration are lambs to the slaughter for hackers. Now you know and are better prepared. If you read my blog, there is a whole piece on backups and recovery. And yes, we will be doing a webinar series on this. In comfort, thousands upon thousands of people’s systems are hacked and wrecked every single day, just like yours was. The real damage is how much data has been travelling out of your system and for how long and to whom? This is the real damage. At least your crash has stopped this. You are going to have to do your due diligence on creating a disaster recovery module. Or else you will be right back here again some time down the road.
Okay, you know what you need to do. The sooner you get started, the sooner you will be back. Finally, the reason the hacker wrote this program is obvious. Remove his file, you loose your entry to your system. You could try and envoke safe mode, but I doubt you will gain entry. Alright let me know when you are on the “other side.” If you can just repair the OS with the install disk, you will be fine. But you will also do the rest if you want to be prepared for the next hacker. And I warn you, there will ALWAYS be a next time. This is what disaster recovery is all about. Sorry to be the one to break it to you Nick, but if it makes you feel any better, my last build I did took two weeks and over 140 hours! So I know what the price is for hackers and their software. Be careful, bit torrent networks are loaded with malware and trojans everyday on every turn. Good luck Nick. I look forward to hearing form you once you get your system in order.
————————–
So as of July 11th, 2008; months after I originally wrote this piece, ther are many who fall prey to hackers. The lesson here as Nick learned, is to plan well ahead for all senarios. Including fire, theft, any acts of God, hackers, hard drive failures, ect. When you start to sit down and write how you are going to prevent disasters from occurring, you wil not get very far. The thinking you need is to accept these events will occur over and over again completely ourt of your control. Then start making the above arrangements to recover as fast as possible from each type of senario. You may want to get started right away, becasue the next batch of malware and trojans are awaiting you and are being written as we speak. They are always being written every day by the hundreds!
Our Latest Stories