Former Hostgator employee arrested, charged with rooting 2,700 servers
arstechnica.comThis all seemed like a pretty run of the mill story about an insider violating company trust, and then getting caught - until the final sentence: "Among other things, a desktop monitoring system that took screenshots of employee workstations in one-minute increments helped Hostgator officials quickly zero in on Gisse."
Not something I'd want on my personal system, but it's exactly the sort of thing that I think every NOC/Secure environment should have for post-mortem assessments.
>Not something I'd want on my personal system, but it's exactly the sort of thing that I think every NOC/Secure environment should have for post-mortem assessments.
Tools of that sort [1][2] are pretty standard in call center environments.
I'm not a fan.
First, the software tends to be incredibly expensive. Second, in my experience, it's primarily used by managers looking for reasons to bludgeon their $30K, entry-level call takers over trivial infractions.
A previous employeer used Spector 360[1] on the majority of workstations. It would monitor everything including taking a screengrab every 5 seconds that you could then watch later.
They'd sit down employees and playback fast-forwarded video showing how much time was wasted on Facebook, personal email, shopping, etc. It's horribly invasive but it meant everyone was too scared to use work computers for personal things.
My old job used spector. I'm pretty sure everyone knew it but people would still be on FB playing games when you walked past. I believe IT was the last to get it and for about a week after they installed it on our machines it didn't work because Microsoft Essentials disabled it as malware.
I don't think they ever looked at it unless they wanted to fire someone and didn't want to pay unemployment and needed proof that they weren't doing their job.
This is pretty much why they had this at HG
It would be fun to see the people's productivity on that system. And who of the tech people will try to break it as a sport.
I'm sure productivity skyrocketed /s
Given the way that hovers over employees, I think there's a fair case in there to unofficially rename the product 'Spectre'. -.-'
shudder
I don't think this is appropriate for general office use simply because you might leach private data from employees (bank accounts, retirement accounts, medical records, whatever), but in secure, restricted environments I think it's totally appropriate and probably needed.
There is an argument to made that you shouldn't be looking at your personal bank accounts or medical records on a work computer, particularly if you work at a company that cares so little about their employees that they'd implement a system like this.
It's not so much that the company cares so little about their employees, but that they have so many responsibilities for the data and systems they are entrusted with.
Seriously, though - what everyone I know does in this situation does is just bring their own laptop into work for personal stuff, and treats the work console for precisely that - work activity.
Is this common?
I have a few coworkers that bring in personal laptops and have them sitting right next to the work laptop. It absolutely blew my mind the first time I saw it. It would have been unheard of at my previous employers and probably gotten you called into an office.
It's fairly expensive (processing and storage), but it's well worth it for secure environments. I've worked for companies who have this set up on their Windows Server environment (since they were administered through the remote GUI) and SSH logging for the Unix/Linux servers when running as root.
Expensive? 1 image a minute for a 40 hour work week is 2,400 images or 120k a year. We're talking screenshots so they aren't large files (you don't need a high bit rate), probably in the range of 500KB which would mean a whole year is less than 60GB. 3TB hard drives cost ~$130 these days and have room to store 50 employee years of screenshots. If money is that tight you can always compress the images and get double or more for your cash.
As for processing, you need to find a new computer if you can notice a screenshot being taken.
"3TB hard drives cost ~$130 these days and have room to store … "
No trying to call out jinknee specifically, but I often see this argument:
"But storage only costs ~$50/TB!",
and I read:
"here's someone who's got no idea - who's never seen what 'enterprise' pays for proper secured/redundant/backed-up/auditable/managed storage."
Does anyone _really_ think sending the IT department junior round to BestBuy to grab ~$400 worth of external USB drives would then let you say "Right, that's then next 5 years worth of storage and archive of high-security and potentially-lawsuit-relevant employee data sorted!"?
If you're talking bare minimum, yeah. The software solution this company had wasn't quite that barebones. It was a robust suite, and actually did take some planning on the desktop and server side to set aside resources for it.
The software was part of a package that managed software installs as well.
I'll never work for any employer who does that, so I'd have trouble calling it a reasonable thing to do.
The areas where it would be important are on security/NOC systems, where there is almost unlimited power given the proper credentials.
The Runbooks that NOC teams have, quite often have them connecting to a lot of systems with greatly heightened privileges - It's not unusual for a NOC employee to have expansive sudo privileges on many of the unix hosts they manage. They are also often on privileged VLANs, with direct IP routing to a lot of hosts that normally wouldn't be reachable.
Most of our NOC guys have their own personal laptops, and they can hop onto the (unprivileged) wireless system and do their own thing when they aren't working an incident.
I'd have no problem having my screen captured once a minute when I was working in that type of environment.
Any powerful IT guy should be monitored and have his power checked. IT personnel have almost unmatched power in an organization to cause damage without detection.
Anything with lots of confidential information, or anything financial, and you are going to want to monitor all the people with access constantly. You may not want to snoop real-time, but you are going to want to be able to find and fix breaches after the fact, and do root-cause analysis.
It's not a matter of trust in the IT people, it's a matter of people go crazy sometimes, and people make bad hiring decisions sometimes.
What alternative do you think is reasonable to help track down problems like this after the fact?
Not nearly as exciting as the microphones they had installed all around the building to monitor discussion amongst teammates. :)
Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.
While this isn't guaranteed (all tools, including the compiler, may be patched), you can use checks and balances: verify /proc doesn't contain phantom processes, compile your own copy of ps, try more-obscure tools like top. If by "understand" you just mean "notice"... well, you don't, until one day you accidentally stumble across one of the above and start digging. (Maybe, for example, you install some kind of server monitoring tool, and when you log in to the web portal it provides you see a process that you find very suspicious; when you use ps, it doesn't show.) In my case, I've noticed this kind of thing twice: once, when the tool was binary pacthed to death (and just crashed), and once when the "patch" was "replace binary entirely", and the replacement was older and did not support a command line argument I knew that it should.
Nice. So both times that you noticed this, was it malice on someone's part due to which the tools were patched? I have just never heard or encountered such a situation and am frankly paranoid about something like this happening to one of the tools I use.
I've been pwned with 0-days in various email servers: sendmail over a decade ago, and exim4 more recently (still many years ago, though). The patched copy of ssh on one of my boxes was then distributing passwords to someone, and which then was used to gain access to another machine.
What I'm always paranoid about is that I work in a community of security researchers that sit on and occasionally drop 0-days: I have very little trust that much software is actually remotely "secure". Meanwhile, the only reason I had noticed those other attacks is just how sloppy they were... a more targeted-to-me run by a more careful attacker would have maybe never been noticed.
It has drastically changed the way I think about security, FWIW; as one example: I don't every store logs on a box being logged anymore. Instead, logs are immediately transported to another machine whose only purpose is to accept and store logs (and so is listening for incoming log packets, OpenSSH, and nothing else. The first thing anyone does is attempt to patch themselves out of logs (one attack I noticed because wtmp was mysteriously damaged).
(When I said "more obscure tools like top" I meant to say "pstree" but edited the statement one too many times before posting.)
Ideally, you should be running some kind of tripwire scan. If the hash for common utilities changes and you didn't update those binaries, then something bad has happened.
Hacked versions of common utilities is a common payload for rootkits.
If you lose superuser, you've almost certainly lost your kernel, at which point your only hope for Tripwire-type scans actually working is an attacker that doesn't know where to download a good rootkit from.
You should still run the scans, just be aware of the limitations.
That would be helpful for rootkits coming from outsiders, but would only serve to slow down (not stop) an insider. An insider knows what protections are in place (probably implemented them too) and can defeat the hash check if she knew how the hash was calculated, or can ship the binary alongside a regular update.
It's been some time now, but I've been there...
Once you've lost root to a sufficiently competent attacker you can't trust _anything_ on that box any more.
One thing that'll help against (some) script kiddies with rootkits is to have available statically linked copies of every tool you might want to use to see what's happening on your box - for a long while, every colo-ed box I managed had a cdrom drive with read-only versions of /bin, /sbin, and useful bits of /usr - all with the binaries statically linked. They can in handy a few times (mostly to confirm that "yep, we're screwed. Get this box off the network and powered down immediately and implement the bring-up-a-new-server-from-scratch plan right now".
At some stage though, you can't trust the kernel or the hardware - an attacker who's got into your booted kernel or your bios or your network card firmware, if they're good enough, they probably can't be detected by examining anything you could see logged into the box itself. The only way to identify that level of attack is by monitoring the traffic from the box from some trusted piece of network gear upstream of your rooted server (and against a sufficiently talented attacker, even identifying unexpected outbound traffic might be impossible. If your list of likely attackers includes three letter agencies or nation states, I hope your getting your secutiry advice from somethere other than HN comments…)
Read only media is good advice and low effort compared to most viable defense. Certainly valuable at times worth doing. Starting from the assumption you'll get beaten typically pays off over perfecting the hack-proof nirvana.
Nobody should lose any sleep about BIOS embedding and similar - that level of attacker and sponsor imply a level of threat that no typical organization has a chance against.
In my opinion, after years of pondering dozens of intrusions with many types of ways in and regular failure of all kinds of defenses I don't think there is much advice to give aside from the flaw is in your custom software, stupid.
I have become a really big advocate of CM and push button provisioning with identical replacement hosts that build from scratch and commonly get refreshed - relying on code and configurations managed centrally.
The best way to remove an attacker is a complete rebuild. If you're already using Chef etc why not just dump them proactively? Some roles don't lend themselves to this, but I assure you it is like a massive weight has been lifted.
Reminds me of Ken Thompson's Reflections on Trusting Trust: http://cm.bell-labs.com/who/ken/trust.html
> I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'.
So this happened to me once, but it wasn't ps or netstat but sshd that had been replaced. We only noticed it because the replacement sshd was bad and didn't set up the PATH correctly so "darcs push" stopped working (it was presumably set up for a different linux distro than we were running).
If it's a Redhat based server just run rpm -Va and that will verify the integrity of every rpm package on the system. If you're in a situation where you can't trust the current computer you can run it in a chroot from a known good copy.
He had root, so he could have instead installed a rootkit, which can hide the existence of processes from all of userland. In a graduate OS class I took, we had an assignment to do hide a process live on OpenSolaris 8 using the kernel debugger (kdb). I wrote some assembly and overwrote some bytes in the syscall functions for process listing. We were on developer builds so you could just use the function symbols by name in kdb. I forgot to cover /proc/ though.
Amateur hour, quite a few rootkits out there are almost completely undetectable...
Really? Can you name one?
Anything that lives outside of userspace.
If one has root and patience - flash BIOS and wait for a coldboot. One can even get an IP stack to pull down new firmware between boots. The user sees a normal post screen and your hypervisor sees normal hardware adapters.
See Jonathan Brossard's 'prior work' slide from his Defcon talk on his work [1] for more details on the state of X86 backdooring.
'Trusted computing' and all that.
[1] http://www.youtube.com/watch?v=yRxDvkKBMTc
https://media.defcon.org/dc-20/presentations/Brossard/DEFCON...
I know how they work, I'm asking GP to name one that he could have used. There isn't actually much "out there" that is undetectable. You'd likely have to write your own, which is highly non-trivial.
Obviously undetectable isn't true, but i'd guess that kbeast is likely to go undetected in all but the most prepared operations. Forensics shouldn't count since you already know you're fucked by then.
You get what you paid for.
> While his root access gave Gisse access to private data stored on a large number of customer websites, there's no evidence he used it, the Hostgator executive said.
I think the article is quick to jump to the conclusion that he was attempting to be malicious with his actions however this could be a case of Hanlon's razor.
His actions could easily be attributed to a less-than-aware sysadmin developing his own solution to get around often arduous security restrictions. Stupid, yes. Malicious, no.
%| You serious? A dude backdooring several system utilities, giving him access to 2700 customers' data and applications, and you're defending him with 'it's just all a mistake' nonsense?
He attempted to access the HostGator from outside computers: Hetzner Data Center in Nuremberg, Germany, and efnet.pe (Peru) are mentioned in the article. The Hetzner access was the day after he was dismissed.
Malicious, yes. Stupid, absolutely.
If he hasn't accessed any of these systems since he was terminated, he could state that it was for "emergency" access to remote systems upon other compromises. Since most of these systems are likely headless, then remote access is the only way to get in. A lot of remote exploits will nuke SSH, and other access tools, so having a "backdoor" is often a good idea.
That said, it's still likely that this guy is just a douche with a bad attitude, and deserves everything he has coming. Big difference between this, and "stealing" a bunch of reports that were government funded, and open to any and all users on the school network they were accessed from.
I don't think this would work. Installing your own solution on customer facing hardware? Might get you out of prosecution, but I think you'd have a nearly impossible time explaining that.
Did these 2700 servers play a role in any DDoS attacks as well?
It would be quite a lucrative stance for the employee to sell access to these servers to one or more groups who could potentially make more use of them.
2700 servers all on the same network makes for far less of a DDoS attack then 2700 similar servers on different networks - and it's far easier to detect and block too.
They would be more valuable for bitcoin mining most likely.
2700 servers probably wouldn't be worth much for that. It would be noticed fairly quickly, given that only CPU mining would be available, and how monitored servers usually are.
If you can hide the fact that you've rooted a box, you should be able to hide the fact that you're doing bitcoin mining. Worst case, run the mining in the kernel idle thread...
The apocalyptic heat and power use might be a giveaway.
Doubt it, based on the timeline. Doesn't sound like he accessed the systems at all beyond implementing the rootkit and/or patching things up.
This smells really funny. They could have buried this instead of going to court (with a 1+ year delay!) and committing PR seppuku by making this public and giving their clients a reason of distrust. Now, this is indeed the right thing to do, the guy shouldn't go unpunished and they should disclose their security breach, but if they are doing it for the "right" reasons, why is it 1 year later?
"Gisse didn't return a voicemail and e-mail seeking comment for this report. A Court docket shows he is scheduled to be arraigned next month and gives no indication he has entered a plea in the case. He's being held at the Harris County Jail on $20,000 bond, a spokeswoman at the district attorney's office said."
Wait, wait, just because the guy's in jail is no reason not to return voicemail and emails!
Would be way cooler if he rooted 2600 servers ;)
> "He did not access customer content," Pelanne told Ars. "We caught it well before he had any chance to do any of that."
> Given the rapid discovery, the malware was on Hostgator systems for less than a month.
Then yes, he did. If the malware was on there for more than a few days, I find it extremely unlikely that at least some data wasn't compromised.
Sounds like he knows Linux...
It's funny you say that because I remember their add. I made a jab at the company and - 2 rep.. If they knew the company they would have +2 rep me haha
I'm not surprised with that company actually, hostgator is a joke and the employees in Austin, Texas are too. I think the one thing that bothers me how did he get the SSH key? The fact that he had it tells me that someone higher up dropped the ball somewhere.