Security Now - If a Website is Popular it can be hacked daily!


Recommended Posts

On the latest Security now, Leo was talking about how TWIT.tv was recently hacked. He was also saying that if a website is popular it can be hacked every day. He said some people are hired at companies whose sole purpose it is to just remove the daily website infections. I don't know how true it is, but if what he says is true, then this just makes it all that more important to keep your software up to date.

In the case of the TWIT it was hacked and a Java vulnerability was put on the site. Now if this happens as often as he says it does, then if you don't have updates, it's like playing Russian Roulette with links.

I included a the audio snip from the podcast and by snip I mean 15 mins long. But if you don't like to listen you can always read from the included show transcript and Vice versa.

Leo: There's the latest on SPDY. You know, we probably should mention, before we go any farther...

Steve: Oh, yes, I meant to, yes, yes, yes.

Leo: For the last 36 hours, should be gone by now, if you went to any site that had TWiT.tv in the domain name, you'd get this warning. This is the Chrome warning, but there's also similar warnings from Firefox, Safari. If you had antimalware software you might have gotten a similar warning from NOD32 and so forth.

Steve: Even a Google search turned up a warning link in your results.

Leo: Google uses a service, there are actually a number of services that track websites. I think Google does their own thing, but there are other bad software/malware databases. The warning, for those of you listening, says "Something's not right here. TWiT.tv contains malware. Your computer might catch a virus if you visit this site. Google has found malicious software may be installed on your computer if you proceed." You can proceed, or you can go back. They say, "We've already notified TWiT.tv we found malware on the site." No. The notification comes from about the eight millionth person who sees this and says - tweets me and emails me. And by the way, I'm glad you do, thank you.

And we worked on this quite a bit. This happens about once a month. And at first I was about to blame code on our site or a problem, something badly done on our site, or our site wasn't fully secured. And then I had a little - and I want to run this by you. I had a little come to Jesus meeting with Bear, Mike Taylor, who's one of the best sysadmins in the world, a part-timer for us, although his part-time job is often full-time, as it was yesterday. Bear and Chris Dieterle, who works with him, as well, immediately went in, they rooted out the malware, they modified the code that was the exploit. Often what happens is you have to figure out, well, where did the exploit happen, and get an updated module.

Steve: How'd it get in.

Leo: Yeah, how'd they get in. So there's two things you need to find out: how'd they get in, and what did they leave.

Steve: And what did it do.

Leo: Yeah, exactly. Now, in this case we were able to get rid of the malware very quickly. It takes a little longer for Google to clear this. They only update this warning every 24 hours. But there is a window of opportunity. If you fix it fast enough, you don't get in this. But because - what happened is we found it, we fixed it, and then inadvertently we reenabled it, the malware. And we thought we'd fixed it and there was a miscommunication, and somebody on my staff turned it back on. And so it ran all yesterday, Monday, what is it, this is Wednesday, so it would have run all Monday night.

Steve: And then got picked up.

Leo: And then Google put up a more serious warning that is harder to clear, takes 24 hours to clear. So that's why the warning persisted for so long. It is gone now. But the malware was removed very quickly, and the exploit I believe has been patched. So I asked Bear, I said, "Well, Bear, this is not acceptable. Who do I fire? Who do I blame for this?" He said, "Leo, this happens all the time." He said, "You and every other site is hacked constantly. What usually happens is the minute we get word, usually from somebody saying, hey, there's a malware alert on your site, we'll go in, we'll fix it, problem solved, move on. Because of the failure here, it lasted longer than it normally does, so you became aware of it."

He said, "But we're doing this at least monthly, and it's not just your site." And Bear works on a lot of very well-known sites, has friends that work on other very well-known sites. He said, "The bigger your site, the more often this happens. And anytime a site is updated in any way, it's likely that you're going to get another exploit." I said, "Well, what's the problem?" He said, "PHP is the problem." The nature of PHP, you remember in the good old days of CGI scripts you had a special directory that was specially permissioned that code could run out of, and nowhere else could it run. It can only run from the web server.

PHP is designed to run and be executed from any directory, any time. So if somebody can modify your file system, they can put a PHP script in an arbitrary folder and then point their browser to it and execute it. That script can then install malware. So that's very commonly how it gets exploited. And it's in the nature of PHP. And he said, "There's no one to fire here. You can blame us all for not getting this cleared sooner, but we do this all the time." I said, "What?" He said, "Every site is being hacked all the time. And there's not much you can do about it except be very proactive about scanning it on a regular basis." He said, "Most big sites have at least one, if not more, full-time employees whose only job is to look for file system modifications, be checking the log, constantly vigilant against this. It's the only way to prevent it." Now, you are in a special case, Steve, because you've written all the code on your site. I bet you it's CGI scripts. It's all...

Steve: No, actually, it's all assembly language, and it's pre- and post-server filters and back ends. So it's actually a single monolithic DLL so that I don't have the overhead of loading and unloading and stopping and starting CGI.

Leo: So that's even more secure. For somebody to be able to access your file system on your server, they'd have to find a hole in IIS, I would guess.

Steve: And there have been some. But I have something - I have my own web filter which is upstream of IIS, and I scrutinize everything coming in through that, also. So, yeah.

Leo: Right. You're much more safe because you're not using off-the-shelf software that people know. You wrote your own.

Steve: I was going to say, and I have a much less exciting site than you do.

Leo: Well, and that's what Bear said. Bear said the reason this is happening more is because you're more popular. He said - I want to name names. Well, I'll give you an example. A friend of his - I don't know if I should say this, anyway, a very, very, very, very well-known site - does this for them. And he said they're hacked daily. And there's not - and even though that this site has ample resources, let's put it this way, the resources of the entire federal government, they still get hacked all the time. It's kind of the nature of the beast.

I was ready - I said, "But no, wait a minute. You can secure a site, can't you?" He said, "No, all you can do is constantly monitor it." So, and people are saying, well, is it because you've taunted Anonymous? No, has nothing to do with it. It's not, was not probably targeted at our site. It was some scanner that's running all the time; right? And there's many of them. I'm sure there's thousands of scanners running on the Internet constantly that are looking for these kinds of exploits. So the minute a modification is made to your site that opens a hole, you're going to get exploited, period.

So in any event, it's probably not something that was targeted at us. It's just this is part of doing business on the Internet. A public server is always going to be attacked all the time. Our code is modified quite a bit, but it is in fact most of it commonly well-known code in PHP. And so it's a full-time job just to secure it. And in fact this happens all the time. Now, the good news is, well, first of all, I think it's great that these browser alerts are happening because it prevents people from going into sites that have been compromised. And all the browsers do that now. You should, if you see it, as much as you might want to watch TWiT, you should absolutely back out.

Steve: Oh, I'll bet you that not a single one of our Security Now! listeners is going to say, oh, yeah, I don't care that there's an infection there. I'm just going to go through anyway.

Leo: Well, a number, yeah, a number of our - I have heard from a number of people who did. The good news is the exploit was a very old Java exploit, and I'm sure that everybody who listens to any of our shows knows enough to keep their system up to date. So it's highly unlikely that anybody had any malware executed on their system. However, you should, as always, you should be proactive, scan it. And really, seriously, I understand. If you see this on any site, but I understand if you see it on our site, it's fine to email me, tweet me and let me know, and not go in. Please, back out. It will be cleared as soon as we've cleared the malware. It does take, unfortunately, if you don't do it right away, it takes longer to get that cleared, 24 hours to get that cleared. But it is cleared now. If you're still getting it, just restart your browser, you shouldn't see it anymore. And I apologize. But apparently there's no one to blame. It's just the way of the web. Does that sound right to you, Steve? Do you believe it's possible to fully secure a site?

Steve: Yes, I do. It's just math.

Leo: That's what I thought.

Steve: Yeah, it is possible. It's probably not convenient. If you restricted the execution rights of the directories for scripting, then you wouldn't be able to load things in default directories. I mean, you wouldn't be able to be nearly as casual as it's convenient to be.

Leo: Well, you secure your site, so we know it's possible.

Steve: Yeah. And I would argue it is...

Leo: Some of your security I would say is through obscurity because you're not using commonly well-known code.

Steve: Well, that's not obscure. It's just smart.

Leo: Well, but if your code were published on the 'Net, it'd be more likely to be a problem. But the way you've structured it, of course, makes it much less likely.

Steve: Well, but I mean, in defense of your approach, you would never have the site you have if it were up to me. So you would never be hacked, but you wouldn't have all the features you have.

Leo: Right, right.

Steve: And it would take about three lifetimes for me to implement everything you have in a way that would make me happy.

Leo: Well, we'd have to custom program everything, of course.

Steve: For example, you're pulling libraries, third-party libraries from their servers on the fly into the pages of the visitors, of the people who are visiting you. Not you alone, I mean, that's common practice now. I see this, and when I see it I just cringe because I see some site that is loading JavaScript on my pages from a URL of some other domain. And I think, well, okay, yeah, it's convenient for them. The theory is that, when that library is updated, they're automatically updated. The problem of course is, when that library is compromised, everybody using that is compromised. So, I mean, what's happened is we've gone for convenience over security. And this show is all about how those two, there's a constant tension between convenience and security. And the fact is, to do it right would end up meaning that it was never done at all. And that wouldn't work for you either.

Leo: So one of the questions I had is, well, why is it that we get this malware warning and nobody else does? And he says, well, there's a number of reasons. First of all, if you have a full-time person doing this, you can clear this before the malware alerts pop up. Second of all, some big, most big sites have an inside line to Google and an inside line to the malware sites, the stop adware sites, and can - see, Google does not, despite what it says there, notify us. The way we get notified is by the malware alert popping up, and then we act upon it. Most other sites will get an internal - this is what I'm told - an internal notification, hey, you've got a problem. They'll fix it. And that's why you don't see these.

The truth is, and we knew this about banking, banks are hacked all the time. It's bad business to talk about it. Sites are hacked all the time. It's bad business to talk about it. In fact, it's probably a mistake for me to talk about it because it just attracts attention and more hacks. The best thing to do, if you can, is act as if you're never hacked, you're a hundred percent secure, and just don't tell anybody. But that's not, as you know, how I operate.

Steve: Well, and it doesn't work in our model, where we have such vigilant listeners and viewers, and they're tweeting, hey, what's happened here?

Leo: Right. So one of the things that we are going to do, and I think we should have done this, but we couldn't afford to, is we are actually hiring - I'm hoping it's going to be Bear. We're going to extend him an offer. But we're hiring a full-time sysadmin to monitor this at all times. It's a fairly high cost. People like Bear are not cheap because they're really good. But it costs us money. We lost considerable audience yesterday. People didn't watch live, and we lost considerable ad revenue, as a result. So we can't afford to have these - not to mention it's embarrassing.

So there's the story. I do feel like - Bear is in the chatroom, if people have questions for him or want to get more clarification on this. But I do feel like this is something that's a little bit the dirty little secret of the Internet. And I was, when Bear told me this yesterday, I was flabbergasted. I thought that we were doing something wrong, that this was something we could fix. And he said, well, it's just the cost of doing business on the 'Net. Unless you're Steve Gibson.

Steve: Yeah, well, again, or unless you expend a phenomenal amount of resources in order to keep it from happening. And unfortunately - for example, PHP, it's very nice and very convenient and horribly insecure when it's not very carefully deployed and managed. So it's a powerful tool. But with that power comes responsibility. And, I mean, the whole model, I mean, we've talked about this. The idea, for example, that someone can send through a forum an SQL backend database command which will be executed by the server when it delivers that page, I mean, that's insane. It's insane that it was ever allowed to be done that way. Why was it? Because it was convenient for the people who were implementing it.

So there are major decisions which have been made which were absolutely wrong, by policy, not by mistake. And this was, for the longest time, this was my argument with Microsoft was they had insecure policies that were causing their problems, like having services that were enabled and running that no one needed. That was dumb. And the consequence, we had all these worms for some period of time. Finally they turned - they put a firewall in Windows and turned it on by default, and all of that problem just went away, bang, oh, it's a miracle. No, they finally fixed their policy.

So the problem is we're still in a position where convenience is trumping security. And the idea that your own database would execute commands that your website visitors gave it, that's just nuts. But the architecture enforces, I mean, it encourages that almost. And similarly, the idea that somebody could put a PHP script on your server, which your server would then execute, that's nuts. I mean, that's just crazy to accept executable commands from a passerby. But the fundamental architecture says, oh, look, here's PHP. I'm supposed to run that because I recognize the extension on that file. Just lunacy. But that's the way these sites are built now. And they're just not secure by design. They're secure by constant vigilance, which is exactly what Bear is talking about. And it's too bad that that's the state of the art, but that's where we are today.

Leo: And people want to blame Drupal or our web designers, and blame PHP. And certainly you already expressed the case. But it is just the way it is. I mean, it's not Drupal's fault. It's not PHP's fault particularly. I agree it was kind of a crazy way to do it. I long for the days of locked-down CGI script folders. But, oh, well. And we are making somewhat of a mistake by talking about it because it does attract attention. And that brings more attempts to hack you. And it's funny, Bear...

Steve: Well, but also, to the degree that we have other webmasters listening who are thinking, hmm, maybe I need to give some better thought to the security side of this...

Leo: My point exactly.

Steve: The problem, yeah, the problem is that people who are building these libraries, they're doing what they need to. The problem is the systems are just not secure by default.

Leo: Right. And Bear tells me that every time we do this, we talk about these kinds of things, that the server logs show a real spike in attacks. So there are absolutely people listening who see this as a challenge. And Bear just said in the chatroom, he says, "I cringe every time you do this." But he understands that that's what we do, and that's what we need to do. And that's, I think, one of the reasons Bear likes working for us. So we will, I'm sure, have a full-time security czar on the site as soon as we can do that. Moving on.

Link to comment
Share on other sites

ahh, yes...Leo Laporte and Steve Gibson....were top notch back in the day.....too bad they got too old. Oh, well, here you go Steve: Great job on SpinRite /thread. @Leo: Too damn bad you were a ********* and ***** and ************** to the ************** power.........and so forth........I'll pleasantly sport my tin foil hat, right about now.....meh.

Link to comment
Share on other sites

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.