Need help finding suspected spamming botnet

[Bryan R.]

I have a new client who contacted me because their previous IT guy couldn't get them up and running following their office move. I got there and got them up no problem.

 

I noticed their Anti Virus had expired since 2011. Not even the next day, they get hit with ransomware and had no running backups for a year.

 

After dealing with that and getting them back up and running again, they are now being blacklisted by 7 different big providers. I've scanned all their PCs and server and found nothing with Malwarebytes and Trend Micro. I've tried delisting but they are detecting spam currently coming from the clients IP and won't delist.

 

I've enabled the firewall in the AV client and denied access to outbound 25 on everything but the server and confirmed PCs cannot telnet over port 25 anymore.

 

I'm going there today to reimage pretty much all of their PCs. I just really hope their server isn't somehow infected.

 

To make matters worse, I should mention they had a user's account with no password and full domain admin rights... So that is why I'm concerned about the server (SBS 2008).

[+BudMan MVC]

Blacklisted why?  What is your public IP, do you have a PTR setup on the IP?  Is it listed in a dynamic range.  Most major players will not accept mail from rfc ignorant setups..

 

PM your domain your sending email from.. I will check it out, I assume your only sending email from the same IP your getting email too?

[Bryan R.]

6 minutes ago, BudMan said:

Blacklisted why?  What is your public IP, do you have a PTR setup on the IP?  Is it listed in a dynamic range.  Most major players will not accept mail from rfc ignorant setups..

 

PM your domain your sending email from.. I will check it out, I assume your only sending email from the same IP your getting email too?

I've PM'd you the public IP. PTR is setup. SMTP diagnostic on MXToolbox checks out.

 

When I check the IP against blacklists, some are specifically saying they are detecting botnet-type spamming coming from our IP. This is from CBL:

Quote

IP Address _______ is listed in the CBL. It shows signs of being infected with a spam sending trojan, malicious link or some other form of botnet.

It was last detected at 2016-07-21 15:00 GMT (+/- 30 minutes), approximately 30 minutes ago.

This IP address was found to be emitting very large amounts of spam.

 

[sc302 Veteran]

Best way is to put a sniffer between your main switch and router. Disable wireless on router and put an so behind your main switch so that everything goes to main switch before it hits router.  Then when the computer decides to communicate to spam honeypot you can capture it and then kill it. 

[Bryan R.]

That's a good suggestion. They only have a Linksys router so I was worried about not having the tools to find the culprit.

 

I'm assuming Wireshark on a laptop would suffice? I have a linux laptop with Wireshark already.. But I'm not familiar with how to set it up to to find this traffic or how this traffic is going to look.

 

I've already blocked outbound 25 connection from PCs so it must be connecting over something else?

[+BudMan MVC]

Yeah I sent you the info I found.  The domain looks good as far as your PTR and forward, and your not an open relay.  But yeah show you on 5 different lists, sent you some details in the PM.

 

What sort of switch do you have, do you have a smart switch that can do port spanning/mirroring?  So you can see all traffic going to the internet - if so plug in your latop into this port and set wireshark to grab traffic on port 25.  Can show you how to set that up for sure.  But really need a smart switch or worse case an old school hub..  So you can grab all traffic going outbound.  How does a business do actual business with a linksys router??

 

I would look to putting something in better - if you have any spare hardware like old pc laying around, pfsense be rocking you out, then you could just block outbound 25 there for anything other than your server, etc.

 

Is there any sort of budget to help uplift your network into this century and more in line with what a business should have?

 

 

[Bryan R.]

35 minutes ago, BudMan said:

Yeah I sent you the info I found.  The domain looks good as far as your PTR and forward, and your not an open relay.  But yeah show you on 5 different lists, sent you some details in the PM.

 

What sort of switch do you have, do you have a smart switch that can do port spanning/mirroring?  So you can see all traffic going to the internet - if so plug in your latop into this port and set wireshark to grab traffic on port 25.  Can show you how to set that up for sure.  But really need a smart switch or worse case an old school hub..  So you can grab all traffic going outbound.  How does a business do actual business with a linksys router??

 

I would look to putting something in better - if you have any spare hardware like old pc laying around, pfsense be rocking you out, then you could just block outbound 25 there for anything other than your server, etc.

 

Is there any sort of budget to help uplift your network into this century and more in line with what a business should have?

I found that their switch is a Netgear Smart Switch which does have port mirroring. I'm going to turn that on with my laptop plugged in and try messing with Wireshark.

[+John Teacake MVC]

What edge firewall do you have? ...

[+BudMan MVC]

So you would want to mirror the port that goes to you the internet, ie the port connected to the router.  If its just some linksys - are any other things connected to any of its other lan ports?.  This should capture all traffic in and out of your network.  Its prob going to be a lot of data so you would want to filter on only tcp port 25 traffic.

[Bryan R.]

I left my laptop running Wireshark while I was on site today so I have a log of what should be everything over an hour or two.

However, I seem to have found a utility that has been blocking these bad connections from the server since this morning. I've confirmed this with CBL stating the following:

Quote

IP Address ______ is listed in the CBL. It shows signs of being infected with a spam sending trojan, malicious link or some other form of botnet.

It was last detected at 2016-07-21 18:00 GMT (+/- 30 minutes), approximately 6 hours ago.

This IP address was found to be emitting very large amounts of spam.

Which is progress... With any luck they will get delisted in 24 hours. But that is only a band aid... I believe the server may be infected. This is a screenshot of the utility blocking these bad connections:

 

So I tried using netstat to pin a process to these weird outbound port connections but nothing comes up. These America Online entries seem to happen at intervals.

 

[chrisj1968]

8 hours ago, BudMan said:

Blacklisted why?  What is your public IP, do you have a PTR setup on the IP?  Is it listed in a dynamic range.  Most major players will not accept mail from rfc ignorant setups..

 

PM your domain your sending email from.. I will check it out, I assume your only sending email from the same IP your getting email too?

I'm thinking before the move, the server had the malware/ransomware on it and just pulled a Godzilla on the entire network. From the OP's verbiage, the problems started after he got everything setup. I'd interrogate the  heck out of the server and clean it up. Coming from a topology angle here. the server is the head of the beast

[+BudMan MVC]

how exactly are you using that software.. That blocks inbound connections to your computer..

 

64.12.91.195 isn't on any blacklist I show..  Nor is that next 152 one.. But yeah that is an AOL mx server IPs.

 

So your sending them mail, or your sending them spam?  10.1.10.4 is your mail server?  Or some box on your network.. That software is stopping you from making an outbound connection?  Because what IP is listed 10.1.10.4???  Those IPs are valid mail servers for AOL.. Now if nobody is wanting to send email to AOL then yeah that is a problem.

 

What does your sniff show for these emails?  What does your exchange server show for these emails.  If you were be blocked from the server from sending email from your actual exchange server then its queue would be filling up.  If it was some other process on that exchange server sending email then you would see its connection in netstat to 25 vs just the mta of exchange doing it.

 

 

[Bryan R.]

3 hours ago, BudMan said:

how exactly are you using that software.. That blocks inbound connections to your computer..

 

64.12.91.195 isn't on any blacklist I show..  Nor is that next 152 one.. But yeah that is an AOL mx server IPs.

 

So your sending them mail, or your sending them spam?  10.1.10.4 is your mail server?  Or some box on your network.. That software is stopping you from making an outbound connection?  Because what IP is listed 10.1.10.4???  Those IPs are valid mail servers for AOL.. Now if nobody is wanting to send email to AOL then yeah that is a problem.

 

What does your sniff show for these emails?  What does your exchange server show for these emails.  If you were be blocked from the server from sending email from your actual exchange server then its queue would be filling up.  If it was some other process on that exchange server sending email then you would see its connection in netstat to 25 vs just the mta of exchange doing it.

 

 

10.1.10.4 is indeed the server IP.

I checked the Exchange queue and there is only legitimate outgoing messages in there.. So there is some other hidden process sending that mail.

[Bryan R.]

Quote

IP Address _______ is listed in the CBL. It shows signs of being infected with a spam sending trojan, malicious link or some other form of botnet.
It was last detected at 2016-07-21 18:00 GMT (+/- 30 minutes), approximately 21 hours ago.
This IP address was found to be emitting very large amounts of spam.

Delisting inhibited Follow the above instructions to get it delisted.

So this is still increasing which is good.. This is basically saying that the spam stopped an hour after I got on site yesterday. So either the scans I ran found it or the SPF record had an effect.

[Bryan R.]

I just got the most beautiful email:

Quote

Hello,

 

We have looked at the data for your IP address _______ and have

reset the reputation score for the IP. As a result, it is no longer

listed in CSS.

 

Regards,

 

--

The Spamhaus Project - CSS Team

 

[+BudMan MVC]

spf record wouldn't stop spam..  Its just the good thing todo..  All a spf record says is I know how to run my email server and play nice in the email community...  Here is the IP address(es) of servers that would send email for my domain.  If someone sends you email that says its from mydomain.tld and it didn't come from one of my listed IPs then its most likely spam - I sure did not send it.  Do what you want with it.  But its not from me.

 

Your use of the ~ vs - says softfail it, its not mine but you can do what you want.  if you would of used - that would of said that is NOT FROM ME - ###### can it!!!

 

Maybe that software you posted is blocking it?  But sure looks like it was blocking mail to legit AOL mx records.. So seems like crap software to me..  Do your users send email to AOL users?

 

BTW I still show your IP blacklisted on 5 different lists

 

[Bryan R.]

2 hours ago, BudMan said:

Your use of the ~ vs - says softfail it, its not mine but you can do what you want.  if you would of used - that would of said that is NOT FROM ME - ###### can it!!!

 

Maybe that software you posted is blocking it?  But sure looks like it was blocking mail to legit AOL mx records.. So seems like crap software to me..  Do your users send email to AOL users?

 

BTW I still show your IP blacklisted on 5 different lists

 

Yeah I read the difference of ~ vs - and figured I'd leave it up the mail server to delete or send to junk.

 

The software Bot Revolt seems to do a pretty good job. Those AOL MX records may be legit but the mail being sent certainly wasn't - it wasn't through Exchange so it wasn't from any of the users. It isn't coincidence that when I turned Bot Revolt on, the spam stopped according to CBL. My scans didn't finish until a few hours later which actually found the bot and removed it.

 

Right now, Bot Revolt isn't finding any bad traffic anymore... So again, I don't think coincidence. Right now, it seems the server is clean according to outside CBL and inside with Bot Revolt and scanning.

 

As far as the current blacklists... Spamhaus has three different lists:

Quote

 

<IPADDR> is not listed in the SBL

<IPADDR> is not listed in the PBL

<IPADDR> is listed in the XBL, because it appears in: CBL

 

 

They just removed me from the SBL like 2 hours ago as per the email above. CBL now says:

Quote

IP Address <IPADDR> is listed in the CBL. It shows signs of being infected with a spam sending trojan, malicious link or some other form of botnet.

It was last detected at 2016-07-21 18:00 GMT (+/- 30 minutes), approximately 1 days, 6 hours, 29 minutes ago.

This IP address was found to be emitting very large amounts of spam.

Delisting inhibited Follow the above instructions to get it delisted.

CBL says it may take up to 7 days to be automatically removed

[Bryan R.]

Finally I was gifted a response from CBL saying they've delisted me. I waited 7 days and contacted them via email (cbl@abuseat.org) which they then replied to. Should take an hour for the TTL in DNS to expire and everything will be back to normal.

 

To summarize what I think had the biggest impact on getting this fixed:

Bot Revolt Free stopped the bot from making outbound connections to a list of blacklisted IPs.

Microsoft Malicious Software Removal Tool is actually pretty good at finding bots. It found quite a few things that all other scans did not find.

Avira PC Cleaner is all in German but found quite a few things as well but I ran it at the same time as the former utility.

 

This case was a bit a of a ****storm in the beginning because I did not know if it was the PCs or the server. The PCs were at risk from shady software and the server was at risk because of the user account with admin privileges and no password.

 

Even if the bot was on the PCs, the work with the firewalls were for nothing since Bot Revolt revealed the bot was communicating over random ports.

 

It amazes me that with DNS protections like this in place, spam is still such a problem. This business was shut down from emailing for like 2-3weeks.

[+BudMan MVC]

"outbound connection on random ports to a list of blacklisted IPs."

 

No it didn't go out on random ports.. That is the source port.. And yeah its going to seem random above 1023..

 

"Bot Revolt revealed the bot was communicating over random ports."

 

Again no - mail servers don't get mail on random ports..  They ONLY accept mail on the smtp port 25..  You can see here it was always talking to port 25

 

 

Your edge firewall should block all traffic outbound on 25 from anything other than your server. How a "server" got infected would be my big question.. What was that other guy doing on his server that it would of gotten infected??

[Bryan R.]

3 hours ago, BudMan said:

"outbound connection on random ports to a list of blacklisted IPs."

 

No it didn't go out on random ports.. That is the source port.. And yeah its going to seem random above 1023..

Ok.. Yeah source port vs destination port. So it blocked outbound connections going to port 25..

 

4 hours ago, BudMan said:

Your edge firewall should block all traffic outbound on 25 from anything other than your server. How a "server" got infected would be my big question.. What was that other guy doing on his server that it would of gotten infected??

i mentioned that i was concerned about the server because one user had an account with domain admin rights and no password. So quite possibly that was the way in for this bot.

[+John Teacake MVC]

Slightly off topic but not that much, Bot Revolt looks like a rip off of PeerGuardian/Peerblock whatever it was back in the day. I have had a quick google around and there are a few people reporting that it's a bit shady. Is there nothing else you can use. Like Wireshark with a filter on SMTP/Port 25. That would even tell you the contents of the packets.

[+BudMan MVC]

simple netstat -an should of shown if there was any process other than exchange making network connections.  tcpview from ms would of worked too.

 

Yeah wireshark would been much easier would of actually shown you would of was being sent..  So you could of seen right away if was spam or legit mail.

[Bryan R.]

Yeah guys Wireshark is great and all but what about after you determine where it's coming from? When I went on site, I focused on the server and that is where the bot ended up being. Would Wireshark have told me what process was sending the packets? 

 

In any case, like I said, the issue is solved. The bot was on the server and I eventually found utilities to find and remove it.

[+BudMan MVC]

no wireshark would of not told you the process running on the box, you would of needed something like netstat -anb to find that or tcpview from ms..  But wireshark would of be able to validate what was being sent, you would of been able to read the emails and see if they were spam, etc.  And what was the sending IP.

[sc302 Veteran]

if it were a worm, you would not be able to identify the process easily as it would mask itself as different processes.  You would need to run process hacker, process explorer, or process monitor to see what dll's are being used by the different processes, that would be the only way to identify what is causing it.  But even then there very well could be a background process that loads the dll in the first place either at windows start up or worse at boot when you can't really use utilities to help identify and diagnose where it is coming from....they could hide in the master boot record which makes it not the easiest to remove or identify. 

 

You really should know what you are looking at with the process monitoring tools, you are pretty much on your own with them..a forum would be of little use other than to possibly help verify if a dll/process is good or not.