win2003 domain controller flood of outbound port 53 UDP

[+Raphaël G. MVC]

hello neowinians,

I will need your expert help on this one. One of my customers has a windows server 2003 domain controller, and lately we've been noticing a flood of outbound port 53 UDP connections originating from the domain controller. While I know port 53 TCP and UDP is being used by DNS, I'm confused about what could possibly generate such load of traffic.

The network performance at the customers is suffering a lot from this flood... I'm afraid this could be some sort of attack, but why are the connections outbound?

Possibly usefull info:

Some of the client computers on the network (they us DNS from the domain controller) show a lot of outbound UDP connections in high ports (+30000). Could they be the cause of the UDP 53 outbound connections on the domain controller?

Many thanks for reading this people!

[+BudMan MVC]

Hmmm if you have clients doing a lot of dns queries, sure the source port is going to be random port above 1024 to port 53 UDP, udp is the default dns port - it only switches to tcp if failure or doing a zone transfer, etc. UDP is limited is size, so it will switch to tcp when doing large requests, etc.

example of dns query from one of my clients to my dns.

13:10:13.082077 IP quad-w7.local.lan.60548 > p4-28g.local.lan.53: 1627+ A? www.google.com. (32)

13:10:13.082634 IP p4-28g.local.lan.53 > quad-w7.local.lan.60548: 1627 7/4/0 CNAME www.l.google.com.,[|domain]

As you can see its source port is 60548 to 53, and then when the server answers back its from 53 to that destination port of 60548

You could watch the dns traffic with your fav sniffer, be it wireshark or something as simple as windump (windows version of tcpdump)

http://www.mirrorservice.org/sites/ftp.wiretapped.net/pub/security/packet-capture/winpcap/windump/

Or you could use something like dnstop - if you a linux box that you can view traffic on your network with -- this can give you great summary and insight to what queries are being made.

http://dns.measurement-factory.com/tools/dnstop/

If a client is infected with something this could generate lots of dns, or say something like p2p can generate a **** load of dns queries if the client is setup to lookup all the connections host name, etc.

You might want to enable logging on dns so you can view these queries and see what is being looked up.

http://thelazyadmin.com/blogs/thelazyadmin/archive/2006/02/16/DNS-Tips-_2300_14-_2D00_-DNS-Logging.aspx

But yeah if you have a client that is generating lots of queries to your dns that are not part of your domain or cached already, then up your dns box is going to have to either forward that to your isp dns, or directly query root servers for all the requests.. So if you have lots of clients doing lots of queries for something like p2p -- it could generate a **** load of outbound queries from your dns box. It also just be typical web browsing -- if you have a lot of clients doing a lot of browsing to different sites that are not cached - yup your dns is going to have to go get the answers.

Getting a handle on what is actually being looked up will help you track down the root cause of all the queries.

edit: BTW you have your clients only setup to query your Domain Controller for DNS -- right?? Members of Active Directory should have only AD dns servers listed.. So when you say your clients are creating outbound dns queries - they are to your DC correct?? Client computers should not have any dns servers listed other than your AD dns server(s).

edit: Also don't be caught off guard on how much dns going to a single site can generate. Its quite possible for a simple site like neowin to generate lots of queries - ads on the page are being served from non neowin servers. You have other hosts on sites like images.domain.tld, css.domain.tld, js.domain.tld, forums., news. it depends on how the site is coded, etc. But a single website could quite easy generate 10 queries for FQDN, etc. So if you have 100 users browsing, it could generate quite a bit of dns traffic. Now hopefully your dns is caching these entries -- so as things become cached you'll see less outbound queries. But yeah it can be amazing how much dns traffic 100 users browsing can create ;) Now throw in some spam malware or p2p and sure it could overload your dns quite easy.

edit: for example I just checked on just going to the news page on neowin.net -- look at all the fdqn that would have to be looked up

a.colletive-media.net

ad.doubleclick.net

ad2.netshelter.net

ajax.googleapis.com

google-analytics.com

googleads.g.doubleclick.net

kona.kontera.com

kona5.kontera.com

konax.kontera.com

neowin.net

pageads2.googlesyndication.com

s0.2mdn.net

spe.atdmt.com

te.kontera.com

view.atdmt.com

so see how lots of people browsing to lots of sites could quite quickly cause lots of queries ;)

Edited February 8, 2010 by BudMan

[Sn00pY]

Best thing you can do is wireshark it (sniffer) really. We can only make assumaptions as to what it is otherwise... DNS is a "chatty" protocol especially if there is a lot of web browsing going on.

Get a dump then we can take a better look.

[+BudMan MVC]

Not sure I would use the term chatty ;)

But yup a name server having to do a full recursive lookup for some fqdn can generate some traffic for sure ;) Keep in mind has to find out what nameserver owns domainx, then has to query domainx name server for the host record, etc. So yeah there can be some steps involved - now multiple that by how many domains can be on a single website - see my neowin.net example. And yup you can create some dns traffic to be sure..

Example to be able to find www.neowin.net

now I snipped alot of this info out -- but to get the www.neowin.net IP had do these queries

; <<>> DiG 9.2.4 <<>> @4.2.2.2 www.neowin.net +trace

;; Received 228 bytes from 4.2.2.2#53(4.2.2.2) in 1 ms

;; Received 501 bytes from 192.33.4.12#53(C.ROOT-SERVERS.NET) in 2 ms

;; Received 100 bytes from 192.41.162.30#53(L.GTLD-SERVERS.net) in 72 ms

;; Received 180 bytes from 209.124.63.212#53(ns1.neowin.net) in 72 ms

Ask for nameservers for .net, then asks .net nameservers for servers for neowin, then ask neowin nameservers for www, etc..

Here is a pretty good write up on how recursion works.

http://www.windowsnetworking.com/articles_tutorials/Understanding-DNS-Recursion.html

So you can see 1 simple query for 1 host record to DNS, generated 3 queries all going to be outbound on 53, etc.

[Sn00pY]

As I said... It's a chatty protocol... Considering 1 request made 3 connections ;)

[+BudMan MVC]

Buts its not really the protocol that is being chatty.. It handles quite a bit of info with very little amount of traffic.. How else would you do recursion with less queries? There has to be some method of looking up who is the owning server of say neowin.net. Its not going to be able to sit in 1 giant database ;)

A Chatty protocol is something that either does a lot of unnecessary advertisement of itself -- ie sending out broadcasts all the time as example.. Or requires alot of back and forth traffic, ie waiting for acknowledgment before transmitting again, etc. CIFS for example has a lot of back and forth traffic, I would say smtp could be seen as chatty, etc. I would not really call dns chatty -- you send a query over udp, it sends you the info back.. Pretty streamlined if you ask me ;)

Problem is recursion, even though the exchange of info is pretty streamlined - you have to ask a few servers how to find the owning server of domainX. Which is where TTLs and caching comes into play.. Once I know this info, I don't have to go ask again until that TTL has expired -- so the next guy asking, I either have the info - or know the IP of the nameserver for domainX already and can ask him directly for a different host vs having to do the full recursion again, etc.

[Sn00pY]

I still find it funny you don't deem it chatty yet each time you go on to mention recusrion - which imho is chatty... eitherway it's not important for this particular issue.

[+BudMan MVC]

Look up the definition of chatty protocol ;)

DEFINITION - A chatty protocol is an application or routing protocol that requires a client or server to wait for an acknowledgement before it can transmit again. Such protocols are called "chatty" because they require a lot of back-and-forth communication.

Does DNS fit that?

chatty protocol

Home > Library > Technology > Computer Encyclopedia

A network protocol in which servers constantly advertise their availability over the network. The continuous transmission of these additional packets adds to network overhead. See broadcast.

Again does DNS fit that?

Also recursion does not always take place -- in an AD environment looking up servers there is no recursion required, only when asking a server for records it does not own does it take place. If you look at it from the point of view of the owning server -- there is not a lot of chatter. It gets asked for say host www or smtp or ftp, etc. And it answers - one query, one response. very small amounts of data with a lot of info in them, etc.

I agree the definition of chatty is not important to this topic, and I agree with you dns can cause a lot of traffic -- depending ;) But I would not consider it a "chatty" protocol in general.

[Sn00pY]

lol BudMan <3

[+BudMan MVC]

heheh -- so we are on the same page - I just personally would not call it chatty ;)

No to a better question - do you think we will ever hear back from the OP..