10+ Minute login times on a Domain

[Frank]

We have about 700 users running off a single Windows 2003 domain controller. We are a remote location for a company that has 3000+ users. It takes a good portion of our users 10+ minutes to login every day. It started happening 5 months ago and as each day goes by it spreads by a few users. After they login it takes them between 10 - 15 minutes to actually see their desktop. Some people say it gets stuck at "Running login scripts" and others say it gets stuck at "Applying personal settings".

If we disconnect the machine from the network, turn the machine on and log the user in with no network connectivity it logs in fine with no delay. We have tried switching the DNS server setting on these machines to the main DNS server at our headquarters but that doesn't make a difference.

I have disabled all of the group policies I have created for our office thinking that there might be one that is causing this issue but it did not make a difference.

No other office is reporting any issues and the company who manages our Active Directory infrastructure are complete morons and haven't been able to figure out what the cause of the issue is.

Any ideas on what I could check or what this might be?

[Shaun N.]

check what policies its trying to apply, are there any changes you have done recently?

Edit after reading it properly..

Do you have roaming profiles?

Edited August 26, 2008 by shaun2312

[Ricardo Gil]

Do you have profiles syncing remotely? If so check the users profile size.

[Joel]

The DNS server your clients are looking for should be the local one for the purposes of your test. Make sure no outside DNS entry is being pushed out in your DHCP settings.

[nautiqueskier]

Its almost definitely a DNS problem, normally caused because the client computers doesn't use the domain dns provider. This often happens if you use your ISP's DNS server which obviously don't have your active directory DNS information.

A network of this size is hard to fix without more knowledge of its technical underpinnings but I would look through the event log on your client and see if anything crops up there. And try changing the clients DNS server to your primary domain controllers IP.

[RichardK]

Try the registry setting change mentioned in this article:

http://support.microsoft.com/default.aspx?...kb;en-us;244474

At the company I work at we had a user that had to wait 20 minutes every morning after logging in before their desktop would come up. Making this change fixed the problem for her.

You can also try changing the timeout for processing GPOs to something like 10 - 15 seconds.

[Frank]

check what policies its trying to apply, are there any changes you have done recently?

Edit after reading it properly..

Do you have roaming profiles?

Do you have profiles syncing remotely? If so check the users profile size.

All Local Profiles

The DNS server your clients are looking for should be the local one for the purposes of your test. Make sure no outside DNS entry is being pushed out in your DHCP settings.

The primary DNS server is set to the local domain controller. The secondary DNS server is set to one of the domain controllers back in the main office. I have been bugging them for a second domain controller out here for a while.

We also have 8 T-1's between us and the main office.

Its almost definitely a DNS problem, normally caused because the client computers doesn't use the domain dns provider. This often happens if you use your ISP's DNS server which obviously don't have your active directory DNS information.

A network of this size is hard to fix without more knowledge of its technical underpinnings but I would look through the event log on your client and see if anything crops up there. And try changing the clients DNS server to your primary domain controllers IP.

All the users machines are pointing to our local domain controller for DNS and a domain controller in our main office for secondary DNS. I will check the logs on the client machines to see if I can find anything.

[jgreenough]

Do you have sites and services set-up correctly? with the remote dc in the correct site? along wth subnet config

[Frank]

I have looked through the logs on one of the machines for the time the user logged in adn what I saw I didn't like. Take a look at the screenshot as well as the text file attached for more info.

emb017eventvwr.txt

Do you have sites and services set-up correctly? with the remote dc in the correct site? along wth subnet config

Yes, thank god (with the help of Neowin) that was setup properly. Just so you can see what I am dealing with - https://www.neowin.net/forum/index.php?showtopic=608521

[Zerosignull]

After the slow login on computers have connected are their any issues connecting to or pining the PDC ?

Can they connect to the internet?

The error log is showing issues with the computers not being able to connect to the DHCP server.

Check the error log on the server computer for problems too.

[Mr Winkle]

as dhcp is a broadcast protocol, it might be that a router is dropping those packets.

i'd set up dhcp on the local server dc in each branch office rather than using the server in the head office, this will reduce quite a bit of traffic across your wan.

just make sure that your servers aren't issuing the same addresses by ensuring the correct address exclusions are in place across all your dhcp servers.

our work network is setup the same way, i did have a single dhcp and dns server doing multiple sites but at busy times i was finding that windows clients couldn't logon to the domain. adding the extra dns (which is really easy with active directory integrated zones) and dhcp has made a notable difference. the other good thing is that if one of the servers does go down, you also have a bit of redundancy.

as a quick test, try giving one of your misbehaving windows clients a static ip address and see if the problem goes away.

Edited August 26, 2008 by Mr Winkle

[Zerosignull]

We have tried switching the DNS server setting on these machines to the main DNS server at our headquarters but that doesn't make a difference.

When you did this did you change the machine's IP too? The loggs are showing that its using APIPA to assign itself an IP and so is in the wrong range&subnet to find your DNS servers.

[Frank]

Some more info I just found out. I originally thought the logs I attached were when the user was logging into the machine. I found out the user did not get in till 8:30 (45 minutes after these errors). I also found another machine that was having the same problem had the same errors that started at the same time (7:45). I am not sure what is causing this to start at 7:45AM. I am trying to find a unaffected machine so I can compare the logs.

After the slow login on computers have connected are their any issues connecting to or pining the PDC ?

Can they connect to the internet?

The error log is showing issues with the computers not being able to connect to the DHCP server.

Check the error log on the server computer for problems too.

I will check to see if they can ping the domain controller. They can connect to the internet fine.

as dhcp is a broadcast protocol, it might be that a router is dropping those packets.

i'd set up dhcp on the local server dc in each branch office rather than using the server in the head office, this will reduce quite a bit of traffic across your wan.

just make sure that your servers aren't issuing the same addresses by ensuring the correct address exclusions are in place across all your dhcp servers.

our work network is setup the same way, i did have a single dhcp and dns server doing multiple sites but at busy times i was finding that windows clients couldn't logon to the domain. adding the extra dns (which is really easy with active directory integrated zones) and dhcp has made a notable difference. the other good thing is that if one of the servers does go down, you also have a bit of redundancy.

as a quick test, try giving one of your misbehaving windows clients a static ip address and see if the problem goes away.

We have a local DHCP server (a Cisco device, not sure which model) and our Local domain controller is setup as our primary DNS server. I will talk to the Active Directory team about temporarily setting up a second DNS server to see if this eliminates the issue. I will also test the static IP to see if that makes a difference.

When you did this did you change the machine's IP too? The loggs are showing that its using APIPA to assign itself an IP and so is in the wrong range&subnet to find your DNS servers.

No, we just manually changed the DNS addresses and ran the following commands.

Net stop netlogon

Net start netlogon

Ipconfig /flushdns

Ipconfig /registerdns

This was not done at the same time as shown in the eventviewer.

[RichardK]

Have you tried that registry change I posted above?

[Frank]

Have you tried that registry change I posted above?

I have not yet, I am waiting for one of the users to log off for the day so I can jump on their machine.

Did you have to do this on every single one of your machines? This problem hasn't always been there but started all at once about 5 weeks ago and each day I get reports of more and more users getting affected.

[]SK[]

How about system resources on the server when people are logging in for the day? 700 users running off one DC sounds like quite a task for that lone DC.

Since Neowin is refusing to let me download your attachments a quick question. Is your DC setup as a Global Catalogue?

[bobbba]

This is easy!

=============================

Event Type: Warning

Event Source: Dhcp

Event Category: None

Event ID: 1007

Date: 8/26/2008

Time: 7:46:26 AM

User: N/A

Computer: US-PBL-D-EMB017

Description:

Your computer has automatically configured the IP address for the Network Card with network address 001321FAEDE4. The IP address being used is 169.254.71.11.

=============================

You have no working DHCP at the site, it's getting an IP from APIPA and with this IP it will talk to little if anything. Ignore the other suggestions until you know that the clients are getting correct ip's. I would also suggest that you make the domain controller the DHCP server site or get the routers at the site to forward to your corporate dhcp.

[+BudMan MVC]

^ yup kind of hard to login with a APIPA address ;) What seems funny is that it would ever work... It would give up an login them in from cache.

But then how are they getting anywhere -- they must be doing something once they login to get a valid IP?

I also have questions on why you would use some cisco device as your dhcp server? Sure you can do that, but dhcp "can" play a big part in your AD dns, as far as placing/removing clients in DNS, etc.

http://technet.microsoft.com/en-us/library/cc787034.aspx

Using DNS servers with DHCP

Im really with bobbba on this -- if your clients are having issues getting IPs, your going to have all kinds of issues! You should not be seeing errors like that in your event log.. All the other errors would follow not having a valid IP.

Your saying more and more clients are starting to have the problem? How long is your dhcp lease? The beginning of the event log has an error about not being able to renew the lease.

[Frank]

How about system resources on the server when people are logging in for the day? 700 users running off one DC sounds like quite a task for that lone DC.

Since Neowin is refusing to let me download your attachments a quick question. Is your DC setup as a Global Catalogue?

I agree. I have requested a second DC but I haven't gotten anywhere. I believe it is setup as a Global Catalog, but I am not 100% sure.

This is easy!

=============================

Event Type: Warning

Event Source: Dhcp

Event Category: None

Event ID: 1007

Date: 8/26/2008

Time: 7:46:26 AM

User: N/A

Computer: US-PBL-D-EMB017

Description:

Your computer has automatically configured the IP address for the Network Card with network address 001321FAEDE4. The IP address being used is 169.254.71.11.

=============================

You have no working DHCP at the site, it's getting an IP from APIPA and with this IP it will talk to little if anything. Ignore the other suggestions until you know that the clients are getting correct ip's. I would also suggest that you make the domain controller the DHCP server site or get the routers at the site to forward to your corporate dhcp.

DHCP does work on site. After the user logs in they will have an IP address. I saw that in the logs as well and was very confused by it. The logs I posted were 45 minutes before the user actually logged in.

I can recommend it but I am not the "Decision Maker" when it comes to stuff like this. I will do some searching on the net and if I can find articles to back this up I can probably get them to change it.

After doing some more research our Network Team states our main router is our DHCP server. I was under the impression that the IP for the router was 10.15.91.254. Inside ipconfig /all it shows 10.15.91.1. I am trying to get them to clarify this.

^ yup kind of hard to login with a APIPA address ;) What seems funny is that it would ever work... It would give up an login them in from cache.

But then how are they getting anywhere -- they must be doing something once they login to get a valid IP?

I also have questions on why you would use some cisco device as your dhcp server? Sure you can do that, but dhcp "can" play a big part in your AD dns, as far as placing/removing clients in DNS, etc.

http://technet.microsoft.com/en-us/library/cc787034.aspx

Using DNS servers with DHCP

Im really with bobbba on this -- if your clients are having issues getting IPs, your going to have all kinds of issues! You should not be seeing errors like that in your event log.. All the other errors would follow not having a valid IP.

Your saying more and more clients are starting to have the problem? How long is your dhcp lease? The beginning of the event log has an error about not being able to renew the lease.

We have TONS of issues. I am not sure if it has an IP during login or if there is no IP and it is trying to get one when we login which is what the delay is from. I will pull my ipconfig and get you the other answers tomorrow but I think it is 5 days.

I will take a look at that article and try and forward it on to the team who manages these boxes.

I appreciate all the help everyone has given!

[]SK[]

What subnet are you using?

Any idea how many IPs your DHCP server has to lease?

DHCP runs from where?

GC = http://support.microsoft.com/kb/313994

Didn't know that it required a reboot to make a server a GC. Learn something everyday :)

[bobbba]

Start with the basics. Is DHCP definitely working ok? The event log entries you posted indicated otherwise. If it is, how about confirming that the slow logon is still occuring. Clear the client eventlog reboot it and post the events that are related to the slow logon.

The registry fixes, profile suggestions and global catalogue suggestions are valid but I would not pursue them until you know the basics are ok. It could take you a long time and require you to make changes that are not neccessary.

Whilst the performance of the DC and it's load could come into it, the scale of the slow logon would be more likely down to basic issue like an ip or dns problem. Next things to check would be that your clients aren't all trying to reach something across you wan link uneccesarily (what speed is your wan link by the way?).

[Frank]

What subnet are you using?

Any idea how many IPs your DHCP server has to lease?

DHCP runs from where?

GC = http://support.microsoft.com/kb/313994

Didn't know that it required a reboot to make a server a GC. Learn something everyday :)

255.255.252.0

We have 10.15.88.x - 10.15.92.x in our DHCP range. There are about 150 IP's outside of the range that are used for static IP's (now and in the future).

DHCP is running from our main router.

Start with the basics. Is DHCP definitely working ok? The event log entries you posted indicated otherwise. If it is, how about confirming that the slow logon is still occuring. Clear the client eventlog reboot it and post the events that are related to the slow logon.

The registry fixes, profile suggestions and global catalogue suggestions are valid but I would not pursue them until you know the basics are ok. It could take you a long time and require you to make changes that are not neccessary.

Whilst the performance of the DC and it's load could come into it, the scale of the slow logon would be more likely down to basic issue like an ip or dns problem. Next things to check would be that your clients aren't all trying to reach something across you wan link uneccesarily (what speed is your wan link by the way?).

I sent an email off to the network team last night requesting them to look into the errors we are seeing on the machines. We put a new re-imaged machine on the floor last night to replace one that a user was getting the slow login times issue. I talked to her this morning and she no longer has the problem. I also checked the logs on her box and I saw a similar result to the logs from her old machine. I have attached a screenshot.

I don't think they would be accessing anything over the wan during login. What would be the best way to check this? We have two routers in place with 4 T-1's each to make a completely redundant connection. When it was first setup it was only set for auto failover. After I bitched they set it to load balance.

[bobbba]

In your latest screenshot there's still DHCP warnings. Either these are a distraction from an unused network connection or there's still basic problems with DHCP.

Are these the same as the earlier ones(when you open them)?

Is this client a laptop or a pc?

are there network adaptors on the client that are not being used(maybe wifi?) but are still enabled?

Is the PC later able talk to network resources?

[Frank]

In your latest screenshot there's still DHCP warnings. Either these are a distraction from an unused network connection or there's still basic problems with DHCP.

Are these the same as the earlier ones(when you open them)?

Is this client a laptop or a pc?

are there network adaptors on the client that are not being used(maybe wifi?) but are still enabled?

Is the PC later able talk to network resources?

Yes, they are they have the same details inside of them.

This is a PC

There is no other network adapters, just the single NIC

Yes, the pc works fine when logged into and this specific user did not have a login issue after we gave her this PC, but she still has this info in the logs.

[bobbba]

in your first screenshot there was a netlogon warning which is could not locate a domain controller which would tie up with the apipa 169.x.x.x address. there is no netlogon error visible(or is it just off the screen!) in your second screen shot so it does not look the same....

from the second screenshot is the dhcp event 1007 still saying "The IP address being used is 169.x.x.x"?

if you reboot the pc, logon locally and do an ipconfig, what ip do you get for the client?

does it stay the same after an ipconfig -release and -renew?

There seems to be a contradiction between the dhcp event 1007 with it's 169.x.x.x address and the user logging on and getting network resources.