Inconsistent routing issue.

[StrikedOut]

Hi All.

I am currently in the middle of a data centre migration, moving from physical boxes to virtual but as part of it I am also separating systems that don't seem to play nice together. So the setup is like this;

3 sites, Site A, B and C. all connected via and MPLS circuit to our new ISP who then routes traffic to the new vDC (virtual data centre). The internet breakout for the users is also at the ISP but all web/public facing servers in the vDC breakout from the vDC. This way if the link between ISP and vDC went down we would still be functional albeit at a reduced capacity.

This means the servers in the vDC that are not public facing have a gateway ending in 200.254 so all traffic is routed back to the MPLS network. The servers that are public facing have a gateway ending in 200.1 which is the gateway at the vDC but persistent routes forwarding all MPLS traffic to the 200.254 GW.

I am based in Site A and this works perfectly for me. I can access all the servers and all the web facing ones are still routing traffic to the public domain when requested. Site C is the same. Site B however where the team who need to test the new servers is based can ping some but not all of the servers. To confuse the matter a little more, some users are able to ping more servers than others and one isn't able to ping any.

So this is what I have done.

Removed the static route and added it again.

Cleared the TCP/IP stack and reset the winsock catalogue re-adding the static routes again.

Reset the server between each step of the above and have done the same on the servers and client PC.

Tried using Wireless and cabled connections as the client is on a laptop.

The servers that the client is unable to connect to is Server 2003 R2 and 2008 R2, the client is on Win 7 x64.

I suspect that there is something obvious that I am missing but looking at this problem for a couple of days and I cant see it.

Any thoughts?

[sc302 Veteran]

perhaps the routing isn't propagating on all switches on the network. check your route tables, do a trace to see where the packet is dropping.

[StrikedOut]

I have done a tracert from both the client and the new server and each time it hits the correct gateway but times out thereafter. I can also ping the gateway in the DC from the site and vice versa. If it was a routing issue wouldn't it be more of an all or nothing scenario?

I also thought about firewalls so all are disabled on the servers (and will stay that way) but no change.

I am going to get the ISP to check routing between the sites, this site only has one difference to the other 2 which is it is an FTTC connection but I cant see this making any difference.

[sc302 Veteran]

If it hits the switch that the server is on, are the route tables on the switch right? Does that switch know about all of the networks and how to get there. Is there another device after the switch causing issues/conflicts?

[S.MULLA]

first thing to check is to see if the routers on each site have the same routing tables for your 3 different networks. If you add a new route at site A then site B and site C should have it within a few seconds (if your using cisco equipment and same eigrp instance).

What you dont want to end up with is site B trying to contact site A- but its routing traffic through to Site C to get to site A. Using Microsoft Network Monitor (download from google) and install on client machine and you will be able to see where his request are going to.

[StrikedOut]

All resolved.

It seems that there was a configuration issue on one of the routers, it seems like they set all the routes the same but from their email it seems like it is a little different if coming through an FTTC connection. So now all servers and clients are connecting as they should.

Anyway, thanks for the input.