Redoing my network... (Part 2)

[riahc3]

Well Ive gotten all my new network equipment so Im ready to go :)

(Thunder sign indicates wireless)

This is my current network:

As you can see I have two APs (one acting as router) both have the same SSID but are on different channels and configured to work as a wired to wireless bridge between both DD-WRT routers.

This is what I want my new network to be:

There is a N54L with ESXi 5.5 that will act as my gateway and firewall with pfSense. It has 3 ports: One integrated and another dual port that will connect to the modem (WAN) and another that connects to the switch. It has a forth one but that is dedicated to RAC.

The red line I drew is because I am confused if that LAN cable is the same as the my LAN side on the pfSense AND the same as my LAN side on WS2012R2. Confused on there and new to virtualization so :)

The one that was my main router will be converted strictly into a AP.

Wireless: Sadly we have 802.11g devices in our home still. The number of wireless clients varies but I wanted to put phones as its more or less what is going to be used wirelessly. Im sure at least ONE of the phones is 802.11g and the TV is also problably 802.11g Both APs are 802.11n capable. The bottom one was used to extend the range of the top one since it didn't cover the entire house.

Don't know if its of intrest but the current main router and access point is a TLWR1043ND and it is running DD-WRT v24-sp2 (03/19/12) std (SVN revision 18777) The middle floor is a TL-WR841ND it is running DD-WRT v24-sp2 (04/13/11) std (SVN revision 16785).

The N54L has ESXi 5.5 U1 installed on a USB drive. Im putting a 320GB to use as the datastore to install both pfSense (Im gonna give it 3GB of space and 1GB of RAM) and WS2012R2 (250GB of space and 7-8GB of RAM). Some left over space for random VMs. 4 x 3TB HDDs also as data space in a storage pool in WS2012R2.

Not sure If im missing any information. Basically wanted to know if my network setup is correct and if I should think about anything before hand. If there is any info Ive left out or something, please feel free to comment.

Thanks to all :)

[Krome]

Your "wireless" is heating up the atmosphere and building up a lot of carbon-monoxide from what I can tell from the art. lol :laugh:

[sc302 Veteran]

Looks and sounds right.  3 nics and 1 rac.  2 nics in use, one rac in use, 1 remaining open for future growth/load balancing/or another subnet.  You can have that other nic dedicated to management if you so choose.

[+BudMan MVC]

I didn't want to redraw your whole network - but need to be clear on virtual and physical network.

Maybe something like this

So your esxi has physical nics (show on drawing) that are connected to physical hardware - your modem and your physical network switch setup. Then inside your esxi host you will create virtual switches one that is for your (wan/internet) and other for other networks you might create.. In this case your physical switched network.

So pfsense will have 2 virtual nics (not shown) one connected to the wan vswitch and other connected to lan vswitch. your VMs would only have virtual nic connections to your lan vswitch.

Your vmkern is not shown, this could be on your lan vswitch or on its own switch with another cable running to your physical switch. Or I would prob break out another network segment to the 3rd nic in the esxi host and connect that to your wireless network.

So what is your plan for the vmkern port group?

[+Fahim S. MVC]

What is the GS105E (192.168.1.5) doing between the GS108E (192.168.1.8) and the switch/AP (192.168.1.2)?

 

Wouldn't you just be better off putting a cable between the GS108E (192.168.1.8) and the switch/AP (192.168.1.2)?

[+BudMan MVC]

^ yeah that makes little sense unless not showing stuff wired on the 105E or the 108E taking up the ports? Or physical location comes into play?

From his esxi host at most I could see 3 connections. 1 for the rac, 1 for the lan and 1 for the vmkern. His workstation and the 2 AP doesn't = 8 ports ;)

I personally would share the vmkern on the same nic and use the 3rd nic for wireless segment

[riahc3]

First, thank you all for the help.

 

Looks and sounds right.  3 nics and 1 rac.  2 nics in use, one rac in use, 1 remaining open for future growth/load balancing/or another subnet.  You can have that other nic dedicated to management if you so choose.

I have 1 nic, the internal one with the N54L, another nic with 2 ports (same as BudMan's, the HP NC360T) and the RAC which has its own port. AFAIK, I cannot use the RAC for any other purpose except remote access. This leaves me with three physical ports. 2 porst off the HP I am going to use for sure for my WAN and LAN on the pfSense. I have no plans for the internal N54L one.

 

 

I didn't want to redraw your whole network - but need to be clear on virtual and physical network.

Maybe something like this

physical-virtualnetwork.png

So your esxi has physical nics (show on drawing) that are connected to physical hardware - your modem and your physical network switch setup. Then inside your esxi host you will create virtual switches one that is for your (wan/internet) and other for other networks you might create.. In this case your physical switched network.

Yes, on the physical side that is correct. vswitches are a new conecept to me so you are going to have to bare with me :) I understand basically they are virtual switches that if I want to make a bit more complex networks, I am free to configure them.

 

So pfsense will have 2 virtual nics (not shown) one connected to the wan vswitch and other connected to lan vswitch. your VMs would only have virtual nic connections to your lan vswitch.

The pfSense should have 2 virutal NICs connected to 2 different ports on the HP NC360T. I am not completely sure on this.

 

Your vmkern is not shown, this could be on your lan vswitch or on its own switch with another cable running to your physical switch. Or I would prob break out another network segment to the 3rd nic in the esxi host and connect that to your wireless network.

Not sure what my vmkern (Virutal Machine Kernel) is for. Sorry BudMan and others.

Why would I connect my wireless AP on its own? Is this for performance?

 

So what is your plan for the vmkern port group?

No idea.

 

What is the GS105E (192.168.1.5) doing between the GS108E (192.168.1.8) and the switch/AP (192.168.1.2)?

 

Wouldn't you just be better off putting a cable between the GS108E (192.168.1.8) and the switch/AP (192.168.1.2)?

Well, my GS105E is basically full right now: My PC, downstairs, the N54L's RAC, the (right now) router and the other port is connecting to the GS108E. The GS108E is right now empty (except obviously to the GS105E). I believe I can get by using ONLY the GS108E if I havent calculated wrong.

 

 

^ yeah that makes little sense unless not showing stuff wired on the 105E or the 108E taking up the ports? Or physical location comes into play?

From his esxi host at most I could see 3 connections. 1 for the rac, 1 for the lan and 1 for the vmkern. His workstation and the 2 AP doesn't = 8 ports ;)

I personally would share the vmkern on the same nic and use the 3rd nic for wireless segment

I messed up on the drawing; My .13 PC is actually connected to the GS105E sorry.

[+BudMan MVC]

Vswitches are not for more complex networks, vswitches are a requirement.. You don't connect nic to nic do you? If have computers that you want to network together what do you do?? You connect them to the same switch right. ;)

Well what switch do you connect a VM too? A vswitch, so all the vms that want to talk to each other normally would connect to the same vswitch. Now how does that vswitch connect to the physical world? Because you connect a physical nic on the host too it. For your dual port nic, it will show up as 2 different nics in esxi

See the 4 nics, and what vswitches they are tied too.

pfsense is not directly connected to the nics on your hp dual card, your ports on the nic ard connected to vswitches. Your vms nics (virtual) are connected to the switches you want to connect to.

This allows you to connect multiple vms to the same physical network. This is why I drew it out so there could be no questions! ;)

Your going to have 2 network segments your wan and your lan.. So pfsense since its your router and firewall needs an interface in both segments. All your other vms would end up connected to your lan vswitch, which in turn is connected to your lan physical network

Look at all the vms I have connected to my lan vswitch - which is in turn tied to my physical lan.

Just how you have your 105E and 108E connected together.. This same thing just one is virtual inside esxi, this is how you connect your physical network to your virtual network.

Your vmkern is what manages the host is one way to look at it, so you connect your vclient to the esxi host to manage it right. Well this is connected to the vmkern

"The VMkernel is the liaison between virtual machines (VMs) and the physical hardware that supports them. VMware calls VMkernel a microkernel because it runs on bare metal, directly on VMware ESX hosts. The VMkernal is responsible for allocating memory, scheduling CPUs and providing other hardware abstraction and operating system (OS) services. "

The vmkern needs a connection to the real world - so you can either put it own vswitch connected to its own physical nic in the esxi host. Or you can put the vmkern port group on the same vswitch that is connected to your lan. As to performance - I did notice that moving files back and forth to the datastore from my real network was faster with it on its own connection. But to be honest its rare that you put anything on the datastore.. Just new iso's you might need to install new VM OSes, etc. So if you don't have the physical ports why waste one on it. I broke mine out because I had a spare nic on my esxi host.

If I find a need to have another physical segment I would not hesitate to put mine back shared on my lan vswitch.

As to why you would put your wireless on its own segment.. Performance has little to do with it, it comes down to security. Since your wireless is on its own segment routed through your firewall you can filter what can talk to what between segments. If you don't see how that is of use, then no you have no need to break it out. Performance wise, creating 2 segments means 2 broadcast domains - so all your wired clients are not sending broadcasts to your wireless network, and your wireless clients are not broadcasting on your wired network, etc.

To be honest the fact that you have to ask means you shouldn't do it ;) It will break stuff that uses broadcast, like chromecast or airprint if you like to use network browsing in windows, etc. You have 2 different segments that won't share that info, etc.

I really am curious here - do you not work in IT? Do you not understand the purpose of network segments? I am not trying to be smart or an ass here - I am really curious. How do you work in IT and not understand these basic concepts? Do you only do servers? I am always just shocked at the complete lack of basic networking understanding from people in IT.. I only can base this on my own experience -- back when I started we didn't even have tcp/ip ;) It was all ipx/spx and netbeui and lanman was new back in early 80's -- when we converted over our stuff to tcp/ip you had to understand how it worked.

I think in this day an age you get people more isolated into one silo or the other, guess I am just old school jack of all trades sort of guy ;)

More than happy to teach a networking 101 class if you want ;)

[riahc3]

Vswitches are not for more complex networks, vswitches are a requirement.. You don't connect nic to nic do you? If have computers that you want to network together what do you do?? You connect them to the same switch right. ;)

Well what switch do you connect a VM too? A vswitch, so all the vms that want to talk to each other normally would connect to the same vswitch. Now how does that vswitch connect to the physical world? Because you connect a physical nic on the host too it. For your dual port nic, it will show up as 2 different nics in esxi

See the 4 nics, and what vswitches they are tied too.

Ah, I see.

Since my VMs are going to have virtual nics, they have to connect to a virtual switch before going to a physical nic. Like you said, (virtual) nic to (physical) nic is a no-no.

pfsense is not directly connected to the nics on your hp dual card, your ports on the nic ard connected to vswitches. Your vms nics (virtual) are connected to the switches you want to connect to.

This allows you to connect multiple vms to the same physical network. This is why I drew it out so there could be no questions! ;)

Your going to have 2 network segments your wan and your lan.. So pfsense since its your router and firewall needs an interface in both segments. All your other vms would end up connected to your lan vswitch, which in turn is connected to your lan physical network

Look at all the vms I have connected to my lan vswitch - which is in turn tied to my physical lan.

lanvswitch.png

Just how you have your 105E and 108E connected together.. This same thing just one is virtual inside esxi, this is how you connect your physical network to your virtual network.

It now makes sense. Thank you for the explanation.

Your vmkern is what manages the host is one way to look at it, so you connect your vclient to the esxi host to manage it right. Well this is connected to the vmkern

"The VMkernel is the liaison between virtual machines (VMs) and the physical hardware that supports them. VMware calls VMkernel a microkernel because it runs on bare metal, directly on VMware ESX hosts. The VMkernal is responsible for allocating memory, scheduling CPUs and providing other hardware abstraction and operating system (OS) services. "

The vmkern needs a connection to the real world - so you can either put it own vswitch connected to its own physical nic in the esxi host. Or you can put the vmkern port group on the same vswitch that is connected to your lan. As to performance - I did notice that moving files back and forth to the datastore from my real network was faster with it on its own connection. But to be honest its rare that you put anything on the datastore.. Just new iso's you might need to install new VM OSes, etc. So if you don't have the physical ports why waste one on it. I broke mine out because I had a spare nic on my esxi host.

If I find a need to have another physical segment I would not hesitate to put mine back shared on my lan vswitch.

OK, Ill problably use it on the same vswitch as the LAN of pfSense.

As to why you would put your wireless on its own segment.. Performance has little to do with it, it comes down to security. Since your wireless is on its own segment routed through your firewall you can filter what can talk to what between segments. If you don't see how that is of use, then no you have no need to break it out. Performance wise, creating 2 segments means 2 broadcast domains - so all your wired clients are not sending broadcasts to your wireless network, and your wireless clients are not broadcasting on your wired network, etc.

To be honest the fact that you have to ask means you shouldn't do it ;) It will break stuff that uses broadcast, like chromecast or airprint if you like to use network browsing in windows, etc. You have 2 different segments that won't share that info, etc.

It seems that it would complicate things on my setup; Being three users and streaming to media devices, I think it would complicate things.

I really am curious here - do you not work in IT? Do you not understand the purpose of network segments? I am not trying to be smart or an ass here - I am really curious. How do you work in IT and not understand these basic concepts? Do you only do servers? I am always just shocked at the complete lack of basic networking understanding from people in IT.. I only can base this on my own experience -- back when I started we didn't even have tcp/ip ;) It was all ipx/spx and netbeui and lanman was new back in early 80's -- when we converted over our stuff to tcp/ip you had to understand how it worked.

I think in this day an age you get people more isolated into one silo or the other, guess I am just old school jack of all trades sort of guy ;)

More than happy to teach a networking 101 class if you want ;)

I don't consider it a insult IMO. I think you have enough knowledge to question someone else's in networking.

My current job (which I hate) is programming. Ive been tasked to do also small network tasks such as setting up equipment, a OpenVPN server, offsite network assistance, etc. small tasks. Also, this year, Ive started with WS2003SBS (horrible way to start but its what is available) so Im wetting my feet in DNS and DHCP without relying on your basic SOHO ADSL router. I have no control over the IT budget which sometimes limits my knowledge and abilities to perform/try certain tasks.

I believe network segments (subnetting) is a way to split larger networks into smaller network segments which cannot communicate with each other directly.

My professional goal in life is networking but (as you can see) I have a LONG way to go.

Ive always thought it would be a treat if you could post in the guides section a "Network 101" I think a lot of people would read it and it would give out pointers on simple concepts that might be needed for simple networking.

Thank you as always BudMan.

[+BudMan MVC]

Ah "programming" its own silo ;) But don't your programs have to talk over a network, so basic understanding if not even higher level understanding of the protocols would be needed in having your program communicate over the "network"

 

If your goal is network, be warned being a switch/router jockey can be "boring" ;)  Now troubleshooting why something is not working is where my passion is and figuring out what is not working from a network sniff is always fun!!! But building out a network that someone else designed not so much ;)  Ie adding vlans/routes to the network is not very rewarding.

 

One piece of advice I would give - is while its great to be good at what your silo is, don't forget to understand how the other pieces of IT work together.  Understanding network is great, but if you don't understand how the "servers" use the protocols over your network to provide the users a service.  Understanding how to manage AD and how 2k12r2 and setting up hyper-V -- great.  But when it can't talk to the other server and you don't know how to check that it can talk to its gateway, or what a gateway even is - or what a route is its sad..

 

Sounds like your getting sucked in -- Hey this guy wrote the code to run the factory machine, he must know how computers work ;)  He can setup do X, it has a computer!!

 

Have fun is the most important part!!

[riahc3]

Ah "programming" its own silo ;) But don't your programs have to talk over a network, so basic understanding if not even higher level understanding of the protocols would be needed in having your program communicate over the "network"

Well, not really: You just read a function that says input your pass, user, etc. and we will give it the proper output. Doesn't care if its UDP, TCP, etc. Most function just do it.

Of course, this is higher-level programming. When you are opening communications sockets, it gets a bit more interesting.

 

If your goal is network, be warned being a switch/router jockey can be "boring" ;)  Now troubleshooting why something is not working is where my passion is and figuring out what is not working from a network sniff is always fun!!! But building out a network that someone else designed not so much ;)  Ie adding vlans/routes to the network is not very rewarding.

I agree that it is a pain probably but at the end of the day, its a choice and of course, the only reward is not only fun, but funDS.

 

One piece of advice I would give - is while its great to be good at what your silo is, don't forget to understand how the other pieces of IT work together.  Understanding network is great, but if you don't understand how the "servers" use the protocols over your network to provide the users a service.  Understanding how to manage AD and how 2k12r2 and setting up hyper-V -- great.  But when it can't talk to the other server and you don't know how to check that it can talk to its gateway, or what a gateway even is - or what a route is its sad..

I completely agree. Sometimes just putting something together and making it work isn't really fun because you know its gonna work. Troubleshooting is fun. For me, it used to be fun but now after programming, it is frustrating because it bores me.

BTW, we are completely getting offtopic :laugh:

 

Sounds like your getting sucked in -- Hey this guy wrote the code to run the factory machine, he must know how computers work ;)  He can setup do X, it has a computer!!

Im not getting sucked in, Im just severely getting underpaid :laugh: ; I started writing a web page and now Im drawing and doing a database scheme/design for our machines in different factories and getting all the DBs to replicate with their master. And Im getting paid the same crap.

Have fun is the most important part!!

Its something Ive heard SO much but Im gonna have to (semi)disagree. You can have fun at work but you need something that fills you inside with something that you like to do or want to learn to do and getting paid at the same time.

BTW, we have gone COMPLETELY offtopic with this conversation :laugh: My apologies to other members.

Im leaving in about a hour so lets me see if I can get some hours dedicated to the N54L.

[sc302 Veteran]

No you have to have fun.  I don't have the mindset for programming, if you do then you can go places.  It can be fruitful and frustrating all at the same time.  If you dont like it then get out now. 

 

You have to have fun and a lot of it.  You are doing choosing to do this for the rest of your life.  If you do not have fun, and enjoy it and enjoy the challenges that it brings you will also grow to hate this.  You have a lot to learn and really need to get your head together before venturing off into a different area.  Fun is first and foremost or it becomes tedious, when it becomes tedious you no longer have the drive or the willingness to work.  It is like a marriage, if you don't have fun you begin to hate the person you are with, when you hate the person you are with it usually ends in divorce or worse.

[StrikedOut]

......

 

Ive started with WS2003SBS (horrible way to start but its what is available)

Could be worth pointing to your boss that S2003 in all carnations is out of mainstream support and only has a little over a year in extended support. Argue that it leaves you open to vulnerabilities. I would also guess that the hardware is out of any sort of maintenance agreement and could all be changed at the same time. I just did this for all of my sites and I managed to bring the disaster recovery times down from days to hours. I managed to make so many improvements that I am trying for a internationally recognised standard in DR (ISO 22301), gonna be an interesting time.

I've always thought it would be a treat if you could post in the guides section a "Network 101" I think a lot of people would read it and it would give out pointers on simple concepts that might be needed for simple networking.

I'm sure I have said Budman should wright a book with his knowledge, Id get it!

[+BudMan MVC]

"You can have fun at work but you need something that fills you inside with something that you like to do or want to learn to do and getting paid at the same time."

I think maybe we are saying the same thing just lost in translation ;)

Whats the old saying

Choose a job you love, and you will never have to work a day in your life ;)

Same goes for this

Do not hire a man who does your work for money, but him who does it for love of it.

[riahc3]

Well, Ive set up pfSense.....sorta :laugh:

 

Ive set up perfectly pfSense and WAN works but now I have pfSense's DHCP server disabled and DDWRT's DHCP server enabled. I seem to have some DNS trouble.

 

Checking it out...

[+BudMan MVC]

Why would you do that? Pfsense should be your dns and dhcp unless your running AD or have some other reason to provide those services on something else? Why would you have dhcp off a AP?? More than likely it points to itself for dns and the gateway, etc.

Do you have public IP on pfsense wan, or are you double natting?

[riahc3]

Here is my DNS settings:

 

Why would you do that? Pfsense should be your dns and dhcp unless your running AD or have some other reason to provide those services on something else? Why would you have dhcp off a AP?? More than likely it points to itself for dns and the gateway, etc.

Do you have public IP on pfsense wan, or are you double natting?

I ment it in reverse: Ive disabled DDWRT's DHCP and enabled pfSense's.

 

Im getting a public IP on the pfSense WAN.

[riahc3]

Found the problem:

 

 

Things in red box were checked.

[+BudMan MVC]

Those are dns forwarder settings.. What is pfsense using for dns?

"The DNS forwarder will use the DNS servers entered in System: General setup or those obtained via DHCP or PPP on WAN if the "Allow DNS server list to be overridden by DHCP/PPP on WAN" is checked. If you don't use that option (or if you use a static IP address on WAN), you must manually specify at least one DNS server on the System:General setup page."

And did you validate that your clients are pointing to pfsense lan IP for gateway and dns? Just because you turn of other dhcp server does not mean client instantly renews its lease and gets the new info from different dhcp server. Did you restart the clients or renew their dns via say a ipconfig /renew?

Why would you check subsequently - that option is not on by default, and rarely should be used. And only in specific situations. Generally that is going to slow down dns resolution. Let pfsense query all its dns you have setup and use the fastest response, etc.

edit: That is NOT the problem, you said you disabled your dhcp.. Those checkmarks would be meaningless if the dhcp server is disabled. More like you clients just didn't update their lease to the new dhcp server is more likely.

[riahc3]

Also (just in case):

 

 

 

 

Those are dns forwarder settings.. What is pfsense using for dns?

"The DNS forwarder will use the DNS servers entered in System: General setup or those obtained via DHCP or PPP on WAN if the "Allow DNS server list to be overridden by DHCP/PPP on WAN" is checked. If you don't use that option (or if you use a static IP address on WAN), you must manually specify at least one DNS server on the System:General setup page."

And did you validate that your clients are pointing to pfsense lan IP for gateway and dns? Just because you turn of other dhcp server does not mean client instantly renews its lease and gets the new info from different dhcp server. Did you restart the clients or renew their dns via say a ipconfig /renew?

Why would you check subsequently - that option is not on by default, and rarely should be used. And only in specific situations. Generally that is going to slow down dns resolution. Let pfsense query all its dns you have setup and use the fastest response, etc.

edit: That is NOT the problem, you said you disabled your dhcp.. Those checkmarks would be meaningless if the dhcp server is disabled. More like you clients just didn't update their lease to the new dhcp server is more likely.

I actually renewed the lease several times. It didnt update and it stayed on the old DDWRT IP and it also thought my old DDWRT was the DNS server. I unchecked the ticks in the red box and then it did, like you comment, renew the leases correctly, setting my DNS to to my pfSense box.

[riahc3]

Now, Im trying to to revise setting my (now ex) router and the other AP as stricly only AP and a switch/AP respectively...

[riahc3]

Thats more or less down.

 

Now (well, not now, problably tommorow) Im gonna install WS2012R2. I installed pfSense as a VM version 8 because of the problems of editing it later and such. BudMan advised me to raw map my drives (without virtualizing them) and I noticed that VM version 10 supports native SATA drives. Should I install WS2012R2 as a VM version 10 so this way it can natively support SATA drives and (I imagine) get a better speeds? Or is this irrelevent?

[+BudMan MVC]

I don't believe the VM hardware version has anything to do with SATA and raw mapping..  Its a simple 2 second command line to get the drives raw mapped to your vm, who gives a #### if there is a button to click in a gui.  Its something you do like once ;)

 

I run my machines at version 9, since at 10 you loose the ability to "edit" via the vclient currently.  Have my fingers crossed vmware rethinks that nonsense..  But if they don't might be time to look at other options down the road.

 

To get a machine to 9, just upgrade it goes to 10.  You remove from inventory - edit the xml, I do it via ssh to the esxi box and then bring it back into your inventory.  Its a one time thing, and take all of 30 seconds to do..

 

 

edit: Curious did you use e1000, or vmxnet3 on your pfsense install?  Did you install the tools from the CD or did you go with the openvmtools package?

[+Fahim S. MVC]

Raw device mapping is easy.

 

I use this tutorial:

http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/

[+BudMan MVC]

That is prob the same link I sent him in a PM while he was banned ;)

 

There are plenty of guides on how to do it - it comes down to really 1 command.

 

vmkfstools -z /vmfs/devices/disks/<RAW_Device_Name> </path/where/youwantit>/<RDM>.vmdk

Or maybe it was this guide

http://forza-it.co.uk/esxi-5-1-using-raw-device-mappings-rdm-on-an-hp-microserver/

Or maybe it was this one

http://www.vm-help.com/esx40i/SATA_RDMs.php

There are plenty of them going over the same simple command..