pfSense: Dual WAN Load Balancing / Fail Over DNS (and possibly more) Issues

[The Dark Knight]

I meant I didn't manually specify a Gateway. By default, pfSense is the Gateway for them (and all other VLAN's). Pi-holes on a seperate VLAN and 192.168.5.5 and 192.168.5.6 addresses. Gateway is automatically 192.168.5.1.

 

The thing is, if I disconnect WAN1, DNS stops, but I also can't access my Pi-holes locally.

 

Yes, I have the Pi-hole IP's specified under my primary LAN DHCP Server settings.

[+BudMan MVC]

On 21/10/2022 at 01:36, The Dark Knight said:

let DNS be overriden by what the WAN interface gives. Not sure, can only check later today.

That is only for pfsense.. That is not something a client would use..

 

Settings in pfsense dns are only for pfsense to use, unless you setup forwarding in unbound. Normally setup would be only 127.0.0.1 for dns in pfsense - its basically asking itself (unbound) and then unbound would resolve.

 

 

When you have an interface go down, unbound might have a hard time knowing since it was bound to that address.. What you might want to do, is just bind unbound to the local host for outbound connection.  This way when it wants to resolve something it will use the default routing in pfsense.

 

[The Dark Knight]

On 21/10/2022 at 17:49, BudMan said:

That is only for pfsense.. That is not something a client would use..

 

Settings in pfsense dns are only for pfsense to use, unless you setup forwarding in unbound. Normally setup would be only 127.0.0.1 for dns in pfsense - its basically asking itself (unbound) and then unbound would resolve.

 

When you have an interface go down, unbound might have a hard time knowing since it was bound to that address.. What you might want to do, is just bind unbound to the local host for outbound connection.  This way when it wants to resolve something it will use the default routing in pfsense.

Oh ok, didn't know that.

 

Yes! I think this along with turning on the "Allow DNS to be overridden" setting has done the trick (partly). The second ISP just doesn't work without the DNS override option being on, meaning they are forcing their DNS.

 

Anyway however this is what is happening now......

 

Failover works. When WAN1 goes down, WAN2 takes over. But when WAN1 comes back online, it is a big task. Killing states, reloading Gateways, toggling Windows network adapter on and off.....nothing works. I have to reboot pfSense for WAN1 to start working properly again. Point to note is, this is only if I specify my Pi-holes as DNS in the LAN DHCP Server settings page. If I only have the "Allow DNS to be overridden" and leave the DNS server fields in the LAN DHCP Server empty, the internet works perfectly and Failover also switches back and forth easily. But then with this way, my Pi-holes are not used. This is despite having DNS Redirect configured.

 

I've also noticed this blocked traffic in the firewall log. The source IP is my WAN1 public address.

 

 

[+BudMan MVC]

Those are all out of state blocks. Yeah if you kill states any non syn traffic would be blocked if there is no state.  Most of those are Fin,ack (close this connection)

 

If you want for your clients to use pihole, you should set your clients to use pihole vs trying to redirect them too it.  Then have pihole forward to pfsense.

[The Dark Knight]

On 22/10/2022 at 17:11, BudMan said:

Those are all out of state blocks. Yeah if you kill states any non syn traffic would be blocked if there is no state.  Most of those are Fin,ack (close this connection)

 

If you want for your clients to use pihole, you should set your clients to use pihole vs trying to redirect them too it.  Then have pihole forward to pfsense.

 

Ok, so I do have static leases for all my devices and can easily specify Pi-hole for them. And that'll work perfectly as long as WAN1 is up. But when the WAN switches, won't I face the same problem like I am currently?

 

Unless it won't cause a problem if I do what you suggested, point Pi-hole to pfSense? But then how do I implement DoH? I need DoH as ISP's here censor some content because of Government orders.

[+BudMan MVC]

You mean dot, unbound can do dot..  pihole is forwarding to unbound, unbound looks up what pihole asked for. Doh is a browser thing.

 

Again if your using localhost for outbound, you shouldn't have any issues with unbound using whatever the default route is.

 

So your saying your isp is intercepting your dns, and if you try and go to xyz.com - they don't resolve it?  Pretty lame blocking if you ask me..  That might stop billy's grandma or something.

[The Dark Knight]

Oh yes, my mistake. Got confused between DoT and DoH.

 

Oh ok, great! Will try this out then. So basically I have to disable the Cloudflared service on my Pi-hole and enter pfSense as Upstream right? But do I enter 192.168.5.1 (Pi-hole VLAN) or 192.168.10.1 (Primary LAN)? LAN network is on the default untagged VLAN 1.

 

Oh yeah, DNS blocking is lame as hell and stupidly simple to bypass! And since the Government order is literally a long list of specific URL's, sometimes even DoH or DoT isn't needed, just have to change the URL to HTTPS! 🤣

 

Although interestingly, the ISP for my second connection....I have heard rumours that they do Deep Packet Inspection. Have also heard they block some VPN providers. I haven't tried, so can't say for sure. And for VPN's I have my own OpenVPN and WireGuard servers setup in multiple locations and those work perfectly.

[The Dark Knight]

So I decided to just go back to a single WAN setup in pfSense. The second connection is a nuisance to setup and use due to their damn restrictions!

But anyways, thanks a lot BudMan, learnt new things as always. 😎👍

[binaryzero]

Sounds like it was more your configuration than a provider thing, you did mention you followed some rando guide...

You also confirmed it works as expected with a new instance.

[The Dark Knight]

On 25/10/2022 at 09:00, binaryzero said:

Sounds like it was more your configuration than a provider thing, you did mention you followed some rando guide...

You also confirmed it works as expected with a new instance.

Yes, very much possible. However even in the fresh setup it wasn't all hunky dory. Was still causing issues. This ISP is heavy handed, no question. For one, they are forcing their own custom DNS, which I don't like. My primary ISP has no such requirements. It defaults to Google DNS, but works with any other just fine.

[+BudMan MVC]

While I don't agree with limiting users to the ISP dns, it can be common.. Think about it from a bandwidth point of view.

Lets say you have 1 million users..  If all of those users were asking say googledns for their dns - that is a lot of traffic off the network.  Sure dns queries are tiny, but if you have enough of them they can add up to some non insignificant amount of data that has to be moved.  Now with 1 million users you also have a big cache for your dns.. Because its quite likely with that many users some user has already asked for www.domainX.com, so the next user wanting to go there, the entry is cached already..

Now multiple this by how many phones a cell company might have, and these little 4G/LTE/5G router things are just to them another phone user..   it can also be helpful in higher latency connections..  DNS only going to wait so long for a response.. I think most clients max is like 2 seconds, which seems like a long time.  But If the connection is real high latency this could cause problems for a user, if the response is not fast enough, but if the item is cached on the isp dns - response should be faster and inside the timeout for dns.

Applications normally are more forgiving for higher latency on getting an answer from say a website or something.. And the isp might even be caching some of that stuff with their own proxies, etc.

if the connection your trying to use is for backup purposes only, then you might be able to live with some of these limitations when your in a scenario that your main isp is down, but at least you have internet even if you have to live with their limitations of connection abilities.

But if you want to load share across these connections, or expect to have the same full functionality as you other main ISP, these cell sort of routers and cell data connections are going to be more limited quite often.

[The Dark Knight]

Hmm, I didn't think of it from this angle about DNS servers getting overloaded. But I think this ISP specifically is forcing their DNS to one, comply more easily with Government orders and two, to mine data about users. This is based on what I've read online. Granted, it could very well be untrue and just scare mongering by people and other companies, but it could be true as well.

Actually this is an optical fiber based connection, not mobile network tower based.

Yes, this connection is just a backup. I mainly got it for my TV because they bundle multiple streaming subscriptions. And since I use pfSense, thought I would fiddle with Dual WAN. Oh well, just have to do things the "old fashioned way" by switching devices over to this connection manually! 😄

[+BudMan MVC]

Oh not sure why I thought it was wireless - maybe another thread somewhere else.  Sorry about that - but yeah could be many reasons an ISP wants their users to use their dns.  I think its fine if they offer that for their users, but forcing it I am against. Control and information could for sure be a major factor in why they are doing it.  But than again from a technical standpoint there are legit reasons why they would want their users to use their dns as well. 

To be honest from the technical point of view - most of their users are just going to use what gets handed to them.  So allowing others to not use it shouldn't be all that much extra bandwidth.  And if really a latency issue then users attempting to use others, would soon end up back using the isp dns.

Here is the thing - the only thing users have to fight such shenanigans is their $, if a isp doesn't provide the services you want - then find a different one that does.  The problem is many locations are very limited to what isp they can even get.. 

If your having issues with dns through this connection, if you want to use it have some devices use connection X vs Y.. That is a simple policy route in pfsense.  And as long as pfsense is getting dns from X doesn't matter what Y does with dns.. But where you could have problems is when X goes down, and you have no way to get the dns you want, etc.

 

 

 

[The Dark Knight]

No worries! I'm pretty confident that control of information is the reason they are forcing their DNS. For instance, they outright also block some VPN connections! Even the Android box they provide...bloody HELL...crazy locks on it!

True. Most users either don't know or don't care, as long as the internet is accessible. Many don't even keep an eye on speeds to see if they are getting what they are paying for. Average Joe will have one TV, one laptop / desktop and one phone and using the ISP provided hardware. They wouldn't even bother using another off-the-shelf router, let alone setup pfSense. 😂

Yes, true. There are actually 3 high speed ISP's in my area. I tried one of the other two providers for my second connection recently....beyond pathetic service! So this one was the only remaining option.

Yeah, DNS is the main problem with this ISP. It's ok anyway, I will just use pfSense with one WAN. My primary connection rarely goes down. And when it does, it is usually fixed within a couple of hours. Perhaps if in the future another ISP starts service here, I can try Dual WAN again. Or maybe take a 3rd connection! 🤣