pfSense: Dual WAN Load Balancing / Fail Over DNS (and possibly more) Issues

[The Dark Knight]

Hello! So I have been using pfSense for a few years now and have been very happy with it. I recently signed up for a second internet connection at home to have as a backup. After configuring everything, pfSense does indeed switch over to the second connection when the first one fails (simulated by disconnecting the cable), but DNS resolution (or possibly some other issue) doesn't seem to work, and as a result I am not able to browse the internet. I have provided some more information about my setup below. Provided some screenshots as well.

 

WAN1 is a PPPoE connection with a public routable static IPv4 address.

WAN2 is a CGNAT address and a completely locked down modem, so no Bridge Mode. pfSense just gets a private IP on the WAN2 interface.

 

pfSense running in a VM on Proxmox with 2 Ethernet cards available for it. One used for WAN1, other for LAN.

Managed Switch has a VLAN on one of the ports for the WAN2. VLAN configured in pfSense.

DNS handled by 2 Pi-hole instances, 1 is a VM on the same Proxmox host, 2nd is on a Raspberry Pi. Both Pi-holes are on their own VLAN in pfSense and DNS Redirect has been configured for the primary LAN network.

Both Pi-hole addresses specified under System/General Setup/DNS Settings. AdGuard DNS IP's provided for WAN2.

 

 

What am I doing wrong?

 

 

Edited October 17, 2022 by The Dark Knight

[The Dark Knight]

Need help with this.....bumping topic.

[Mindovermaster Global Moderator]

@BudMan..

[+BudMan MVC]

On 17/10/2022 at 07:31, The Dark Knight said:

Both Pi-hole addresses specified under System/General Setup/DNS Settings. AdGuard DNS IP's provided for WAN2.

So you setup the resolver to forward?  Out of the box unbound resolves, putting stuff in dns settings under general doesn't do anything but provide dns that pfsense itself could use.  Unless you told unbound to forward.  Also trying to redirect dot is pretty pointless.. Since the client should validate the cert of the dns its trying to talk to, trying to send that to your pihole - that shouldn't work, even if pihole was listening for dot.  Unless it presented a cert for whatever fqdn the dot client was trying to talk to.

 

So your clients get handed what for their dns, pfsense IP, and then you redirect that to pihole - that goes where then for dns?

 

Your clients should get handed pihole IPs for their dns, now your pihole should forward to where you want, or it should resolve - and it would use whatever failover you have setup for your routing, etc. In that last rule.

 

If you redirecting it?  You could have a loop in how your dns is suppose to work from that setup.

[The Dark Knight]

Not using DNS Forwarder, but DNS Resolver. Although I just looked at it properly, the option to forward to the DNS servers under General Setup is unticked.....yet everything works (with just one WAN). It's possible I've done something wrong here. I did this a long time back following a guide on using pfSense with Pi-hole. Maybe the guide I followed was wrong as well. But everything was working, so I didn't go deep into it. I do have both Pi-holes specified under the LAN DHCP Server settings as well.

 

On both Pi-holes I have the Cloudflare DoH service running, so my Upstream is 127.0.0.1#5053.

 

Oh ok, didn't know that redirecting DoT is not useful. But then why is the state table for it so large?

 

 

[+BudMan MVC]

Those IPs in general dns have nothing to do with clients asking unbound, as you show unbound is resolving - not forwarding.. It will walk down from roots, then the gltds then the authoritative ns.. Those settings are only for pfsense to forward to if you it isn't asking itself (127.0.0.1) or that is down or something.

 

They would never be used by unbound (unless you set forwarding in unbound), nor would they get handed to a client to use. 

 

Your 2nd wan might be having a problem resolving? Or dns in general.. I would disable your main wan.. And then try doing some directed dns queries via pfsense.. Or a client to different dns, ask pfsense directly on its IP to look up somethiing new that is not in the cache, etc. Does it work, ask 8.8.8.8, ask cloudflare, ask quad9, etc. query the roots directly..

 

example.

 

[22.05-RELEASE][[email protected]]/root: dig

; <<>> DiG 9.16.26 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4687
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.                              IN      NS

;; ANSWER SECTION:
.                       26647   IN      NS      d.root-servers.net.
.                       26647   IN      NS      e.root-servers.net.
.                       26647   IN      NS      f.root-servers.net.
.                       26647   IN      NS      g.root-servers.net.
.                       26647   IN      NS      h.root-servers.net.
.                       26647   IN      NS      i.root-servers.net.
.                       26647   IN      NS      j.root-servers.net.
.                       26647   IN      NS      k.root-servers.net.
.                       26647   IN      NS      l.root-servers.net.
.                       26647   IN      NS      m.root-servers.net.
.                       26647   IN      NS      a.root-servers.net.
.                       26647   IN      NS      b.root-servers.net.
.                       26647   IN      NS      c.root-servers.net.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Oct 18 11:03:32 CDT 2022
;; MSG SIZE  rcvd: 239

[22.05-RELEASE][[email protected]]/root: dig @d.root-servers.net com. NS

; <<>> DiG 9.16.26 <<>> @d.root-servers.net com. NS
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39862
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1450
;; QUESTION SECTION:
;com.                           IN      NS

;; AUTHORITY SECTION:
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.

;; ADDITIONAL SECTION:
a.gtld-servers.net.     172800  IN      A       192.5.6.30
b.gtld-servers.net.     172800  IN      A       192.33.14.30
c.gtld-servers.net.     172800  IN      A       192.26.92.30
d.gtld-servers.net.     172800  IN      A       192.31.80.30
e.gtld-servers.net.     172800  IN      A       192.12.94.30
f.gtld-servers.net.     172800  IN      A       192.35.51.30
g.gtld-servers.net.     172800  IN      A       192.42.93.30
h.gtld-servers.net.     172800  IN      A       192.54.112.30
i.gtld-servers.net.     172800  IN      A       192.43.172.30
j.gtld-servers.net.     172800  IN      A       192.48.79.30
k.gtld-servers.net.     172800  IN      A       192.52.178.30
l.gtld-servers.net.     172800  IN      A       192.41.162.30
m.gtld-servers.net.     172800  IN      A       192.55.83.30
a.gtld-servers.net.     172800  IN      AAAA    2001:503:a83e::2:30
b.gtld-servers.net.     172800  IN      AAAA    2001:503:231d::2:30
c.gtld-servers.net.     172800  IN      AAAA    2001:503:83eb::30
d.gtld-servers.net.     172800  IN      AAAA    2001:500:856e::30
e.gtld-servers.net.     172800  IN      AAAA    2001:502:1ca1::30
f.gtld-servers.net.     172800  IN      AAAA    2001:503:d414::30
g.gtld-servers.net.     172800  IN      AAAA    2001:503:eea3::30
h.gtld-servers.net.     172800  IN      AAAA    2001:502:8cc::30
i.gtld-servers.net.     172800  IN      AAAA    2001:503:39c1::30
j.gtld-servers.net.     172800  IN      AAAA    2001:502:7094::30
k.gtld-servers.net.     172800  IN      AAAA    2001:503:d2d::30
l.gtld-servers.net.     172800  IN      AAAA    2001:500:d937::30
m.gtld-servers.net.     172800  IN      AAAA    2001:501:b1f9::30

;; Query time: 223 msec
;; SERVER: 199.7.91.13#53(199.7.91.13)
;; WHEN: Tue Oct 18 11:03:56 CDT 2022
;; MSG SIZE  rcvd: 828

[22.05-RELEASE][[email protected]]/root: 

 

[The Dark Knight]

Ok, so I disabled WAN1 and did a Dig test like you asked.....it fails! So DNS is the issue.....

 

 

[The Dark Knight]

Also, another thing I can't figure out why.....if I have the Fail Over group specified in the Gateway for the LAN outbound rule, even though I am able to browse the web, I can't access my Pi-hole URL's and also can't access my WAN2 modem web page! I have another VLAN with a .20 network, I can access the web servers and services on that perfectly.

 

If I change the Gateway in the outbound rule back to default, everything is accessible again.

 

Edit: I just checked, I actually can't access everything. Some work, some don't. 😵‍😵‍

 

Edit 2: If I try to access directly with the IP on the .20 network, it doesn't work. Only accessible with the domain that I have reverse proxied for those web servers. This is all when the Fail Over is specified on the LAN Outbound.

Edited October 18, 2022 by The Dark Knight

[+BudMan MVC]

did you do that from pfsense?  That is just asking loopback, itself you didn't direct that to 8.8.8.8 you asked 127 for 8.8.8.8 you need to use the @ to tell dig where to ask.

[The Dark Knight]

On 18/10/2022 at 22:18, BudMan said:

did you do that from pfsense?  That is just asking loopback, itself you didn't direct that to 8.8.8.8 you asked 127 for 8.8.8.8 you need to use the @ to tell dig where to ask.

Yes, logged in via SSH to pfSense and ran the Dig command.

 

Edit: Logged in again after disabling WAN1 and ran dig @8.8.8.8. No response.

[+BudMan MVC]

well then if you can not talk to 8.8.8.8 from pfsense, or the roots can you talk to the dns of of that connection.  It is possible for ISP to only allow dns to their dns servers.  You might see that on say a sat connection, or even cell - did you mention this was a cell sort of backup link?

[The Dark Knight]

I actually suspected this ISP must be forcing their DNS....but I tried just now, doesn't work either. No, this is an optical fibre connection. But locked down like hell! I had another thought.....this ISP prioritises IPv6. Could it be possible I'm facing problems because I'm trying only IPv4 in pfSense?

 

 

[The Dark Knight]

Sorry, late over here, need to go to sleep. 😄

 

Thanks for all your help so far BudMan! 😎👍

 

Want to really get this working....otherwise worst case scenario, I dump this ISP and get another connection from a not so restricted provider. I signed up with these guys because they bundle in streaming subscriptions with the internet plan, and is a damn good deal.....provided you use everything they way they want you to. Their supplied Android TV box for my TV....freaking HELL....insanely locked down!

 

[+BudMan MVC]

that you get a timeout is odd, I actually get a refused which is more like what should happen when you try and talk to a ISP dns, but not from ISP network.

 

22.05-RELEASE][[email protected]]/root: dig @49.45.0.3

; <<>> DiG 9.16.26 <<>> @49.45.0.3
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 51706
;; flags: qr rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; Query time: 266 msec
;; SERVER: 49.45.0.3#53(49.45.0.3)
;; WHEN: Tue Oct 18 13:04:22 CDT 2022
;; MSG SIZE  rcvd: 12

[22.05-RELEASE][[email protected]]/root: dig @49.45.0.3 www.google.com

; <<>> DiG 9.16.26 <<>> @49.45.0.3 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 26050
;; flags: qr rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; Query time: 265 msec
;; SERVER: 49.45.0.3#53(49.45.0.3)
;; WHEN: Tue Oct 18 13:05:18 CDT 2022
;; MSG SIZE  rcvd: 12

[22.05-RELEASE][[email protected]]/root: 

 

But yeah if you can not do dns over this connection - what is the point of it??  It useless connection.

[The Dark Knight]

On 18/10/2022 at 23:36, BudMan said:

that you get a timeout is odd, I actually get a refused which is more like what should happen when you try and talk to a ISP dns, but not from ISP network.

 

But yeah if you can not do dns over this connection - what is the point of it??  It useless connection.

Ok. Yes, true, it is useless in pfSense. I can still use the connection with just their device directly via Ethernet or WiFi. Not ideal of course.

 

But what I don't understand is.....however locked down it is, since pfSense is just seen as a DHCP client by their device, shouldn't it just work?

My computers do, if I connect them to ther device. 🫤

[+BudMan MVC]

Dns should work - kind of hard to use the internet if dns doesn't work..  When you plug your device into their device - what does it give you for dns, the IP of their device? You can set pfsense to use that for dns.

[The Dark Knight]

On 20/10/2022 at 04:52, BudMan said:

Dns should work - kind of hard to use the internet if dns doesn't work..  When you plug your device into their device - what does it give you for dns, the IP of their device? You can set pfsense to use that for dns.

Ok, so here's something important. There is definitely something in my configuration that is blocking pfSense from getting DNS resolution on WAN2. I created a brand new VM and setup a fresh install of pfSense. I then configured only the second connection as my primary WAN. pfSense is pulling a private IP from their device, and everything is working!

 

So it definitely works, just not in my current setup. I'm sure it has something to do with the way I am using Pi-hole for DNS resolution and also probably because of my redirect rule so that clients are forced to use my Pi-holes.

[+BudMan MVC]

your redirect would have nothing to do with a directed query on pfsense, which is why I asked you to do it from pfsense, etc.

[The Dark Knight]

Ok. But then that still leaves the question....why isn't it working?

It's not an absolute deal breaker if it doesn't, but would be nice to be able to take advantage of automatic dual WAN functionality.

[+BudMan MVC]

so when your connected directly to this isp device, what does your client use for dns?

 

Your saying when you redid your pfsense its now all working.. So your saying the directed queries you did before are now working?

[The Dark Knight]

DNS IP is showing their router IP - 192.168.29.1

 

Yes, I tested dig @8.8.8.8 and dig @d.root-servers.net com. NS like in your screenshots, works perfectly.

 

 

[+BudMan MVC]

Well add your other line now..  But if dns isn't working, then yeah hard to get anywhere even if you have connectivity.

[binaryzero]

What's the gateway set to on your pi-hole, does it have internet connectivity when you failover to your secondary connection? If you've set your DNS forwarders to use pi-hole, and it can't get out..well...rip.

[The Dark Knight]

On 21/10/2022 at 07:49, BudMan said:

Well add your other line now..  But if dns isn't working, then yeah hard to get anywhere even if you have connectivity.

Yes, will try adding it soon. Not at home currently, so will do it tonight and test. I think I also enabled the option to let DNS be overriden by what the WAN interface gives. Not sure, can only check later today.

 

But if I do have to keep that enabled for WAN2 to work, pfSense will do that for my primary connection as well. Can I still have my Pi-hole handle DNS instead of the WAN1 DNS (they use Google DNS)?

 

 

On 21/10/2022 at 08:07, binaryzero said:

What's the gateway set to on your pi-hole, does it have internet connectivity when you failover to your secondary connection? If you've set your DNS forwarders to use pi-hole, and it can't get out..well...rip.

When I simulate WAN1 being down, WAN2 has no DNS resolution and my Pi-holes are not accessible.

 

My Pi-holes are on a seperate VLAN, and no Gateway was specified under the Interface settings on pfSense. On both Pi-holes, Upstream is Cloudflare DoH using the Cloudflared service.

[binaryzero]

What? The Pi-Holes will need a gateway to communicate to the internet...

 

If you disconnect WAN1 and DNS stops, its probably because the pi-holes have lost internet connection?

 

You can specify whatever you want as DNS servers on your LAN, use DHCP to hand out the pi-hole IPs for DNS.