DNS and Secure DNS

OVERKILL

$100 Site Donor 2021
Joined
Apr 28, 2008
Messages
58,094
Location
Ontario, Canada
Most of the prosumer guys on here already know how this works, but for those that don't, figured it might be valuable to have a little post on why DoH (and DoT) is considered an important development and why we are seeing it as an option in browsers, bypassing the system's built-in DNS settings, which can be a pain for network admins who have invested in DNS filtering solutions.

To learn some of the history of how we got here, I recommend these CIRA articles:
Part 1: https://www.cira.ca/en/resources/news/cybersecurity/dns-encryption-evolution-or-revolution/
Part 2: https://www.cira.ca/en/resources/news/cybersecurity/dns-over-https-doh-who-do-you-love/

Under normal circumstances, when you enter a site like "bobistheoilguy.com" in your address bar, a query is sent to whatever DNS server your computer is configured with, which in turn resolves to the IP address hosting that site:
Screen Shot 2023-08-15 at 5.13.38 PM.jpg


This is how the browser knows how to get to that site, by having that domain and/or hostname converted to the ip address. With HTTPS, there is also a certificate confirming that this IP address is indeed authorized for this domain/hostname. Hostnames can be subdomains like fredserver, so you could have the domain "goofygoober.com" and a specific server/hostname that's hosting the subdomain fredserver.goofygoober.com. On internal networks, often you are just using the hostname, like "billyserver", without appending the domain.

Now, DNS queries are sent in plain text, which means if we run a PCAP on the traffic coming from your computer, or from your router, or from your VPN termination point, we can see what queries are being made, and what the responses are. This is what that looks like (pulled from an old PCAP I had kicking around):
Query made: s.mzstatic.com from 192.168.0.27 to OpenDNS server 208.67.222.222:
Screen Shot 2023-08-15 at 5.30.12 PM.jpg


Response received: Standard query response 0x40bf A s.mzstatic.com CNAME s.mzstatic.itunes-apple.com.akadns.net CNAME s-mzstatic-applak.itunes-apple.com.akadns.net CNAME mzstatic.com.edgekey.net CNAME e673.dsce9.akamaiedge.net A 23.15.204.33 from 208.67.222.222 to 192.168.0.27
Screen Shot 2023-08-15 at 5.30.36 PM.jpg


We also see the hand-off between the LAN client (Apple) and the firewall (Cisco Meraki) and vice-versa due to PAT taking place in these exchanges.

So:
- If you are using your ISP's DNS servers, and no VPN, then these queries pass in plain text through all of the network equipment located between you and your ISP's DNS server. This will be a minimum number of hops, often only 1 or 2.

- If you are using a 3rd party DNS provider, and no VPN, then these queries pass in plain text through all of the network equipment located between you and that DNS server. This can be a considerable number of hops.

Here is what that looks like between my computer and CIRA, whose D-Zone DNS filter I use, note that there are only 6 hops:
Screen Shot 2023-08-15 at 6.01.13 PM.jpg


Interestingly, because of the distributed nature, if I run the same query to Cisco's OpenDNS system, it lands at the same point (TIX, but different IP) also only having 6 hops. Cloudflare has 7 hops from my location, also, not too bad.

- If you are using a VPN and their DNS servers, those queries pass in plain text through whatever equipment exists between the VPN endpoint and the DNS server.
- If you are using a VPN and a 3rd party DNS provider, those queries pass in plain text through whatever equipment exists between the VPN endpoint and that DNS server.

This is where DoH (DNS over HTTPS) and DoT (DNS over TLS) come in.

Instead of the DNS queries traversing in plain text, they are encrypted between your computer, and the DNS server!

Sounds great, right?

From a privacy perspective, absolutely. The only time there's a plain text traversal is if the upstream DNS server has to reach-out to the authoritative DNS server because it doesn't have the entry cached. This definitely improves security/privacy from a snooping perspective.

Because most OS's don't support native DoH and/or DoT, the use of this is typically done in an application. You can have a service running on your desktop that runs as a DNS proxy daemon, where the DNS queries get forwarded over DoH/DoT. Both Firefox and Chrome (and chromium-based browsers) support native DoH (example from Chrome):
Screen Shot 2023-08-15 at 6.23.01 PM.jpg


The issue of course is DNS filtering. If you are using OpenDNS, CIRA, Cloudflare or some other DNS protection system and the computers you are supposed to be protecting are using Chrome/Firefox/derivatives and DoH, sending the queries to a 3rd party not configured with the protection settings you have setup, ultimately, your degree of protection; your security, is effectively circumvented. This is now emerging as a major issue, as DoH is being utilized in browsers by default.

On my home network, I have DNS queries (TCP/UDP port 53) blocked with a rule allowing queries to CIRA. I also have DoT blocked, but DoH is harder to pin-down, as it's effectively indistinguishable from HTTPS in most firewalls, so blocking it involves using a list of known DoH DNS servers, blocking those IP's so that the queries never make it past the firewall. My browsers are all configured to use the CIRA DoH DNS server as shown in the Chrome screenshot.

You can also setup a local DNS server using something like a Raspberry Pi, that makes all its queries over DoH, and your DHCP options make that the default DNS server. However, you still have to block DNS on the firewall and will still have to try and lock-down DoH due to the browser-level configurations and the fact that many smart devices are hard-coded to use specific DNS servers, typically google at 8.8.8.8. Both of my smart TV's, a Samsung and a Sony Bravia, try and contact Google's DNS servers, despite being configured to use CIRA.
 
I can't believe I missed this, but this is a great write up @OVERKILL.

My ER-X via DHCP assigns my Windows DC for DNS which then has forwarders setup as my primary and secondary Pi-Hole Ubuntu VMs. For the longest time I used to also run an Unbound recursive DNS server on my Pi-Holes with DNSSEC.

With the upgrade to a newer Ubuntu LTS release, I broke Unbound and abandoned it and have Pi-Hole(s) looking at Cloudflare DNS for the time being.

Long story short, I gotta look back into DNSSEC and make sure I'm comfortable with how I'm using DNS going forward. But this is a great explanation.
 
Also, DNS related... kinda dumb, on Ubiquiti EdgeRouter's, when you're using the router as a DNS server/cache, you have to explicitly set the router to use the system defined DNS servers. Otherwise for it's internal DNS servers, it'll pull and use the WAN DNS via DHCP or if statically set.

I am a big fan of running DNS leak tests to ensure the DNS settings I have in place are functioning as designed.

 
I can't believe I missed this, but this is a great write up @OVERKILL.

My ER-X via DHCP assigns my Windows DC for DNS which then has forwarders setup as my primary and secondary Pi-Hole Ubuntu VMs. For the longest time I used to also run an Unbound recursive DNS server on my Pi-Holes with DNSSEC.

With the upgrade to a newer Ubuntu LTS release, I broke Unbound and abandoned it and have Pi-Hole(s) looking at Cloudflare DNS for the time being.

Long story short, I gotta look back into DNSSEC and make sure I'm comfortable with how I'm using DNS going forward. But this is a great explanation.
Thanks, appreciate the kind words :)

And here I thought you'd be running CentOS or at least Fedora, based on your username ;)
 
Last edited:
Thanks, appreciate the kind words :)

And here I thought you'd be running CentOS or at least Fedora, based on your username ;)
Yeah... Ubuntu is just too easy. I used to manage RHEL boxes about 10 years ago, some brainchild intranet servers. It was wild... company would "demo" or purchase/use commercial software for 6-10 months, find what they liked, then have our developers re-create it/build what they wanted/liked, and we'd migrate off. All hosted on these boxes.

They are no longer in business. It was a unique experience.
 
So do you trust the people who run the DoT and DoH servers you use any more than the people who run the ISP you use?
Personally, I have no issue with my ISP, though I know some members are quite paranoid about theirs. They don't offer DNS filtering, while CIRA does, and since they don't host CIRA, I'd much rather my queries to CIRA be encrypted when possible.
 
Personally, I have no issue with my ISP, though I know some members are quite paranoid about theirs. They don't offer DNS filtering, while CIRA does, and since they don't host CIRA, I'd much rather my queries to CIRA be encrypted when possible.

I'm not sure how much privacy encrypted DNS actually provides. Even if your DNS queries are encrypted, the IP addresses and ports of the TCP connections your browser is making are not. It's not difficult to correlate IP addresses to a website--anyone at your ISP who is motivated enough to snoop on your DNS traffic will be able to do so.
 
I'm not sure how much privacy encrypted DNS actually provides. Even if your DNS queries are encrypted, the IP addresses and ports of the TCP connections your browser is making are not. It's not difficult to correlate IP addresses to a website--anyone at your ISP who is motivated enough to snoop on your DNS traffic will be able to do so.
The same could be said for HTTPS in many respects, when one isn't sending credentials. While the IP addresses of the TCP socket connections are visible to either your ISP or your VPN provider, they don't see the actual domain being accessed, so there is at least a degree of obfuscation going on there. Hosting providers typically have multiple sites on a given IP, so your service provider (ISP or VPN) doesn't know if you were resolving "hornybaldmigets.com" or "bigberthascatwarehouse.com" if they are both bound to the same IP.

Anyway, the point of this thread wasn't as much about the value of encrypting your DNS traffic as much as it was about how browsers leveraging DoH are circumventing DNS filtering where this is being employed. From an admin perspective, this is a pain in the ass.
 
Great information. I’ve always been curious about technology stuff haha it’s all above my head but I think I understand a little bit now between the differences. Love the Goofy Goober website you made up that’s a throwback to SpongeBob for me haha. That’s the first thing that popped into my mind when you said that website 🤣.
 
So, after experimenting with forcing my browsers to use DoH through CIRA for about a month now, I've seen some interesting behaviours.

- Firefox, at random, will stop being able to resolve sites. Turning off DoH and turning it back on immediately fixes the problem. Chrome and Brave both work fine while this is taking place.
- Edge has a somewhat similar problem to Firefox, except it self-resolves. Seems to be when it "sleeps" a tab, it sometimes loses name resolution for that tab when the tab awakens.

So, despite being in production, I'd say this stuff isn't 100% sorted yet.

Since I ended up with a couple of HP EliteDesk 705 G4's (35W) which have 256GB SSD's and 16GB of RAM, I figured why not toss together a proxy box to keep the queries over DoH, but take the task away from the browsers, which seem to be giving mixed results.

So, I slapped Ubuntu server 22.04 LTS on one of them yesterday, installed PiHole and cloudflared and set it up with the CIRA DNS servers. Well, then I decided I'd get creative and put it on a trunk port and set it up with my VLAN's, which was a learning experience doing it with Netplan, as my old RHEL boxes don't use that.

Got that working and things seemed good. I had played briefly with dnscrypt-proxy as an alternative to cloudflared, but I couldn't seem to get the stamps generated correctly for the CIRA DNS servers (they are not in the database), and since cloudflared was working, I said screw it and stopped messing around with it.

Well, around 3:50PM today, name resolution stopped working. Now, I had read some accounts of the cloudflared daemon doing this, so I wasn't caught off guard, but was still a bit disappointed, it hadn't even gone 24 hrs. I checked the status of the service and:
Screen Shot 2023-09-11 at 10.37.34 PM.jpg


Which, seems to be similar to the problem other people have had.

Restarting the service immediately got it working again. Obviously not a viable long-term strategy however.

So, back to round 2 on getting dnscrypt-proxy operating, which, after spending some more time on generating the stamps, managed to get them correct finally, and the daemon working. Which is what I'm using now. I haven't seen any complaints about it doing the same thing as cloudflared, so I'm hopeful that it's stable and reliable.

This little 35W tiny is laughably over-spec'd for the software, which is supposed to run on a Raspberry Pi. Memory use is at 2.6%, load 0.02%. I may delve into tweaking some of the settings later due to this, as I suspect there are some areas with room for improvement with more substantial resources available.
 
I set my DNS to Cloudflare 1.1.1.1 but I also have Eero’s add-on security enabled too - looks like my DNS settings is being overridden to Eero’s DNS servers. I used that DNS site and while it did give me my WAN IP, Eero, not Cloudflare is handing my DNS it seems. I’ll play around with my setup.
 
I set my DNS to Cloudflare 1.1.1.1 but I also have Eero’s add-on security enabled too - looks like my DNS settings is being overridden to Eero’s DNS servers. I used that DNS site and while it did give me my WAN IP, Eero, not Cloudflare is handing my DNS it seems. I’ll play around with my setup.
Yeah I turned off my EERO security and moved to Cloudflare Family DNS (https://blog.cloudflare.com/introducing-1-1-1-1-for-families/) and I run pihole as well. Zero issues.
 
Back
Top