What a home lab taught me about DNS

The most reliable way to learn what DNS does is to break it and watch everything else stop working.

That is what happened in my home lab. Pi-hole runs on a Raspberry Pi 3 and handles DNS filtering for the house. One evening I made a small change to its upstream configuration, restarted the service, and then spent twenty minutes convinced the internet was down.

It was not. DNS was down.

The setup#

The lab is deliberately modest:

• A Raspberry Pi 3 running Pi-hole and a local resolver.
• A Raspberry Pi 4 running containers for services I want to reach from anywhere.
• Tailscale providing private access to those services without opening ports.
• The router handing out the Pi's address as the DNS server.

This is a common setup, which is why the failure was also common. The router pointed every device at a DNS server I had just broken. No device could resolve a name, so nothing worked, including the tools I would normally use to diagnose the problem.

What the failure taught me#

Three things.

First, DNS is infrastructure, and infrastructure needs a fallback path. If the resolver is the only one the network knows about, then a resolver problem becomes a network problem. Devices with a cached answer keep working for a while, which makes the outage look intermittent and confusing.

Second, I had made a change without a way to undo it from outside the thing I was changing. The Pi has no screen, no keyboard, and no serial console. If I could not SSH in, my only option was to pull the power and hope.

Third, "it's always DNS" is a joke until it is your evening.

What changed#

The setup is now less clever and more recoverable:

• The router hands out a secondary DNS server, so a broken Pi-hole does not take the whole house offline.
• Configuration changes are made in files that live in a backup, not only on the Pi.
• Before restarting a core service, I confirm I have a working path back in.
• The Pi 4 runs a second copy of the resolver for failover.

None of this is sophisticated. That is the point. A home lab is a good place to learn that reliability is mostly a series of boring decisions made before something breaks.

The transferable part#

The same pattern shows up in professional work. A system that has no fallback path turns every small change into a risk. A system with a fallback path, a backup, and a documented way back is one where changes are routine.

The difference is rarely technical sophistication. It is whether someone thought about the failure before it happened.

I still run Pi-hole. I just stopped treating it as something that cannot fail.