How to tap into network debugging

I know there’s no connection between my local area network and our house’s plumbing system, but when the faucet started running uncontrollably literally seconds after I fixed a network outage, I had to wonder: did we have a poltergeist? The jury is still out. But I can tell you that my hour of fruitless examination of router settings is to your benefit: you won’t have to do this when it happens to you. The weakest TP-Link As a guy known for Wi-Fi since the early 2000s, you might think my house is ablaze with wireless signals.1 Yes, it is: we have six Wi-Fi access points and gateways in a relatively small house by modern standards (about 1,200 sq. ft. on the main floor and 800 in the finished basement). But we also have Ethernet strung through the house to connect most of them, with powerline networking adapters where that doesn’t work due to the ancient and mysterious innards of the walls. The innards’ outsides are covered with heavy, old plaster, which absorbs some of the power of the Wi-Fi signals—hence the large number of emitters. Normally, this works fine. We have four TP-Link routers, some purchased a while back, and a Netgear Nighthawk as our primary base station, which distributes network addresses and is connected to the fiber-optic ISP router.2 When something goes wrong, it’s usually an issue with a router crashing, even though its various green lights remain in a happy state. This time, something was more deeply wrong. The TP-Link routers showed orange Internet connection lights, which in this case meant they were connected via LAN to the Netgear, but they couldn’t reach the Internet. Perversely, I seemed to be able to connect from my iPhone and my office Mac, which has both Ethernet and Wi-Fi connections active.3 TP-Link has one of the most irritating admin approaches for its routers of nearly any networked product I’ve ever used. You can connect directly, but it’s difficult to store the device’s password because it uses a potentially changing local IP address on your LAN. There’s also a centralized system where you can use an app called Tether or a TP-Link website to log in remotely to a router, even when you’re on the same LAN, rather than finding its local address. Tether also scans the local network, letting you connect even when Internet access is disrupted. I don’t think this screenshot can convey how deeply irritating this interface is when you’re trying to solve a problem. Without the routers “seeing” the Internet, I couldn’t use the website, and Tether worked erratically. I eventually was able to figure out the IP addresses of two of the four routers, connect via a web browser, and find a working password for one of them. (To reset the password, you have to factory reset the router. That’s for a future day.) But I remained baffled. My ISP gateway was up. My Mac could reach the Internet. The Netgear router seemed to be routing my Mac’s traffic to the Internet. Some of the devices on our network were also connected to the rest of the world. Had my TP-Link routers been compromised? Was something more sophisticated going on? Nope. It’s coming from inside your smart house Some of you may be yelling at me from the safe distance of your phone or computer. “Glenn, Glenn, why didn’t you check DNS?” The circulatory system of the Internet may be the IP-addressed traffic that passes over the series of tubes that make up the network. But DNS is the helpful labeling system, like an editorial cartoon that uses captions like “taxes” and “fat cats” to mark characters in an illustration. DNS turns human-readable domain names into the underlying numbers required for connections. The NTP settings were a clue: they hadn’t updated. Here, I set them to an IP address, and the current time was immediately retrieved. Surely DNS was working? My iPhone, Mac, and other devices were resolving addresses, or turning them from names into numbers. Then I found a clue in Network Time Protocol (NTP), a somewhat obscure bit of plumbing that also makes things tick. When I went to Advanced: Administration: NTP Settings on the Netgear, the last time it had been able to grab the current time and date was hours earlier. That seemed problematic. Digging more deeply, literally, I found that the dig Unix command couldn’t pull up IP addresses from the Netgear router. On a LAN in which you’re using DHCP to handle local address assignment, your router may require that you enter Internet IP addresses for one or two DNS servers, or it might retrieve them automatically from your ISP. When a device on your network requests an address, the DHCP server assigns an address and tells the device to use the router address for DNS resolution. The router relays your requests to the real DNS server. I tried dig @10.0.0.1 apple.com (use the router to return the IP address for apple.com) via Terminal on my Mac, and it timed out. I then tried the same lookup using my ISP’s DNS servers. Same timeout. The Mac had no problem with domain lookups, however, nor did my iPhone. Your router provides its own address as the DNS server in most DHCP configurations. Now comes the time to insert a head-slapping emoji. Some time ago, and I can’t recall how long ago, I set my Netgear router to use my ISP’s primary and secondary DNS addresses. I know many people have switched to Google, Quad9, Cloudflare, or other free, public DNS servers to improve lookup speeds, which, at one point, would have helped speed up network operations tremendously because ISPs invested so little in their DNS operations. However, using public DNS can cause issues in particular cases. For one, it can confuse content delivery networks (CDNs), which try to guess the closest server to you topologically based on your DNS lookup request to their servers. When that works, you’re getting the shortest path and likely faster throughput. Public DNS can misdirect CDNs, resulting in severe slowdowns in data transfers for big files and with streaming. Much of that has been resolved, but not all of it. I had had some issues, so I reverted from one of the public DNS services to my ISP’s servers. (This has largely been solved now.) So what happened? My ISP, CenturyLink, must have had a DNS failure. I couldn’t get replies from its DNS servers when I checked (directly or via my router). I found postings later from other CenturyLink customers around the same time, but no public message from the company. Why did my Mac and iPhone keep resolving? My Mac’s Ethernet connection had a manually set DNS server that was an alternate one provided by my ISP, one that apparently continued to work. I can’t explain the iPhone situation, unless it’s possible that iOS will pull DNS over a cellular connection when it fails over Wi-Fi—I don’t know that it does that, but I don’t have another explanation. I removed my ISP’s DNS values, replaced them with 1.1.1.1 (Cloudflare) and 8.8.8.8 (Google), clicked Apply, and an hour of my time suddenly was marked on my internal spreadsheet as “wasted.” Where I went wrong I’m always trying to learn from my mistakes, particularly because I turn mistakes into opportunities to write columns like this one to save time for you, dear reader. In this case, I was led astray by the half-working network and the fact that, seemingly, only one brand of router in one configuration was failing. I should have remembered to start with DNS—and I will in the future. If you’ve read this long, I’m sure you’re dying to know about the faucet. I had just finished fixing the network, walked to the freezer, grabbed a fruit-only frozen treat, and was walking across the kitchen when a small stream of water came out of the faucet. I tested the taps. I put the frozen delight back into the freezer, turned off the taps while my spouse was on speaker so we could figure out which tap’s cartridge decided at that moment to give up the ghost, and ordered a replacement. This was another kind of DNS failure: do not spout! For further reading While this column may have given you less confidence in me than you had before you read it, I share much more information about Wi-Fi, networking, and troubleshooting in my book Take Control of Wi-Fi Networking and Security. [Got a question for the column? You can email glenn@sixcolors.com or use /glenn in our subscriber-only Discord community.] [Glenn Fleishman is a printing and comics historian, Jeopardy champion, and serial Kickstarterer. His current books in preparation, which you can pre-order, are Flong Time, No See, and That One Matt Bors Comic. Other books include Six Centuries of Type & Printing and How Comics Are Made.] If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.