Damn Beautiful Nameservers

May 10, 2024 Update and SPOILER ALERT (jump ahead if you want to read through the post first)

I incorrectly attributed my success to DNS cache. But it wasn't DNS cache that saved my butt. It was nameservers (and I got lucky with timing.) 

The information I was searching for was available not because of cache, but because GoDaddy's nameservers hadn't yet received the updated instructions to remove the DNS records.

Looking at some of GoDaddy's documents for removing domain entries, they do list out recovery times if something was done in error, but that is likely some other process. I also asked someone at GoDaddy how the UI and DNS were connected, and they stated that although UI changes are typically updated on the nameservers quickly (within an hour), it can take longer. 

THIS is why I was able to look up values well beyond the often generic TTL of 1 hour and the records I needed were therefore published long enough for me to go back and look them up almost 2 days after 'the incident.' 

So what was I thinking? I don't know. Obviously while troubleshooting I knew I was looking for nameservers, but I jumbled it up when I was recounting my story, thinking the nameserver was wiped immediately on GoDaddy's side and the reason the data was still there was because they cached it. It's clear, my brain wires must have gotten disconnected. Heavens! ISKTBN! 

So, sorry cache, you ain't the hero this time. Maybe next time. All references to cache (except one) have been updated and it's awesomeness is now attributed to the true hero. The rest of the story remains the same.

And look at that, 2 ISKTBN stories in one!

Special thanks to the Anonymous commenter for pointing this out!

P.S. DNS IS COMPLEX! 

______________________________________

Today on "Things I learned", I find out nameservers when they want to, will 'effing save your DNS.

This is one of those stories that others might be like, "Well, duh, that's obvious." Or not. I don't know, but in the heat of the moment, I got a thrill from the fact that I thought of it AND I thought of it quickly. It wasn't a "go sleep on it" or "need to go vacuum and think" or "go on a revelation walk." It was a brain slap where suddenly I had an urge to pretend I was a professional and use my knowledge of DNS things.

And so begins my story about nameservers and why they're so damn beautiful...at least in this story.

Wednesdays are supposed to be joyous. You're halfway through the week and the weekend is in sight. At day end, you know you only have 2 days left before rest. If you're lucky, you can use the end of the week to catch up on stuff.

But a recent Wednesday for one of my clients turned out to be panic-inducing with an urgent email going out to me at 9:29PM.

--begin panicked paraphrased email copy---

"I think I royally messed something up a few hours ago. I plugged in Cloudflare (for reasons) so I had to update the nameservers on my main domain to ones Cloudflare recommended... When I did that, all of my DNS records from GoDaddy got moved to Cloudflare. I think our evening send went mostly to people's spam since DMARC and everything moved...

Are you available at all tomorrow to help me solve this? This is a code red..."

--end panicked paraphrased email copy---

I reached out at the butt-crack of dawn when I woke up to get a call scheduled for when we both had functioning brains.

When we got on the call, the first thing my client said was, "I've ruined everything and I've tanked my business." To which I replied, "No, we'll figure this out. Take me through what you did."

So they did and to protect our conversation, all I can share was they decided to take advantage of one of Cloudflare's feature to help identify and block bots. To make this work, they had to point their nameservers to Cloudflare. The instructions given (according to my client) were to just update the NS records. They did so and 'TADA!' DNS records were pulled over! Their last provider, GoDaddy, no longer had the records listed in the account so it all must have gone smoothly...

...at least until they launched their evening campaign.

No client is immune from spam folder placement, but when you are typically seeing 30% opens and then suddenly tank to 1% and emails to your own account are going to spam, something is clearly wrong.

Sometimes clients are so focused on their email they don't realize a change happening outside of email could impact their work (Damn CNAME.) The positive here is that my client was hyperaware of all things about their program, so they knew that their recent DNS change was the issue and we were able to focus on it straight away.

After they recounted what happened, I asked them to pull up their DNS and send me a test. Once I got the test, I checked the headers, knowing full well authentication was busted. What I was looking for was the detail around the authentication. 

• Were they using their own domain for SPF? 
• What domain were they using in DKIM and what was the selector?

Once I had this, we went back to Cloudflare's UI to see if the essentials were in the DNS records.

And, as you may have guessed already, the transfer of the DNS wasn't complete. Only some of the records were brought over.

--so begins another paraphrased conversation - apparently I like to recount conversations in my stories---

"Do you have a document containing of all your DNS records? If so, pull it up and we can just add in what's missing for now then we can file a ticket with GoDaddy and see if they can pull up things on their end to make sure we got everything and that we don't have duplicate records floating around." (FYI, GoDaddy couldn't pull up anything.)

"NO!"

"Ok."

Brain begins to furiously think...

We need the DNS records.

Where can I get them if they aren't in Cloudflare and GoDaddy's UI is empty?

"Can you go back into your ESP and see if the records on their setup pages are still there for you to copy and paste?"

"Nothing."

As an observer, you could feel the stress through the screen.

💡 idea has struck. I think I know where I can find them! I'm getting giddy, but I don't dare say anything because I could be wrong.

"Give me one sec."

--end another paraphrased conversation---

I begin to query the DNS (EEEEE the excitement!). BUT not via a standard look up on the domain - that pulls up no results or what is hosted on Cloudflare. 

Example - not real life results: 

~ % host -t txt selector._domainkey.isktbn.com

Host selector._domainkey.isktbn.com not found: 3(NXDOMAIN) 

I start looking on GoDaddy's nameservers because they may still have the information! And that same good friend that told me I need to write a blog showed me how to do this within the Mac terminal. Before I plugged it in, I did have to look up GoDaddy's nameservers (one of which was ns45.domaincontrol.com) because I don't have that information stored in my brain (just not enough room.)

Example - not real life results: 

~ % host -t txt selector._domainkey.isktbn.com ns45.domaincontrol.com

Using domain server:

Name: ns45.domaincontrol.com

Address: 97.74.102.23#53

Aliases: 

isktbn.com descriptive text "k=rsa; t=s; p=MIIBIjANBgkqhkfweafd87zvjdf87"

And blamo, RECORDS. I specifically looked up the SPF and DKIM and then sent them over. In the real-life scenario, we had to enter two CNAMEs and not TXT records like my mocked up example shows.

We then sent tests. They worked beautifully for me, but my client still had issues (<--THIS is cache.) I let them know we need to give it a little time to make sure everything updates correctly.

Time goes by and more tests were sent to myself and others and the panic started to subside because they were now passing.

My next task for my client was to see if GoDaddy was able to recover anything or if their account stored their records for a period of time. I could find the DNS records needed based on headers, but if they had ANY other records added for other services, I wouldn't know or be able or look them up without knowing the host.

Then we ended the call. Me feeling elated and a bit full of myself.

And then, of course, reality has a way of laughing at you and grounding you when you need it.

NEXT DAY

"The send also went to spam for everyone 😭 ... and the DKIM showed "fail" when I hit 'Show Original'"

What in the?!

We hop on another call. Turns out they had many publications in their ESP, which meant they had many accounts, each with their own authentication settings.

Ok, we can fix that, but DAMN I should have asked if they had anything else mailing and I should have learned from my past experience to test everything (aka all publications, had I known/asked if they had more and if they used the same domain/IP/configuration.)

So we fixed it. Then they monitored their next send and thankfully, things normalized. There was a little extra filtering than normal, but if it continued, then ticket time!

So what did I learn?

1. Always ask what other streams or domains are in use and if there is anything else tied to the issue/topic at hand - ISKTBN
2. DOCUMENT EVERYTHING 
• This one wasn't so much for me. But, in general, ask yourself, "If I lose this, can I recreate it or will I be in deep doo doo without it (especially if it's critical to your operations)? If your in the doo, then WRITE THAT SHIZZ DOWN.
3. Nameservers are "a beaut, Clark!"
4. DNS IS HARD!

I'm sure there are times where nameservers be cursed. But this time, they made me look f'ing awesome. And the timing made me damn lucky. Had they waited another day, I can't say I would have been such a "rock star."

Bless your damn beauty.

DAMN NAMESERVERS (+ lucky timing!)