Tell HN: Microsoft.com added 192.168.1.1 to their DNS record
Is this bad?
$ nslookup microsoft.com
Non-authoritative answer:
Name: microsoft.com Address: 192.168.1.0
Name: microsoft.com Address: 20.112.250.133
Name: microsoft.com Address: 20.231.239.246
Name: microsoft.com Address: 20.76.201.171
Name: microsoft.com Address: 20.70.246.20
Name: microsoft.com Address: 20.236.44.162
Name: microsoft.com
Address: 192.168.1.1 Through a series of connections I know a guy that knows a guy that works at Microsoft that was made aware and the changes have been reverted. Give 'er 30 minutes TTL ;) This is my favorite HN comment of 2023. HN is a wormhole through which you can connect pretty much anyone in tech industry. This is my second favorite HN comment of 2023 192.168.1.1 is gone now, but all authoritative nameservers are still offering 192.168.1.0. Oops. A few are dropping 192.168.1.0 now: Only one of those is authoritative. All of the authoritative servers have dropped it. Microsoft has fixed the issue. Yea, sure, but that's like saying "company X stopped selling poison, it's Shop Y's fault that they're still selling it". Nope. This is still Microsoft's fault while non-auth servers update. First we wanted to see Microsoft update the authoritative nameservers, then move on to monitoring propagation. Conflating the two makes it difficult to monitor whether Microsoft actually fixed it correctly this time, as they appear to have screwed up their first attempt. I wasn't attempting to address blame since I figured that was obvious enough. 30 minutes minutes or 30 Windows minutes? :P Actually, it’s looking more like 6 days. No wait, 30 seconds. This isn’t something that I think should be diluted. If it’s that simple for a stray record to be included in the dns round robin it could have been bad if it was an external ip with a machine setup by a phisherman especially since control of a domain is all you need to get an ssl cert now. Couple this with the fact that it’s Microsoft, one of the most relied on companies in our computer world, this is pretty darn horrible. Microsoft also has some of the phishiest looking domains when you are redirected around the O365 cloud. Absolutely, I am so glad I'm not the only person who feels this way. Microsoft does not understand domain names. They use 1drv.ms as the domain in OneDrive emails, and sometimes it almost looks like the .ms ccTLD belongs to them - it very much doesn't, anyone can register a .ms domain. windows.net being an Azure domain that third parties can have content under is fitting. It sometimes looks like they want their users to be phished. Microsoft is a smart company though, I really hope they can sort this out. 100%. Starting with "onmicrosoft.com". A phisher wouldn't really have to control Microsoft.com to take advantage of confusion. There were several phishing attempts from that domain, onmicrosoft.com, to my personal email account this past week. Microsoft.com has been constantly trying to fool me into subscribing to their Office tools. The only thing that competes is the redirecting when you log into any health portal. Indeed, take a look at the lists of azure and o365 domains, they're all over the place: https://learn.microsoft.com/en-us/microsoft-365/enterprise/u... https://learn.microsoft.com/en-us/azure/security/fundamental... Yeah. Easy to spot when your session expires - first they're still using login.live.com, just to redirect you to login.microsoftonline.com. For all this to work you need to control the domain.
Is that easier than simply breaking into their systems and owning their servers? That's exactly what they're saying. > it could have been bad if it was an external ip with a machine setup by a phisherman I.e. one of the IPs for microsoft.com belongs to $phisher, which means they control (a subset of the traffic going to) the domain. They can't add CNAME records for certificate validation, but LetsEncrypt for example offers HTTP-based validation. Not sure how Microsoft sets up their certificate pinning, it might not be quite that easy. For the Microsoft.com domain, proper, there seem to be no existing CAA rules, allowing each and every CA on earth to issue certificates based on whatever criteria the CA requires. What could possibly go wrong with that approach? It might also be a highly targeted attack on someone with precious information wherein someone was able to hack a simple router and in order to get access to their actual microsoft.com account, they simply setup a phisherman's clone on the router and captured the login/password/2fa and got into the account. Sometimes it’s easier to bribe then to break I know myself through a series of connections as well. ;) ;) If you know yourself through a series of connection then that's a false you. Not necessarily, in the same sense that I'm my neighbor's neighbor. You're also an alien's alien. That's just a sentence. Not who you are. https://en.wikipedia.org/wiki/Self-enquiry_(Ramana_Maharshi) I see them both. My TTL will run out at 16:39 PST, though. TTL appears to be set to an hour. But either way, its been 45 min and the primary ns1-39.azure-dns.com is still offering up 0.1 Few months early April 1 stunt? GG, Gone for me now. Do we need to restart our Windows machines? :) that's the solution to all Windows problems, so yes :) I was getting an empty answer for microsoft.com. Turns out my dnsmasq is blocking it: Wait wait wait wait. Bunny.net accidentally changed their DNS to 127.0.0.1 and took a bunch of their CDN users down today too. Coincidence? Weird day. Employees mixing up SSH consoles while setting up their smart home Christmas lights over weekend. <singing voice="Chris Rea">Becoming 'home' for Christmas </singing> Probably just developers trying to meet deliveries/targets before the holidays. microsoft.com is currently IPv6-only on my network, because OpenWrt's DNS rebinding protection filters out the A records: I'm trying to figure out how this could have happened, but I control so few IP addresses that many of my DNS entries are manually assigned. And you'd have to be incompetent if you have access to set DNS records and you set them to RFC 1918 addresses. Anyone have any theories on how this could happen? I'll go with Joseph Conrad on this one. "It's only those who do nothing that make no mistakes, I suppose." Now the persons that did it have some proof that they did something. They will surely put some check in place because there should be another adage somewhere that says that you only learn to use the handrails after you fell in the stairs. erro ergo sum about ten years ago, apple added a stray record to the apple.com zone .... a DNAME[1] record ....... that pointed to apple.com 1: https://en.wikipedia.org/wiki/DNAME_record this had some pretty disastrous results[2] 2: https://mashable.com/archive/apple-tunes-app-store-icloud-pr... bad things happen everywhere It continually astounds me that DNAME got standardized. Scary stuff. * Copy/Paste * Copilot told me * Sabotage (internal or external) Or hard to reason about IaC my wild idea: counterattack for DDoS Certainly just some automation bug, perhaps a few things strung together, like a dev environment setup that leaked into production. A human in the loop making a mistake probably as well. This is the kind of thing you look at and put up a few guardrails to prevent it happening again. My guess is that someone at MS was testing Windows Updates or other changes from a local source. They also had some other DNS updates in their config they were testing. They took all of their config and pushed it out, when they should only have taken the other changes. I hope that's not their workflow, but few people I've ever worked with have know how to create operating procedures that had half a chance of succeeding, usually it's based on "oh, don't worry, i would never do that" >> And you'd have to be incompetent if you have access to set DNS records and you set them to RFC 1918 addresses. Clearly the following is not in play for a root domain (Microsoft.com) but assigning a DNS entry to a class C address does have a purpose. If you have an intranet server, giving it a DNS name allows for HTTPS serving, with an automatic, CA signed, certificate. (Using say LE with DNS challenge.) I provide this simply as an example of how this might come about. that's what happens when you buy address space from the back of a van in the parking lot ;) How dare you besmirch the reputation of The Awesome Store! https://theamazingworldofgumball.fandom.com/wiki/Awesome_Sto... I'm surprised 192.168.1.0 is still there 2.5 hours later https://dnstools.ws/lookup/microsoft.com/A/ I get it too, with .1.0 as well 1.1 is gone but I'm still seeing the 1.0 entry. Same here with 1.0 They probably asked copilot to manage their DNS servers "help us bring Microsoft to every home's network" AI-driven type coercion is a new type of debugging hell. Potential timeouts for clients/workstations trying to reach microsoft.com. Which entry is picked for use is generally random depending on the client. Most systems will retry using another entry though on issues connecting through. That said, if you are on a network that is 192.168 based, trying to get to Microsoft.com may just send you to your local router! 29% traffic lost I imagine it has something to do with how MS creates bypasses for host files for systems xpsp2 onwards ... by [1] it suggests Win10 still does. [1] https://superuser.com/questions/1111582/does-microsoft-preve... So let me see if I understand. With this DNS record, if me or Windows tries to hit “microsoft.com” there’s a 1/7 chance it hit my router instead? Yes I don’t know man, putting microsoft.com on your router sounds like a massive reduction in latency. Congrats on the achievement. Was someone able to generate a *.microsoft.com SSL certificate when doing domain validation on non-microsoft machine ? for uninitiated (me), why is it bad? Well in my case (and a lot of other people), 192.168.1.1 is the local address of my home router. So if I go to microsoft.com I have a 1 in 7 chance of getting my home router instead (if I ignore the certificate warning). Other random breakage will happen depending on what that local address is assigned to for you. In theory this could be leveraged for hacking, but I think that would require setup in advance. yep. If a hacker can somehow control 192.168.1.1 or 192.168.0.1 they get access to your microsoft.com cookies at least. I'm sure there are more microsoft specific ways to leverage this too (e.g. data/updates hosted on microsoft.com that misuse HTTPS as a poor man's authentication. The curl | sh crowd are especially susceptible to this problem.) They would still need the private key to a https cert that your browser considers valid. Since 2 out of the 7 IPs are 192.168 (private ips), 2/7 visitors to microsoft.com will load the private ones assumign equal weight and not get the page to load. It's an IP address reserved for private networks: So if you go to microsoft.com with probability 1/7 you'll hit 1.1 on your private network of 192.168 - likely router, and with probability 1/7 you'll hit 1.0 maybe printer Y'all, instead of the constant confirmed here. Just do an authoritative lookup. dig +trace +short microsoft.com NS a.root-servers.net. from server 100.100.100.100 in 10 ms. NS b.root-servers.net. from server 100.100.100.100 in 10 ms. NS c.root-servers.net. from server 100.100.100.100 in 10 ms. NS d.root-servers.net. from server 100.100.100.100 in 10 ms. NS e.root-servers.net. from server 100.100.100.100 in 10 ms. NS f.root-servers.net. from server 100.100.100.100 in 10 ms. NS g.root-servers.net. from server 100.100.100.100 in 10 ms. NS h.root-servers.net. from server 100.100.100.100 in 10 ms. NS i.root-servers.net. from server 100.100.100.100 in 10 ms. NS j.root-servers.net. from server 100.100.100.100 in 10 ms. NS k.root-servers.net. from server 100.100.100.100 in 10 ms. NS l.root-servers.net. from server 100.100.100.100 in 10 ms. NS m.root-servers.net. from server 100.100.100.100 in 10 ms. RRSIG NS 8 0 518400 20240101050000 20231219040000 46780 . fG/YHtUJu3YMAm9Mlzzvp3xG4UCPG01aYNnlyF1HfAHdZpR+L88CVUcz NFHq9M45KjB7ZTlSFt2JvEyK/8FcavZLOthkXRREbJQswjLCbhiPQCbq tQLF+tKaNYUihqawCfjgZy1i5YwYjmphbjfzwoKo1POtepf0YCIcuLBi nQFw4Lr79O6cjyg6qlYnqaK6z4Xi5qt6ocohJafjs86LuuRo2WvmJ1IK k0ZUoAC6Qyjz4MVhqHMvQGdp7EnzjoL8Y9PTXeUuD6Ixp/Aklj2psLjD TZDPYN1q+zDd1giFyuwNRX9DG1zrxzN2lzQiLWmGKrzP3DvFWL1L2Ts1 FWjy/Q== from server 100.100.100.100 in 10 ms. ;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable. ;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable. ;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable. A 20.112.250.133 from server 150.171.10.39 in 20 ms. A 20.231.239.246 from server 150.171.10.39 in 20 ms. A 20.76.201.171 from server 150.171.10.39 in 20 ms. A 20.70.246.20 from server 150.171.10.39 in 20 ms. A 20.236.44.162 from server 150.171.10.39 in 20 ms. A 192.168.1.0 from server 150.171.10.39 in 20 ms. Or from a bunch of dnses: What could a TLA do with this if it had time to plan ahead? Serve malicious updates from a locally controlled machine, for one. Lord knows about auth. Do most DNS forwarders not block addresses that resolve to a local IP these days? I know dnsmasq does, and NextDNS too I think. I think most will see it as a DNS rebinding attack [1]. That’s the phrase I was looking for! Why? Having local IPs on a public DNS is a legitimate use case. As another reply mentioned, to prevent DNS rebinding attacks. The general expectation is you will whitelist domains from which you expect RFC1918 responses. In fact, some people block domains by routing them to 127.0.0.1 in their host files. I've used private ranges too, in places where loopback might possibly do something funky. > Serve malicious updates from a locally controlled machine. Lord knows about auth. Wouldn't they have to break into my local machine first, plant an update service, and an update? That doesn't seem to scale well at all, and wouldn't it be easier to just break into the machine they want to 'update'? A fairly prominent update service already runs from the domain microsoft.com Many machines come with it preinstalled. The erroneous DNS change wouldn't help that sort of exploit. It just redirects attempts to contact microsoft.com to a local address, probably a router. That's exactly what I said in the first post. Why doesn't windows update use authentication (eg https)? They do: the updates are signed so our hypothetical spies would need to have a zero day in Authenticode or to have compromised the signing keys. For the uninitiated, can some traffic get sent to 192.168.1.1. Is it round robin? Yes it is. I could be wrong but in my experience OS just selects one random and uses that for some time not round robins it. Even if that is the case, if it is random, some section of DNS would send traffic to it. Maybe it was OK because most resolvers would ignore the local address on the list?? Maybe Sydney tried to breakout… Good point. Let's watch Microsoft executives & employees closely for signs of panicking over an escaped AGI. Employees mix up SSH consoles while setting up their smart home Christmas lights over the weekend. https://www.rtasks.online/ Looks like somebody made a booboo An entry-level admin is now unemployed, just before the holidays. > An entry-level admin is now unemployed, just before the holidays. I highly doubt that entry-level admins at Microsoft have access to DNS for their primary domain. My guess is that this incident is a lot more interesting than that. Yep, this doesn't seem like the kind of thing that you can just toss a couple approvals on and change at a company as big as Microsoft. How this made it through the review process would be very interesting Nah, if it's already reverted, they're good to go. A post-mortem with how something like that got through will definitely be on the table though. I'm wondering how such a change would get "merged" in to begin with. I imagine even non-network engineers would get this huge itch having a large corporate contain a private IP in the changelist (I'm the non network engineer and can't really explain why it's bad. But it FEELS wrong and sometimes you at least need to use instinct to get another pair of eyes on something). I hope not. Failures are on a spectrum and this was unfortunate but probably not malicious. All things considered this should be a lesson learned. There should be more failsafe mechanisms in place so juniors can fail safely and learn from them. The absolute worst thing we can do is shame an individual so they don’t attempt to try new things in fear of ridicule. > There should be more failsafe mechanisms in place so juniors can fail safely and learn from them. And if not, whoever put the junior in that role is the person responsible for the problem. well theoretically you could argue the structure of this task should have 'dual control' / multiple people should be involved in the process checking each others work. preferably even split it up people who do not know or interact with each other on a regular basis. yes it would be slower but its important to get it correct. might as well throw in some automated poke-yoke or whatever too. in that case there is no fault in any of the juniors or operators, the fault is in management for failing to implement infrastructure to force a critical process to have more than one control The seniors all go on leave and the interns are left to run the place. If they fired the juniors the seniors would have to come back from holiday! This is most likely a honest mistake. Smart managers don't fire employees for such mistakes unless their behavior regarding that mistake is inappropriate. As the story goes, after a junior admin wiped a production database. The boss was asked if he should be fired. To what he answered: "Fire him? No way! Not after such an expensive training." Now, he knows. "Works on my machine" Another confirmation here. Whoops! Damn you Murphy's law. Confirmed here. stumbled on this thread, my device just got blocked by my at home router. My dns is 192.168.1.1, any suggestions for how to troubleshoot this? Wait... Can DNS resolvers be configured so that RFC1918 is respected? I mean: I don't expect anything less from Microsoft than doing stuff like that and it cannot affect me for I nullroute microsoft.com from my unbound server (unboud takes wildcard when nullrouting or NXDOMAINing crap domains like microsoft.com or meta.com etc., which is sweet). However I'd expect my trusty DNS resolver to also prevent me from anyone not on my private LANs to impersonate addresses reserved for private uses. Does anyone know here if it's easily doable? You're looking for DNS rebinding protection, many DNS servers support it. However there are some cases where things do use private IPs in DNS records outside of the local domain, one example is Plex (e.g. https://support.plex.tv/articles/206225077-how-to-use-secure... suggests turning off DNS rebinding protection) -- although in some cases you can allow particular domains which is a much better way than turning it off entirely. (See also the sibling comment about microsoft.com being IPv6 only as a result of a particular implementation of DNS rebinding protection: https://news.ycombinator.com/item?id=38704159) Yes, some can. Unbound's "private-address" and "private-domain" directives control this. Similarly, bind9 has "deny-answer-addresses" (with an "except-from" option so you can specify local domains that are allowed to use them): https://bind9.readthedocs.io/en/v9.18.20/reference.html#cont... Not sure about others. I wouldn't expect it to. I have plenty of RFC-1918 addresses in a subdomain of my public DNS zone for my home network. It's been that way for decades. (Perhaps I should use split DNS, but...) My Unbound servers strip RFC out. Public resolvers keep DNS answers intact because they can carry alt data like how dodgy a SMTP server is. This seems the opposite of good. How the hell did that pass any sort of responsible review process at Microsoft? Now Microsoft owns all your home networks, only like the default address on every home router out there... You have the danger of this backwards: this is a very bad security problem for Microsoft, and not a problem for people outside of MS (except to the extent that we're all indirectly reliant on MS being secure). Pointing a domain at an IP address does not give you any power over than IP address, and you can point a domain at anything you want. Any risk here is nearly the opposite of what you seem to think it is No they don’t. Going to Microsoft.com will take you to your router. > Now Microsoft owns all your home networks Only if you’re slumming around 192.168.x.x I for one only use Class A CIDR, 10.0.0.0/8 Same click click click -- "it's done, boss." Microsoft's mindnumbingly dense ClickOps culture strikes again. at a serious org this would have involved at least some level of oversight or intervention No, it's that when you open microsoft.com it could open your router page.
as of 1703035296:
ns1-39.azure-dns.com no longer has 192.168.1.0 for microsoft.com
1.1.1.1 still has 192.168.1.0 for microsoft.com
8.8.8.8 still has 192.168.1.0 for microsoft.com
76.76.2.0 no longer has 192.168.1.0 for microsoft.com
9.9.9.9 still has 192.168.1.0 for microsoft.com
208.67.222.222 still has 192.168.1.0 for microsoft.com
185.228.168.9 still has 192.168.1.0 for microsoft.com
76.76.19.19 still has 192.168.1.0 for microsoft.com
94.140.14.14 still has 192.168.1.0 for microsoft.com
$ dig microsoft.com. | grep EDE
; EDE: 15 (Blocked)
resolver.log:Dec 20 00:43:57 router dnsmasq[8172]: possible DNS-rebind attack detected: microsoft.com
$ ping -4 microsoft.com
ping: microsoft.com: Address family for hostname not supported
$ ping -6 microsoft.com
PING microsoft.com(2603:1030:c02:8::14 (2603:1030:c02:8::14)) 56 data bytes
64 bytes from 2603:1030:c02:8::14 (2603:1030:c02:8::14): icmp_seq=1 ttl=112 time=68.4 ms
"ooopsie!" Name: microsoft.com
Address: 192.168.1.1
Name: microsoft.com
Address: 192.168.1.0
$ export srch="192.168.1.0"; echo "as of $(date '+%s';):"; for dns in 1.1.1.1 8.8.8.8 76.76.2.0 9.9.9.9 208.67.222.222 185.228.168.9 76.76.19.19 94.140.14.14; do dig @${dns} microsoft.com +short | grep "${srch}" > /dev/null; if [ $? == 0 ]; then echo "${dns} still has ${srch} for microsoft.com"; else echo "${dns} no longer has ${srch} for microsoft.com"; fi; done
as of 1703033639:
1.1.1.1 still has 192.168.1.0 for microsoft.com
8.8.8.8 still has 192.168.1.0 for microsoft.com
76.76.2.0 still has 192.168.1.0 for microsoft.com
9.9.9.9 still has 192.168.1.0 for microsoft.com
208.67.222.222 still has 192.168.1.0 for microsoft.com
185.228.168.9 still has 192.168.1.0 for microsoft.com
76.76.19.19 still has 192.168.1.0 for microsoft.com
94.140.14.14 still has 192.168.1.0 for microsoft.com
$ pbpaste | sed 's;^; ;' | pbcopy