Azure customers running Canonical Ubuntu 18.04 experiencing DNS errors
azure.status.microsoft> running Ubuntu 18.04 (bionic) VMs recently upgraded to systemd version 237-3ubuntu10.54 reported experiencing DNS errors when trying to access their resources. Reports of this issue are confined to this single Ubuntu version.
Buried lede: it's yet another problem caused by systemd.
Why anyone ever switched from GNU OS to Systemd OS is beyond me. Oh wait it's not: they were lied to and scammed.
This is more of a Canonical issue than an Azure issue IMO. They pushed a broken systemd package. Is there any QA even going on at Canonical?!
MS push having automatic unattended updates turned on. Azure's security recs complains without end if you don't. Much like desktop Windows deciding when it should update & reboot.
I see the argument for getting security updates out there ASAP, but this case proves my point that it is a bad idea to do it automatically [unless perhaps there is a serious remotely vulnerable actively exploited issue] on any sort of production environment. Deploy to test environments and verify (at very least smoke test) first then apply elsewhere. If you don't have test environments, at least do the deploy to prod/other at your control when someone is available to quickly take action regarding any unexpected issues.
So the initial problem may be with Ubuntu, or upstream systemd, but MS policies magnified it significantly.
systemd resolved has been broken in many releases of Ubuntu, this isn't surprising. it works most of the time but frequently will return errors.
How unreliable azure is, it is impressive, just like with recent version of Windows
It seems like something at this company is not functioning well, it seems to be operated like it's no big deal, no QA, no quality and performance culture, it's a giant mess
It has infected Github too, impressive..
Google Cloud / Microsoft Azure, they have a looooooong way to go before being able to properly compete with AWS
But the same issue would happen on AWS, it's an Ubuntu problem caused by Canonical
MS push having automatic unattended updates turned on. Azure complains without end if you don't, sometimes Defender (which Azure complains about if you don't have installed) updates override you having turned them off when it updates.
There being a bug in the package may be an Ubuntu thing, me losing time to diagnosing problems caused by the update getting applied without my knowledge is very much an MS/Azure problem.
At least this incident gives me fuel for strongly suggesting that we turn unattended-updates off and darn well keep it off (obviously we need to keep alerts on, but we should install updates in an orderly manner, when we can monitor the results and smoke-test, preferably applying to test resources before uat/prod/other).
It's hitting loganalytics. That's not really a service for which customers deal with VMS directly.
What other cockamamie opinions do you have? You should write a book, publish it to /dev/null