Azure DevOps is down globally
https://status.dev.azure.com/
US Central Azure cloud is down also, must be related: https://azure.status.microsoft/en-us/status I suppose this explains why OneDrive is down. Attempting to save to my OneDrive folder on my Mac gets an error in Excel, and suggests that I save a copy to somewhere else. Attempting to view my OneDrive files via the web interface shows no files and then eventually says something went wrong. Retrying cycles among different errors includes GeneralError, ServerTooBusy, and telling me to make sure I don't have any firewall or plugins that could be blocking access to api.onedrive.com. GitHub Actions is also degraded, coincidence? For me, pipelines are queuing and not even scheduling jobs on self hosted runners. Feel free to use this free time to take a peek at Bitbucket Pipeline, now running up-to 8x faster ;) https://bitbucket.org/blog/announcing-our-new-ci-cd-runtime-... Come check out some of our cool new features while you're at it: - https://bitbucket.org/blog/introducing-dynamic-pipelines-a-n... - https://bitbucket.org/blog/custom-merge-checks-are-now-gener... Great to see you guys engaging the community. Bitbucket may be 10 years behind GitHub, but the simplicity of pipelines is nice. Not a coincidence, no. From the "self-hosted" perspectively, interestingly what we're seeing at Depot (https://status.depot.dev/clyrvud6i57402igofm6jtb7id) is API requests to provision new runners or receive runner jobs are receiving rate limit errors, but without the regular rate limit status HTTP headers[1]. I imagine the GitHub API / Actions control plane is rather overloaded at the moment with the outage. [1] https://docs.github.com/en/rest/using-the-rest-api/rate-limi... No, from the documentation it seems they share a lot of infrastructure between GitHub Actions and Azure DevOps pipelines. very surprised by all the infrastructure hosted in us central. the page claims codespaces is functional but it clearly is not. it fails at loading vscode after bootstrapping the environment. Is it surprising? Dallas, STL, KC, Chicago all count as central. More or less equidistant for east/west coast markets. Seems like natural edge node coordinates along the scale ladder. "Central US" in this case, means the Des Moines, IA set of data centers. They're claiming only Central US is having issues on the cloud side. But, the portal itself is also having trouble loading pages and "global" services like identity management. Wider than DevOps. My entire company has been dead in the water for over 2 hours. It's funny how this story has not that much traction because it is Azure ... if it was AWS, half of the internet would be in flames. Airports are also at a standstill. Handwriting tickets in KC at the moment and “awaiting the manifest before we can board” - where do you think they store that manifest, a drawer in Mr Airline’s basement?? Uhhh same place as their entire system, which is DOWN "Big Paper and Pencil" can thank the recent MS and CDK Global hacks for making writing fashionable again ADO vs Teams, who takes the crown for worst software of all time? Teams, and it's not even close. Azure DevOps usually fails in understandable ways. Teams usually fails in ways that make me doubt my understanding of technology, society, and the nature of reality. Teams easily, IMO teams, for sure Guessing this is related: Minecraft is down, and possibly other games that use XBox Live Login: https://support.xbox.com/en-US/xbox-live-status "You may not be able to sign-in to your Xbox profile, may be disconnected while signed in, or have other related problems. Features that require sign-in like most games, apps and social activity won't be available." We definitely won't be getting a postmortem. never do. i can recall just before COVID hit seeing Azure AD go down -- broke all of our shit. but you look at the dashboard and it's 100% green. "it's fine" they said, and fought us for months. My web interface and pipelines appear to be down, but it seems like git operations are still working. Frontier Airlines is dead in the water... >Our systems are currently impacted by a Microsoft outage, which is also affecting other companies. During this time booking, check-in, access to your boarding pass, and some flights may be impacted. We appreciate your patience. Xbox Live was totally messed up earlier, I bet it's related. It seems like Azure DevOps migrated to some new login mechanism recently which is visibly slower than the old one, and I was seeing random failures prior to today. I have also noticed strange behavior that was not present before this perceived update, e.g. basic auth challenges and pop-up windows if I'm waking up my computer and still have azdo pages open. I believe that was a change to auth from a few weeks ago and is affecting every MS service. Not sure though and it does work, but it doesn't work gracefully. This seems to be affecting Windows VMs hosted in AWS, as well. I'm seeing status check failure for Windows VMs across multiple AWS accounts in our Org. What could possibly cause this kind of outage? Loss of a datacenter? Physical problem? How can a "DNS" problem (or something similar) be this widescale? I remember something with facebook? where their internal DNS was misconfigured. edit: "This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse. " https://engineering.fb.com/2021/10/05/networking-traffic/out... https://blog.cloudflare.com/october-2021-facebook-outage/ https://www.thousandeyes.com/blog/facebook-outage-analysis Not sure if MS is going through the same issue but DNS could be a reason why It's not DNS There's no way it's DNS It was DNS DNS is definitely capable of widescale disruption. A dns error can take down the entire internet. VSCode extension marketplace is also unaccessible spent a lot of time trying to troubleshoot why vscode can't access extension marketplace as I just installed a new computer and am trying to bring in my profile but getting XHR errors, and devtools showing CORS errors... after an hour+ of trying to get it to connect, finally figured the services are down. Entire Azure been offloaded to WITCH beware. I have noticed something along these lines in the GitHub repos. Tons of bugs piling up, issues closed out as "by design" where clearly is was not. Etc. I honestly get the impression Azure is in decline, particularly around the .Net integrations which is their flagship ecosystem.. I don't necessarily share this impression, but the teams that simply close items after being unable to sufficiently develop a solution, leading to the original issue raiser giving up after realizing that those who are supporting the issues are useless, are not helping with the public's impression of azure. It's a culture issue. The culture of the teams they are.. Offshoring ownership of these projects to. A lot goes into company and team culture, but I'm sure a lot here are familiar with the dynamics of outsourced development; offshore or not. Blame seeking, blame shifting, CYA, etc.. "by design".. It's all on full display in Microsofts Azure GitHub repos. It's a matter of culture in outsourcing. I'm a CTO in Mexico and outsource to companies in Mexico (insource modality) and it is evident outsourcing
people don't give a danm about the product . Why should they? They get paid by the hour, and if the client goes bust after they are paid, another will bite. Do you have a source for that? I couldn't find anything. What is WITCH? Been down hours. Those poor 9's Those 9's are still there. Just that the decimal places were misinterpreted, The decimal is going door to door, menacing the 9's. 89.99999999999 also has a lot of 9s. 99.999999999999999999999999% high availability fault tolerant and resilient infrastructure Ooooof.