Beginning 02:52 on Mar 3, 2023 an incident occurred impacting access to the linodeobjects.com domain name. This impacted all customers without cached DNS entries attempting to access the Linode Object Storage service, and customers who utilized the domain via a CNAME record. The internal teams were notified at 02:57, and after initial triage, further teams were paged by 03:10 to investigate further.
The root cause of the incident was a non-billing hold placed on the linodeobjects.com domain by the registrar which prevented DNS from functioning for the domain. After initial attempts to contact the vendor were unsuccessful, attempts were made to begin mitigating this issue through other means. By 05:43 UTC, access to the Object Storage service was restored for customers accessing the service via the internal network (such as from a Linode instance to the Object Storage service). Full service restoration was not achieved until 11:56 UTC when the domain was moved to a new registrar. Customers may not have seen this restoration for up to 30 minutes depending on local caching.
We are still investigating with the former registrar to determine why the non-billing hold was placed on the domain.
As a result of this, we will be conducting an internal audit of critical services and the vendors they are tied to in order to better assess support accessibility and establish escalation paths to more easily reach these vendors during urgent incidents. As this domain is no longer on the same registrar, we do not anticipate a further recurrence of this issue impacting this service.