Release asset failing to download due to account egress limit · community · Discussion #8535

9 min read Original article ↗

Hey all! We've seen this strange error when running terraform a few times today in our CI/CD environments, and it appears be saying that the release asset cannot be downloaded from github due to the account y'all are using for storage being over the egress limit?

The error:

│ Error while installing cloudposse/utils v0.14.2: unsuccessful request to
│ https://github.com/cloudposse/terraform-provider-utils/releases/download/0.14.2/terraform-provider-utils_0.14.2_linux_amd64.zip:
│ 503 Egress is over the account limit.

It's consistently this one package from what we've seen, and we arent using any proxies that would limit egress on our end between our servers in AWS + Azure DevOps that would trigger an error like this. Retries also generally result in success, so it's intermittent and not consistent. Brief googling looks like this error message is associated with azure blob storage which isnt something that we currently make use of either, so I wanted to open up a report here in case it's an actual issue on the github side of things since that's what it feels like to me currently.
You must be logged in to vote

It looks like this incident has been resolved so I am going to mark this discussion as answered.

Thanks everyone for your patience. We are continuing internal work and coordination to minimize future incidents like this and improve our ability to monitor and act quickly when there are issues like this.

View full answer

I am seeing the same in our TF pipelines, although the asset is not always the same.

│ Error: Failed to install provider
│ 
│ Error while installing terraform-aws-modules/http v2.4.1: unsuccessful
│ request to
│ https://github.com/terraform-aws-modules/terraform-provider-http/releases/download/v2.4.1/terraform-provider-http_2.4.1_linux_amd64.zip:
│ 503 Egress is over the account limit.

│ Error: Failed to install provider
│ 
│ Error while installing gavinbunney/kubectl v1.13.1: unsuccessful request to
│ https://github.com/gavinbunney/terraform-provider-kubectl/releases/download/v1.13.1/terraform-provider-kubectl_1.13.1_linux_amd64.zip:
│ 503 Egress is over the account limit.
You must be logged in to vote

0 replies

Same here - running at home pulling a GitHub release. Command is

wget \
  https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-aarch64.sh

Response is

--2021-12-08 07:44:45--  https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-aarch64.sh
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github.com/conda-forge/miniforge/releases/download/4.10.3-10/Mambaforge-Linux-aarch64.sh [following]
--2021-12-08 07:44:45--  https://github.com/conda-forge/miniforge/releases/download/4.10.3-10/Mambaforge-Linux-aarch64.sh
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/221584272/15c82678-e7c8-4951-a768-5a1808b482c8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211208%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211208T074446Z&X-Amz-Expires=300&X-Amz-Signature=ea7077f51983587470258cc230a7aa46c0555c4bc36b8ee1db3a9a45bf27d3ec&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=221584272&response-content-disposition=attachment%3B%20filename%3DMambaforge-Linux-aarch64.sh&response-content-type=application%2Foctet-stream [following]
--2021-12-08 07:44:46--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/221584272/15c82678-e7c8-4951-a768-5a1808b482c8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211208%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211208T074446Z&X-Amz-Expires=300&X-Amz-Signature=ea7077f51983587470258cc230a7aa46c0555c4bc36b8ee1db3a9a45bf27d3ec&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=221584272&response-content-disposition=attachment%3B%20filename%3DMambaforge-Linux-aarch64.sh&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 503 Egress is over the account limit.
2021-12-08 07:44:51 ERROR 503: Egress is over the account limit..
You must be logged in to vote

0 replies

I'm getting a GitHub release on GitLab CI and running the pipeline, but I'm getting the same issue.
image
You must be logged in to vote

0 replies

You must be logged in to vote

0 replies

We have a similar issue with

https://www.github.com/fabpot/local-php-security-checker/releases/download/v1.0.0/local-php-security-checker_1.0.0_linux_amd64

Also HTTP Error 503: Egress is over the account limit.

Does this mean the limit of the repository owner or the limit of the account requesting (Which I assume only is possible when using an auth token in your requests) ?
You must be logged in to vote

1 reply

@chapitos

Same here for the newrelic provider:

Error: Failed to install provider │ │ Error while installing newrelic/newrelic v2.23.0: unsuccessful request to │ https://github.com/newrelic/terraform-provider-newrelic/releases/download/v2.23.0/terraform-provider-newrelic_2.23.0_linux_amd64.zip: │ 503 Egress is over the account limit. ╵

Same here for Loki:
Failure downloading https://github.com/grafana/loki/releases/download/v1.3.0/promtail-linux-amd64.zip, HTTP Error 503: Egress is over the account limit.

I'm building images wirth packer - first seen last night with AWS, then this morning with Azure and Alicloud. It seems intermittent.
You must be logged in to vote

0 replies

Experienced the same issue;

failed to GET https://github.com/krallin/tini/releases/download/v0.19.0/tini with status 503 Service Unavailable: <?xml version="1.0" encoding="utf-8"?><Error><Code>ServerBusy</Code><Message>Egress is over the account limit.

After retry it worked again.
You must be logged in to vote

1 reply

@mvdwaeter

and it happened again after..

Yep, same here with rancher

│ Error: Failed to install provider │ │ Error while installing rancher/rancher2 v1.21.0: unsuccessful request to │ https://github.com/rancher/terraform-provider-rancher2/releases/download/v1.21.0/terraform-provider-rancher2_1.21.0_linux_amd64.zip: │ 503 Egress is over the account limit.
You must be logged in to vote

0 replies

Same error

https://github.com/electron/electron/releases/download/v9.3.2/ffmpeg-v9.3.2-win32-x64.zip
You must be logged in to vote

0 replies

We see the same issue in our CI pipeline when initializing terraform:

2021-12-08 10:16:05  Initializing provider plugins...
2021-12-08 10:16:05  - Finding terraform-provider-openstack/openstack versions matching "~> 1.35.0"...
2021-12-08 10:16:12  - Installing terraform-provider-openstack/openstack v1.35.0...
2021-12-08 10:16:17  ╷
2021-12-08 10:16:17  │ Error: Failed to install provider
2021-12-08 10:16:17  │ 
2021-12-08 10:16:17  │ Error while installing terraform-provider-openstack/openstack v1.35.0:
2021-12-08 10:16:17  │ unsuccessful request to
2021-12-08 10:16:17  │ https://github.com/terraform-provider-openstack/terraform-provider-openstack/releases/download/v1.35.0/terraform-provider-openstack_1.35.0_linux_amd64.zip:
2021-12-08 10:16:17  │ 503 Egress is over the account limit.
You must be logged in to vote

0 replies

Same here

Error: Failed to install provider

Error while installing psychopenguin/gandi v2.0.0-rc3: unsuccessful request
to
https://github.com/psychopenguin/terraform-provider-gandi/releases/download/v2.0.0-rc3/terraform-provider-gandi_2.0.0-rc3_linux_amd64.zip:
503 Egress is over the account limit.
You must be logged in to vote

0 replies

Same here

Error: Failed to install provider
│ 
│ Error while installing pagerduty/pagerduty v2.2.0: unsuccessful request to
│ https://github.com/PagerDuty/terraform-provider-pagerduty/releases/download/v2.2.0/terraform-provider-pagerduty_2.2.0_linux_amd64.zip:
│ 503 Egress is over the account limit.
You must be logged in to vote

0 replies

I contacted GitHub support about this yesterday and have posted their reply (and my summary) at lovell/sharp-libvips#121 (comment)

We have indeed had multiple reports of this bug over the last couple of hours and our engineers are working on a fix. It's very useful for us to have more data points, so thank you for submitting this report! I'll pass it on to the team.

Anyone else experiencing this problem should contact GitHub support to provide more evidence via https://support.github.com/request
You must be logged in to vote

2 replies

@jpike88

'data points'? Do they not have server logs?

@lovell

You must be logged in to vote

1 reply

@rfrerebe-stx

Using cache solved our issue.
Terraform should probably enable it by default.

Same issue for us downloading an artifact as part of a build script:

wget -c https://github.com/zmwangx/ets/releases/download/v0.2.1/ets_0.2.1_linux_amd64.tar.gz -O - | tar -xz -C /usr/local/bin
Connecting to github.com (140.82.121.4:443)
Connecting to objects.githubusercontent.com (185.199.108.133:443)
wget: server returned error: HTTP/1.1 503 Egress is over the account limit.

Simply retrying has worked for us so far.
You must be logged in to vote

1 reply

@ancient07

Same issue with prometheus
/usr/bin/curl https://github.com/prometheus/node_exporter/releases/download/v1.0.0/node_exporter-1.0.0.linux-amd64.tar.gz -o /tmp/node_exporter-1.0.0.tar.gz_20211208-1666-1bkfa2k -fsSLg --max-redirs 5' returned 22: curl: (22) The requested URL returned error: 503 Egress is over the account limit.
You must be logged in to vote

3 replies

You must be logged in to vote

4 replies

@sheilalarmour

I can't wait to meet you.

@sheilalarmour

Someone said I can't stay there with you. So I was wondering if they were interested in you.

@sheilalarmour

So I decided to go day-by-day.

@nocive

You must be logged in to vote

0 replies

You must be logged in to vote

0 replies

Issue is affecting my PoB today, as well.


HTTP request sent, awaiting response... 503 Egress is over the account limit.
2022-10-13 18:25:58 ERROR 503: Egress is over the account limit..
You must be logged in to vote

0 replies

Currently we are also getting this when downloading files for ansible provisioning scripts
Failure downloading https://github.com/github-asset, HTTP Error 503: Egress is over the account limit.
You must be logged in to vote

0 replies

We are also getting this issue:

 curl -s https://api.github.com/repos/DataDog/dd-trace-php/releases/latest | grep "browser_download_url.*apk" | head -1 | cut -d / -f 8,8 > DATADOG_VERSION     && wget https://github.com/DataDog/dd-trace-php/releases/download/$(cat DATADOG_VERSION)/datadog-php-tracer_$(cat DATADOG_VERSION)_$(uname -m).apk     && apk add --no-cache --allow-untrusted datadog-php-tracer_$(cat DATADOG_VERSION)_$(uname -m).apk     && rm datadog-php-tracer_$(cat DATADOG_VERSION)_$(uname -m).apk
 ---> Running in d7b22fa8c30d
Connecting to github.com (140.82.113.4:443)
Connecting to objects.githubusercontent.com (185.199.111.133:443)
wget: server returned error: HTTP/1.1 503 Egress is over the account limit.
You must be logged in to vote

0 replies

Still getting this error occasionally.
You must be logged in to vote

0 replies

We are facing the same issue right now. Is there a way how to download public release assets with api token or something?
You must be logged in to vote

0 replies

Happened again today:

npm ERR! node-pre-gyp http GET https://github.com/ranisalt/node-argon2/releases/download/v0.30.2/argon2-v0.30.2-napi-v3-linux-x64-musl.tar.gz
npm ERR! node-pre-gyp ERR! install response status 503 Egress is over the account limit. on https://github.com/ranisalt/node-argon2/releases/download/v0.30.2/argon2-v0.30.2-napi-v3-linux-x64-musl.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for argon2@0.30.2 and node@18.13.0 (node-v108 ABI, musl) (falling back to source compile with node-gyp) 
npm ERR! node-pre-gyp WARN Hit error response status 503 Egress is over the account limit. on https://github.com/ranisalt/node-argon2/releases/download/v0.30.2/argon2-v0.30.2-napi-v3-linux-x64-musl.tar.gz
You must be logged in to vote

1 reply

@mburri

similar issue for us since yesterday

/var/lib/jenkins/tools/com.cloudbees.jenkins.plugins.customtools.CustomTool/gitcrypt/root/bin/git-crypt: line 1: `���<?xml version="1.0" encoding="utf-8"?><Error><Code>ServerBusy</Code><Message>Egress is over the account limit.'

We have seen similar errors in our deployments as of recent as well for various provider packages. Here is one of the most recent examples from today that lead me to find this post.

13:58:53  Error: Failed to install provider
13:58:53  
13:58:53  Error while installing cloudflare/cloudflare v3.22.0: unsuccessful request to
13:58:53  https://github.com/cloudflare/terraform-provider-cloudflare/releases/download/v3.22.0/terraform-provider-cloudflare_3.22.0_linux_amd64.zip:
13:58:53  503 Egress is over the account limit.
You must be logged in to vote

0 replies

It looks like there was a breakage again. Our CI failed to download dependencies from github.com/**/release URLs.
You must be logged in to vote

0 replies 

Step 10/44 : ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions /usr/local/bin/
ADD failed: failed to GET https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions with status 503 Service Unavailable: <?xml version="1.0
" encoding="utf-8"?><Error><Code>ServerBusy</Code><Message>Egress is over the account limit.
RequestId:e2e26f22-301e-0062-62c6-55e042000000
Time:2023-03-13T16:14:17.1478266Z</Message></Error>
You must be logged in to vote

0 replies

Another one today:

node-pre-gyp ERR! install response status 503 Egress is over the account limit. on https://github.com/Automattic/node-canvas/releases/download/v2.11.0/canvas-v2.11.0-node-v64-linux-glibc-x64.tar.gz 
node-pre-gyp WARN Pre-built binaries not installable for canvas@2.11.0 and node@10.23.1 (node-v64 ABI, glibc) (falling back to source compile with node-gyp) 
node-pre-gyp WARN Hit error response status 503 Egress is over the account limit. on https://github.com/Automattic/node-canvas/releases/download/v2.11.0/canvas-v2.11.0-node-v64-linux-glibc-x64.tar.gz
You must be logged in to vote

0 replies

Another one today:

After some minutes of downloading, it fails. Tested on Chrome, Firefox, and wget. At least, wget shows me the error message.

HTTP request sent, awaiting response... 618 jwt:expired
You must be logged in to vote

0 replies