git checkouts extremely slow or timing out from EU · community · Discussion #196638

enescakir · GitHub

9 min read Original article ↗

🏷️ Discussion Type

Question

💬 Feature/Topic Area

Actions Checkout

Discussion Details

We operate a fleet of self-hosted GitHub Actions runners in an EU region. Starting around 2026-05-19, multiple workflows have begun experiencing severely degraded actions/checkout performance.

Is anyone else experiencing similar issues? We haven’t been able to find any active incidents

Symptoms

git fetch against GitHub from EU runners exhibits one of three failure modes, all of which look like the same underlying throughput collapse:

  • Silent stall — no output for 15–25 min, killed by timeout-minutes.
  • HTTP/2 stream cancelled — RPC failed; curl 92 HTTP/2 stream X was not closed cleanly: CANCEL, fatal: early EOF.
  • Extremely slow transfer — Receiving objects crawls along at 10–25 KiB/s.

Handshake (DNS, TCP, TLS) is consistently healthy; the problem is mid-transfer.
You must be logged in to vote

Ok, so we identified degraded behavior on a network transport path that could increase transfer latency for some requests while leaving others unaffected, which is why reproduction was inconsistent. On May 29, traffic was moved away from the affected path and that 'should' mean this is mitigated. We are also digging into our alerting as to why we didn't autoflag this earlier.

Can folks let me know if you are still seeing issues here.

View full answer

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

  • Your input will be carefully reviewed and cataloged by members of our product teams. 
    • Due to the high volume of submissions, we may not always be able to provide individual responses.
    • Rest assured, your feedback will help chart our course for product improvements.
  • Other users may engage with your post, sharing their own perspectives or experiences. 
  • GitHub staff may reach out for further clarification or insight. 
    • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

  • Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
  • Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

  • Upvote and comment on other user feedback Discussions that resonate with you.
  • Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐
You must be logged in to vote

0 replies

We're seeing the same behavior with runners in eu-west-3.
Our only mitigation is adding timeout-minutes: 1 to all checkouts, which makes them fail fast.
You must be logged in to vote

0 replies

We observed the same issue when running Git operations from AWS EC2 instances in the EU region.

Git over SSH is significantly slower during git fetch / checkout operations
HTTPS (likely backed by CDN endpoints) performs noticeably better
Multiple tests confirm consistent results across different runs
Using HTTPS instead of SSH improves repository fetch performance
You must be logged in to vote

0 replies

You must be logged in to vote

0 replies

This comment was marked as low quality.

@JulyanXu forcing HTTP 1.1 does not fix the issue on my projects.
You must be logged in to vote

0 replies

I want to mention I get the same issues even with a US-based fleet of runners (us-east-2).
You must be logged in to vote

0 replies

We are experiencing this as well and it has blocked the development process many times in the last few days.

 /usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
  Error: error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
  Error: error: 321 bytes of body are still expected
  fetch-pack: unexpected disconnect while reading sideband packet
  Error: fatal: early EOF
  Error: fatal: fetch-pack: invalid index-pack output
  The process '/usr/bin/git' failed with exit code 128
  Waiting 10 seconds before trying again
You must be logged in to vote

0 replies

Another +1 to this, we're experiencing many extremely slow git checkout action runs (from EU).
You must be logged in to vote

0 replies

Also nothing of the changes suggested by JulyanXu helped.
You must be logged in to vote

0 replies

Same here, both in CI and and on my local machine.
You must be logged in to vote

0 replies

Still experiencing severe slow downs for object crawls,
with a throughput of around 15-30 KiB/s. Our runners are hosted in Europe at Hetzner (Germany DC).
You must be logged in to vote

0 replies

We got feedback from Github support that there was an issue on their side and that it should be fixed now.
You must be logged in to vote

0 replies

This comment was marked as spam.

It runs the HTTP/2-vs-HTTP/1.1 comparison with full tracing so the output is exactly what GitHub Support needs.

Standalone shell (run directly on an affected EU runner):

#!/usr/bin/env bash
set -x
REPO="https://github.com/<owner>/<repo>.git"   # <-- set this

echo "===== ENV ====="
date -u; git --version; curl --version | head -1

echo "===== TEST 1: default (HTTP/2) ====="
rm -rf /tmp/h2 && GIT_TRACE=1 GIT_CURL_VERBOSE=1 GIT_TRACE_PACKET=1 \
  timeout 300 git -c http.version=HTTP/2 clone --filter=blob:none --no-checkout \
  "$REPO" /tmp/h2 2>&1 | tail -n 80
echo "TEST 1 exit: $?"

echo "===== TEST 2: forced HTTP/1.1 ====="
rm -rf /tmp/h11 && GIT_TRACE=1 GIT_CURL_VERBOSE=1 GIT_TRACE_PACKET=1 \
  timeout 300 git -c http.version=HTTP/1.1 clone --filter=blob:none --no-checkout \
  "$REPO" /tmp/h11 2>&1 | tail -n 80
echo "TEST 2 exit: $?"

If Test 1 stalls/CANCELs but Test 2 succeeds, that's your confirmation it's the HTTP/2 edge path.

As an Actions workflow step (same test, in CI):

      - name: git transport diagnostics
        run: |
          set -x
          REPO="https://github.com/${{ github.repository }}.git"
          echo "::group::HTTP/2"
          GIT_TRACE=1 GIT_CURL_VERBOSE=1 timeout 300 \
            git -c http.version=HTTP/2 clone --filter=blob:none --no-checkout "$REPO" /tmp/h2 || echo "H2 exit=$?"
          echo "::endgroup::"
          echo "::group::HTTP/1.1"
          GIT_TRACE=1 GIT_CURL_VERBOSE=1 timeout 300 \
            git -c http.version=HTTP/1.1 clone --filter=blob:none --no-checkout "$REPO" /tmp/h11 || echo "H11 exit=$?"
          echo "::endgroup::"

When filing with Support, attach the full output of both groups plus date -u at the start of each, so they can line the timestamps up against their edge metrics.
You must be logged in to vote

0 replies

Ok, so we identified degraded behavior on a network transport path that could increase transfer latency for some requests while leaving others unaffected, which is why reproduction was inconsistent. On May 29, traffic was moved away from the affected path and that 'should' mean this is mitigated. We are also digging into our alerting as to why we didn't autoflag this earlier.

Can folks let me know if you are still seeing issues here.
You must be logged in to vote

3 replies

@ggjulio

Hey, this is occuring again i think.

2h22 to clone a repo of 1 or 2gb. and it is very frequent.
For context this was not occuring with ghe and data residency in europe. What are the solutions ?

Any rapid answers or hints are very welcomed.

edit: i forgot to mention our runners are ec2 instances in europe.
edit 2: I'm using action/checkout. on ~30 builds, about 7 timeouted after 3h of pulling. It seem the build has a 40% chances taking a route that is very slow. When it falls on a good path the checkout only takes 1m30sec .

@RazvanLiviuVarzaru

@ggjulio

@nebuk89 authenticated checkouts were failing at our 3-hour job timeout with transfer speeds between 15 and 1 KB/s.
45% of our builds failed at checkout. The incident span were something like a 13-18 hours overlapped on two different days. (France)

If that doesn't count against the SLO, the SLO isn't measuring what customers actually experience, and the status page didn't reflect reality. Worse, there was no communication at all, so we — like many other customers, presumably — wasted hours troubleshooting a problem that was on GitHub's side.

Incidents are multiplying and clients are looking for alternatives, example (btw probably commented because of the checkout issue):
runs-on/runs-on#483 (comment)

Please consider reflecting this incident in the status history, or at least communicate to all clients when there's ongoing issues.

@nebuk89 not sure if this is related to this but i'm experiencing issues with the actions/checkout GH Actions Job:

Error: fatal: missing blob object '2c050de201eca7738deea0f7ae1e5f7bd670ce97'
Error: error: remote did not send all necessary objects
The process '/usr/bin/git' failed with exit code 1
Waiting 16 seconds before trying again
Run actions/checkout@v6
Syncing repository: cloudeteer/squad-test
Getting Git version info
Temporarily overriding HOME='/tmp/3/_temp/e18799d5-1eaa-48f4-95cb-401aec72fb8c' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /tmp/3/squad-test/squad-test
/usr/bin/git config --local --get remote.origin.url
https://github.com/cloudeteer/squad-test
Removing previously created refs, to avoid conflicts
/usr/bin/git submodule status
Cleaning the repository
Disabling automatic garbage collection
Setting up auth
Fetching the repository
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +218679353f5c448c47905eb7dc3282fa03468a46:refs/remotes/pull/683/merge
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +218679353f5c448c47905eb7dc3282fa03468a46:refs/remotes/pull/683/merge
  Error: fatal: missing blob object '2c050de201eca7738deea0f7ae1e5f7bd670ce97'
  Error: error: remote did not send all necessary objects
  The process '/usr/bin/git' failed with exit code 1
  Waiting 12 seconds before trying again
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +218679353f5c448c47905eb7dc3282fa03468a46:refs/remotes/pull/683/merge
  Error: fatal: missing blob object '2c050de201eca7738deea0f7ae1e5f7bd670ce97'
  Error: error: remote did not send all necessary objects
  Error: The process '/usr/bin/git' failed with exit code 1
You must be logged in to vote

0 replies