I am encountering the same thing when trying to browse through code in repositories. Gods forbid you cannot find what you're looking for in the first few clicks or you get locked out. If I were a wikipedia mod, this new "feature" of locking users out of open source code, unless they have an account and log in, would be the textbook example of Enshittification, and your logo would be placed right next to that handsome photo of Cory Doctorow. Although, this could also be a great r/LeopardsAteMyFace moment, as it was likely implemented to mitigate the AI bot hellscape you helped unleash on the net. Please remove this dark-pattern nonsense as it is an insult to the open source ethos you had at one point claimed to support. If you are reading this and a developer or maintainer of an open source project, I highly encourage you to move your project to a less predatory and more welcoming platform like Codeberg.
5 replies
If you are reading this and a developer or maintainer of an open source project, I highly encourage you to move your project to a less predatory and more welcoming platform like Codeberg.
Unfortunately, I see it on the horizon that this will be a common trope amongst every service that has expensive (in terms of CPU time) API calls to render necessary functionality... AI training bots are nasty.
The point which explains why the AI training bots then won't crawl logged in to burner accounts is missing. Currently it looks just like we throw away usability because it is easy not because it helps in any way.
Unfortunately, I see it on the horizon that this will be a common trope amongst every service that has expensive (in terms of CPU time) API calls to render necessary functionality...
Which can be solved by making it CPU-intensive for the scrapers. Though FOSS-adjacent metadata could be valuable enough that they would try to scrape it anyway. I’m trying to but failing to see a way to interpret Microsoft’s actions here other than just a big middle finger to anybody who is still hosting their projects on GitHub.
Whoa there! I just came across this. Please don't do this. As comment above says. We have users of our project who don't have GH accounts who would like to see more than three files in our repo.
It's on HN - https://news.ycombinator.com/item?id=43981344
0 replies
Limit seems to be way higher now, took me around 20 attempts to reproduce the issue.
0 replies
2 replies
this blog post talks about using the REST API and downloading raw files, not about navigating the website in the browser.
0 replies
This comment was marked as disruptive content.
This comment was marked as disruptive content.
How does this affect Golang code when using "go get" or "go mod download"? Sounds to me like it might fail when fetching dependencies.
3 replies
why would go get and go mod download be hitting github.com APIs?
e.g. when using GOPROXY=direct.
I wonder how GitHub handles Google's Go module mirrors (i.e. the default GOPROXY), given that SourceHut reported that they're quite aggressive in their fetching.
@Kryan90 By default, requests go to through the module mirror hosted by Google. This proxy does not work (well) from everywhere. Also, there are reasons against using the proxy (e.g. data protection).
Even when using the proxy, this proxy will be sending a lot of requests. We can only assume Google and Microsoft have some kind of deal.
This comment was marked as spam.
This comment was marked as spam.
This does not address the issue that was raised. This issue is about blocking end users from browsing via a web browser using the web interface. Nobody is using curl, nobody is trying to clone, nobody is trying to use the API here. This is like getting blocked at the library because you are turning the pages of a book too fast. Saying you now have to create an account and log in to view something that was freely created and given to the world is a betrayal of the social contract and the very community that built this platform and is a 180 degree turn to the intentions GH once claimed to have, hence my strongly worded admonishment above. A classic "bait and switch" if you will.
Hey folks 👋 - Martin from GitHub here. Thanks for reporting the issues. As a few folks mentioned, this was an unintended consequence of throttling limits we've had to put in due to the amount of anonymous scraping activity we've been seeing. However, it's not our intention to limit legitimate human browsing of open source code on GitHub, or adversely restrict legitimate research and archiving activities. But sorry for the issues you've ran into. As of 7:00 UTC this morning we've adjusted the throttling logic and I'm seeing a lot less requests getting caught. Hoping this has stopped the problem for most folks now, but we'll keep monitoring until we've found the right balance.
15 replies
It seems like bad faith to suggest there's something wrong with GitHub just because it got bought by Microsoft. Pre-acquisition GitHub was extremely unstable and lacked many features like reviews, since then they have become quite robust, and more importantly so popular for OSS as to change the face of computing forever, single-handedly. Seems like a pretty good contribution to me. Now GitHub, still mostly Ruby-on-Rails as far as I know, is trying to manage their load and may have had some issues but I'm sure are working diligently to help their users. What AI teams in a huge company like Microsoft are doing many buildings or even regions away doesn't seem to have much to do with them.
Can't we respect GitHub's contributions and treat them with good faith instead of being so negative automatically just because of the "M word"?
What AI teams in a huge company like Microsoft are doing many buildings or even regions away doesn't seem to have much to do with them.
Can't we respect GitHub's contributions and treat them with good faith instead of being so negative automatically just because of the "M word"?
Good grief. I must have simply imagined that GitHub specifically, was forcibly cramming some "Copilot" AI product down the throats of OSS users recently.
Thanks for pointing out that GitHub isn't involved in AI.
It seems like bad faith to suggest there's something wrong with GitHub just because it got bought by Microsoft.
Call it experience.
Pre-acquisition GitHub was extremely unstable and lacked many features
Does not match my recollection.
What AI teams in a huge company like Microsoft are doing many buildings or even regions away doesn't seem to have much to do with them.
I strongly disagree. If several departments of your business are effectively at an arms race against each other, you are doing a poor job running your business. And I will make sure mine does not depend on how well you do.
Can't we respect GitHub's contributions and treat them with good faith instead of being so negative automatically just because of the "M word"?
I did. For years. To this day I keep engaging with projects on GitHub and I am not advocating against them. When they made the move to use 'our' code for training Copilot with no regard for the licensing terms, I thought 'the storm is coming'. And now look where we ended up. A proper solution would be to re-evalutate if the cost of training LLMs without consent are worth the benefits.
Long before chatbots, Wikipedia was a place that 'gathered all knowledge of the internet' and people contributed. I am sure many programmers would be fine with their code being used in LLMs. You could get chatbot time in exchange for checking in your code with them. Then they wouldn't need to scrape. The core problem here is that the FOSS comunity is not considered when big tech makes a decision. And Microsoft promised differently. So now we call them out. And if they don't listen, this platform will loose relevance, like many others have before.
Instead of receiving 429 we are now receiving 403 errors. We are authenticated via token.
I am currently getting rate-limited after responsibly running a manual python script with a token that first uses PyGithub to check for the right list of a dozen or so commits (this works) and then using requests with the same token to fetch github.com/USER/REPO/commit/SHA1.patch (this is being rate limited).
I cannot get even a SINGLE request through, to commit/.
api.github.com works fine. What is the logic in this? What are we "expected" to do?
2 replies
I suppose you could use git directly, but that’s harder to implement. It also would probably place more burden on GitHub’s servers, not less, but it’s harder to block without breaking unauthenticated clones altogether, so it might work for you for a while.
I'm sure there's lots of things I could do to be more of a burden on Microsoft resources while evading their ChatGPT throttling, but there's only so much interest or energy I have to constantly rewrite my project's tooling to add more and more elaborate state tracking. :)
It is probably practical to do it anyway. But on the other hand probably the most practical thing to do would be to find a non-github forge that is run with an interest in supporting FOSS communities.
Having this issue myself as of the last 2 days. only it happens to me when doing http requests with curl to anything github related. Changing ips via a vpn temp solves it, but doing one request every couple minutes to get rate limited seems extreme.
0 replies
I was having this issue earlier. I was downloading about 10 stl files from a 3d printing repo and hadn’t bothered to log in. In my case I think there needs to be some work done. There was no messaging on the site to indicate an issue, I just stared getting 0B downloads with the appropriate name for the file I had requested to download. Only realized what was happening after opening the browser console and seeing the 429 coming back. I don’t especially mind a rate limiting approach (within reason) but definitely there should be an actual message of some sort instead of just acting as if it’s downloading and giving a 0B file.
0 replies
There's many scripts floating around that install things, and most of them use unauthenticated HTTPS to pull down binaries or the scripts themselves. For some examples, check out Atuin's and Babashka's. With these new rate limits I'm getting 429s while using my Ansible scripts on my personal desktop machines. I could add in authentication for the requests I manage, but the scripts themselves often run their own HTTPS requests. Is the expected outcome of this for all these third party scripts to manage GitHub authentication?
In addition, projects such as Linux GSM make extensive use of unauthenticated HTTPS requests to GitHub. This will also require extensive work from the community to handle authentication. Even if that work is done it adds significant friction to usage.
3 replies
Also, nobody should be forced to even have a Microsoft account to access FOSS code in the first place.
I understand why Microsoft wishes to add limits as the AI scrapers are a drain on resources. The problem is really that the unauthenticated limits are extremely low and hamper even light legitimate usage. I have a couple machines that I just tried to update by using my Ansible playbooks to run a few of the above named install scripts (Atuin, Babashka, and also Clojure). I run them occasionally, one machine at a time, and the second machine already hit the limit. Since I have a static IP, my entire house is now blocked from unauthenticated GitHub requests for the next hour.
I understand why Microsoft wishes to add limits as the AI scrapers are a drain on resources.
Remind me whose AI it is?
I've been getting the same 429 Too Many Requests errors when updating from GitHub based APT repos.
0 replies
Getting the same error on Github Actions. Any solution?
0 replies
Is there any explaining how to avoid 429 and "Failed to resolve action download info. Error: Internal Server Error" errors in github actions?
Is it related to github yellow status?
0 replies
Dear GitHub Team,
While I understand the need to mitigate scraping, the current rate limiting on raw.githubusercontent.com and https://api.github.com/ is disrupting legitimate open-source workflows and automated scripts. A more nuanced approach is required to balance security with open access. Specifically, smarter throttling techniques and exempting GitHub Actions (including self-hosted runners) would greatly alleviate these concerns and reaffirm GitHub's commitment to open source.
Thank you for your consideration.
0 replies
0 replies
GitHub releases is now completely useless due to rate limits for me, I have a project with an installer that auto downloads from one of my repos latest releases. Most my users cannot use my project anymore due do being rate limited simply just from downloading the installer and opening it. I'm having to tell them to wait or manually install stuff now, or use a VPN to change IPs and try again.
0 replies
I'm seeing this issue when searching for a public repository without logging in. I actually am unable to use Github at all, the very first search runs into this "rate limiting". This throttling is essentially making Github useless on the browser for non logged in users.
0 replies
I also stumbled upon this when searching for a repository via github.com, without being logged in.
I literally entered github.com in my URL bar, clicked the search, and entered the name of a repository. Nothing else, no detours. Boom, immediate 429. If a single search is already too much, just removing the search bar would be a simpler solution...
0 replies
Copy and RAW does not work and both fail with 429!
0 replies
0 replies
I wonder how openAI is bypassing this rate limit;
I'm running my actions inside a self-hosted runner, and I can't push more than 4 or 5 releases a day
1 reply
Do you think after Microsoft bought GitHub for $7.5 billion they add such a rate limit feature to harm OpenAI, a for-profit organisation that Microsoft invested even more money in so that now they own more stakes than any other stakeholder, including the non-profit OpenAI Foundation itself? If anything, this is meant to given OpenAI and advantage over their competitors. And forcing even more open source developers to sign up (and in!) to GitHub so openAI can get their usage data as well (and 'learn' from their search patterns in the context of their recent/future comment/commits) might be a nice little bonus. 😉
At this point may just move to raw pastebin again, at least one can put their code text there and people can access and copy if they so wish without getting 429 threw on their faces.
0 replies
I’m seeing the same “temporary” 429 that somehow lasts 24+ hours and only affects logged-out /blob/ browsing. Meanwhile, /raw/, /blame/, /commits/, .patch, and .diff still work fine without authentication. Logging in fixes it instantly, so this looks less like abuse prevention and more like anonymous users getting caught in IP reputation/CGNAT collateral damage. Brilliant anti-scraping design: punish normal humans while barely slowing anything else down. If GitHub keeps enshittifying basic public browsing like this, people are obviously going to start switching to other platforms.
0 replies
Me too, I'm the only one looking at GitHub at home.
I just tried to look at the example of the library, but it spits out the 429 error. It's solved by turning off Wi-Fi or reconnecting the Internet, but it's uncomfortable doing this every time.
User/reponame/blob/master/examples/example.html
0 replies
2 replies
@EiJackGH Please stop tagging me in these spam posts. One more from you and you are reported as spam and blocked forever.
Stupid question: couldn't they just block Microsoft IPs?
like with https://www.iblocklist.com/list?list=xshktygkujudfnjfioro
or maybe there needs to be a block list specific to AI training bots.
Couldn't they also make it against the EULA (to train an AI off the site)?
Also, send them, Microsoft or whoever, a bill just for the bandwidth used (not legally extending to the value of the stolen data).
3 replies
@AenAllAin: Why would they do this? Microsoft owns Github. They want you to log in so they can track your behaviour across sessions so they can feed their AI with it.
@AenAllAin It's not just MS. It's tons of companies, some of who are going to great steps to hide themselves, which makes IP blocking very difficult.
And you can put what you want in the EULA, many will ignore it. See the various law suits at the moment.
As for billing them, first you have to work out who they are and then they will probably ignore you anyway. Some people have got money back, but very rare.
https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/
Hmm ...well that does make blocking them a little difficult!
I guess that leaves us with migrating to a new GitHub not owned by MS
...and/or filling their GitHub with shitty forks to ruin their training data.
Hello,
The behavior you’re experiencing is expected and is related to GitHub’s rate limiting for unauthenticated users. When you access repositories without logging in, GitHub applies stricter limits on the number of requests that can be made within a short period of time. After exceeding that threshold, an HTTP 429 (Too Many Requests) response is returned temporarily.
This is why the issue appears after a few repeated file views and persists across different repositories — it is tied to your session/IP rather than a specific project.
How to resolve or avoid this:
Log in to your GitHub account — authenticated users have significantly higher rate limits
Wait a few minutes before retrying, as the limit resets automatically
Avoid rapid repeated requests when browsing without authentication
Regarding your feedback:
You’re correct that the current error message provides limited guidance. Improving user-facing messaging for rate limit scenarios would help clarify what’s happening and how to proceed.
In summary, this is not a bug in repository access but a result of GitHub’s rate limiting policy for unauthenticated usage.
Straight truth:
This isn’t something GitHub will “fix” — it’s intentional
The only real solution is log in or slow down requests
2 replies
@DurgaVaraPrasadBandi: Are you speaking on behalf of Microsoft / Github or are jsut sharing your suspicions as 'facts'? No offense - I largely agree with your assessment but I would be curious to have an official source that this is indeed company policy. As for the 'fix' - I guess leaving Github behind and migrating to other forges will be the best solution long-term. Git is distributed by design, there is no need for a centralized upstream in principle. And maybe distributing the 'main' repo copy on different sites, ideally with a common way to login/communicated (think fediverse) could give us all the advamtages of Github without the lock-in effect and as a side-effect distribute the (apparently unavoidable) bot-load across many more shoulders.
I still think the answer is terrible code...
everyone should commit branches of anti-patterns and examples of "what not to do" along with their regular code.
You can even use AI to help, "Computer, rewrite my code in the form of a Shakespearean play."
For those who don't know that is proper LLM/AI prompt etiquette: to start every sentence by addressing it as "Computer," like this is Star Trek.
...together we help AI reach its full potential!