GitHub waited 3 months to notify about potential compromise
It seems GitHub became aware of the issue around March 2 (or before, since the fix was released on March 2), and waited until June 16 to disclose the problem.
See full text below:
Hi zwass,
We're writing to let you know that between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC, due to a bug, GitHub Apps were able to generate new scoped installation tokens with elevated permissions. You are an owner of an organization on GitHub with GitHub Apps installed that generated at least one new token during this time period. While we do not have evidence that this bug was maliciously exploited, with our available data, we are not able to determine if a token was generated with elevated permissions.
User privacy and security are essential for maintaining trust, and we want to remain as transparent as possible about events like these. GitHub itself did not experience a compromise or data breach that either created or resulted from this event. Read on for more information.
* What happened? *
GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour.
GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.
GitHub Apps generate scoped installation tokens based on the scopes and permissions granted when the GitHub App is installed into a user account or organization. For example, if a GitHub App requests and is granted read permission to issues during installation, the scoped installation token the App generates to function would have `issues:read` permission.
This bug would potentially allow the above installation to generate a token with `issues:write`, an elevation of permission from the granted `issues:read` permission. The bug did not allow a GitHub App to generate a token with additional scopes that were not already granted, such as `discussions:read` in the above example. The bug did not allow a GitHub App to access any repositories that the App did not already have access to.
In order to exploit this bug, the GitHub App author would need to modify their app's code to request elevated permissions on generated tokens.
* What information was involved? *
The following GitHub Apps generated scoped installation tokens during the bug window for your organization(s). We are not able to determine if a token was generated with elevated permissions.
Organization: GitHub Apps <redacted>
* What GitHub is doing *
GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited.
We are notifying all organization and user account owners that had GitHub Apps installed and had a scoped installation token generated during the bug window so that they can stay informed and perform their own audit of their installed GitHub Apps.
As a followup to this investigation, GitHub is looking at ways to improve our logging to enable more in-depth analysis on scoped token generation and GitHub App permissions in the future.
* What was the potential impact? *
Due to the variety of GitHub Apps, their possible scopes, and the repositories they may have been given access to, we are unable to advise on any potential impacts as each customer's situation will be unique.
* What you can do *
While we have updated our systems to resolve this bug and no action is required on your end, we do recommend you review your installed GitHub Apps. You can use the following guidance for assessing GitHub Apps, their permissions, and their access to your private organization repositories:
https://docs.github.com/en/organizations/keeping-your-organization-secure They explained it themselves - there's no evidence of abuse/exploitation.
They literally have no legal requirement to even tell you as much as they did.
You should be commending them for filling you in at all. > They literally have no legal requirement to even tell you as much as they did. Is ‘fulfilling legal requirements’ all you look for in a business relationship? A restaurant has no legal requirement to make this food tasty but it’s what I’m looking for when choosing where to go. As someone who works in the reputation management sector, fulling legal requirements is crucial in establishing a presence in key markets. However, oversharing of internal information that's not required by legal requirements can lead to unnecessary reputation damage, which would lead to a decrease in value for key stakeholders. I think many engineers often overlook the business implication of disclosing security issues, as it would impact multiple business units as well as the board's stance on security, resource allocation, and potentially the stock price too. >A restaurant has no legal requirement to make this food tasty
Food is a core deliverable for a restaurant, whereas information on a potential breach is not for a SaaS service unless it is legally required. Some people are in tech because they want to build technology ethically and responsibly, not to maximise the stock price at all costs. GH has no evidence this was not exploited. They just didn't log enough things to know if it was exploited or not. > which would lead to a decrease in value for key stakeholders. I couldn't care less. I want value as a customer. Any company that prioritizes stockholders to customers doesn't deserve my customer money. Since almost every popular tech company is a quasi monopoly, they use this "fulfilling legal requirements" strategy to abuse the market providing overpriced services with bad quality. Unfortunately, people got used to this practice and gladly accept when such companies fulfill all their legal obligations, even when this hurt them or their business. How much is GitHub overpricing their bad quality services? With some price transparency into what companies actually pay for services such as GitHub, maybe we can find out. > A restaurant has no legal requirement to make this food tasty Somewhat tangential but I'm not even sure that's entirely true. It gets all sorts of tricky due to the subjectivity, but surely fit-for-purpose laws apply here? I'd be really surprised if a five star restaurant selling $500 tasteless gruel with chunks wouldn't manage to get into trouble if they refused refunds. Dishes similar to described ones could be served in the top "Scandinavian food movement" restaurants [1]. [1] - https://www.nbcnews.com/news/world/ants-cod-liver-moss-denma... > ... we were unable to determine whether this bug was ever exploited. > ... > Due to the variety of GitHub Apps, their possible scopes, and the repositories they may have been given access to, we are unable to advise on any potential impacts as each customer's situation will be unique. Absence of evidence is not evidence of absence. That's true, but feels like these are always judgment calls. We can always armchair quarterback their judgment calls, but none of us have the full info. At least GH is sharing this info, which is a good call for trust building IMO. That's not fully Github's choice to make. They made a judgement call based on seemingly incomplete evidence, and have different incentives that everyone else. Repository owners may well have a different level of acceptable risk or legal obligations over the integrity of their source code. For example, if I was maintaining security software or a popular package, it would be entirely appropriate to stop everything and look for abuse. Waiting three months makes that harder. I'm not sure that's trust building. Fair point - I think that's very fair criticism This is only a judgment call because they have no idea. The fact that they have no idea whether your organization's data was leaked is exactly what people here are complaining about. I would be commending them if this notice went out March 3 after they had remediated the problem and were aware that they had no logs to determine whether there was abuse. This is incorrect you should re-read the post here cause I think you misunderstood the implication. They lacked the logging at the time to know what apps were impacted and the extent to which customers were compromised by this. They are legally obligated to disclose security risks like this which is why they did. You should consider setting a higher bar for your commendations. >They explained it themselves - there's no evidence of abuse/exploitation. They literally have no legal requirement to even tell you as much as they did. You should be commending them for filling you in at all. Do they keep logs, or is that also not a legal requirement? (See how these things can combine?) Folks literally should be mindful no oneis obligated to do business with you if they think you're untrustworthy. They’re obviously capable of doing more, and they’re in a competitive market, so I wholeheartedly support the freedom of dissatisfied customers to publicly shame them. Either they will learn from this and do better in future, or the community will enrich their competitors at Microsoft’s expense. And that is how we can have nice things rather than the mere minimum that our corporate overlords are “legally required” to provide us with! My recent experience with GitHub regarding a security issue was not very positive either.[1] It turned out, unlike two vendors I notified that were affected, they just didn't care. And they didn't bother to even tell me that they didn't care. It's a very edge-case issue in Enterprise SSO, so I wasn't really able to generate any blowback with disclosure either. But if you find an org with just the right setup it blows a huge hole into the SSO product, to the point of making it useless. There also seems to be an asymmetry between the core product and everything else. GitHub Enterprise has issues that aren't even considered UX issues (i.e. notifications showing "3 of 0" notifications if no SAML session exists) that'd warrant bounties if they were in the core product. [1]: https://notes.acuteaura.net/posts/github-enterprise-security... How is tailscale mitigating this? They can’t enforce GitHub SAML at their end, right? The membership API returns a 403 if no SAML session exists. Check the remedy section of the post. This was an accidental find and GitHub has refused to document it. My guess is that GitHub ignores issues so they don't have to pay out bug bounties. I find these takes so silly. Bug bunties are a rounding error in the companies budgets, even if they paid out much more freely. There are many I think much more obvious reasons orgs are slow on issues - everything from figuring what is an issue, trying to chase down impacts and more. I think it's not a matter of not wanting to pay, but not wanting to have your departments "we had to pay someone to fix your security bugs" metric go up. That's also likely why issues in the core product are taken more seriously. Given it existed for 5 days and you’re only now finding out about it, it sounds to me like it was perhaps a bug that was fixed without realising the full impact of it, or perhaps without realising it made it to production; and only an audit that happened later caught it. Not ideal by any means. I’d be curious to know if my theory is correct or not. Their statements indicate they were aware and investigating. My frustration is that they didn't give users the opportunity to do their own timely investigation. > GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour. > GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC. > GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited. I am really curious how SecOps works at GitHub. Why not after remediation inform users of the flaw and potential impact. Then follow up with detailed impact. Instead we get this 3 months later all they can say is "Some of your apps refreshed their tokens during a 5 day period" which is not news... This is also the second time this year there has been significant delay in communication. Granted those involved other third parties so who knows where the delay lived. I would assume the lawyers have to get involved first to write up some document proving that github is not liable for what users do with their app tokens. CYA, then tell the public they might assume that the account holder are a likely perpetrator, which might be true but also enables their intermediation of the context and control of the sequence of communication. I just got it as well and don't understand what I can do. Can I somehow force all generated tokens to be revoked and get apps to generate new tokens to be on the safe side? Or, rather, is there a way to do this without uninstalling the apps and installing them again? >Each of these tokens are valid for up to 1 hour. > GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC. It doesn't sound like there is anything you can or need to do with respect to these tokens (whether they were used to take action with elevated permissions is a different thing, but it doesn't sound like that was the case either) It seems like the appropriate thing to do would be to inform anyone who had tokens created during the affected time period, so they could assess if any of the permissions led to undesired changes. Instead of GitHub saying “we don’t have hard proof of anything bad happening” and waiting 3 months, just give the customer the time of relevant token creations. The email listed the apps that issued tokens in the specified time window. If your notification email listed 0 apps, it means no apps created tokens during the time window in question (I got 1 app listed). I only missed that those tokens had 1h lifetime. Listing token times instead of just apps, and doing so nearer the actual time instead of 3 months later, are the key parts of my suggestion. Aren't these very time limited anyways (hours vs days)? So once they fix the bug and wait an hour the old tokens are gone anyways? Per the email: > Each of these tokens are valid for up to 1 hour. So these tokens would've been expired for 3 months now, according to when the fix was deployed. You could revoke the pem that generated these tokens, but as others have pointed out the tokens in question are only valid for 60 minutes at maximum. Can somebody tell me if I'm wrong on my take but this bug/issue means: - a github app which had read permission on issues could elevate its permission to write - a github app which had read permissions to discussions could elevate its permissions to write. So far if the org/user would have been compromise they would have seen with issues or conversations containing content from the app. Since these are only examples, I can imagine the case with major impact would be a contents:read elevate to content write. But again with commit signing, this would also be caught by the user.
What did I miss where the impact would have not been visible to the end user/org ? contents:read to contents:write is a big deal! Just to pick out a random widely used project, nodejs [1] has a number of unsigned commits to the main branch. Their commits could have been tampered with during this timeframe. What about release artifacts? I guess I can see it, but branch protection rules and pull requests reviews would also prevent that to happen in my opinion (also ability to do it with content:write is just speculation from my side, they don't make it clear if it is possible, that would need to be confirmed by github) In the same minute that I learned GitHub had been acquired by Microsoft, I cancelled my pro subscription and began moving my critical repositories elsewhere. I'm old enough to remember the MS that tried to choke the life out of GNU/Linux and spread FUD about all FLOSS, the one that engaged in anti-competetive behavior during the "Browser Wars". I'm not suggesting that this blunder of a delay is related to the Microsoft acquisition, but the abusive "Look at me, I'm changing" spiel never cut any mustard with me. Vote at the ballot box and with your dollars. Do not reward executives, lawyers, or engineers who dissemble or obfuscate. memory of that stuff is just never ever going to die, is it? yet the same people use google, and facebook, and AWS, like those companies sit upon moral high ground. they do not. Linux succeeded and defeated Microsoft in every single way that matters to open source people, and the response is to continue to hate Microsoft for their loss? I do not understand. Just admit your motivation for saying things like this: you hate Microsoft because Slashdot told you to, or tells you to, and you want to let people know about that, unprompted. Nothing about Microsoft's behavior in the 1990s has any bearing on what GitHub does today. Microsoft is still doing this anti-competitive things with web browsers today. Every so often, my Windows 10 machine switches back to Edge, or adds Edge back to the task bar, or prompts me to try out Edge, "the recommended browser for Windows". And no one is claiming that Google, Facebook, and Amazon are better. Those are not the examples I would pick to show Microsoft could do better, for sure. This is whataboutism. > yet the same people use google, and facebook, and AWS, Speak for yourself. For me, the lesson I learned while growing up with the MS of the 90's was to not trust any big corporation. The power imbalance is too large, individuals have no way to protect themselves and they will take any and every opportunity to exploit that. This pattern can be seen with in the 90's with Windows, it can be seen today with Github and LinkedIn (talk with recruiters and they will tell you how MS is jacking up the prices and removing functionality) and it can be seen with any of Big Tech in the last 20 years. Seems like you’re being just as hostile in the other direction. If this isn’t the appropriate time to complain about Microsoft, then what is? I can’t speak for everyone, but I continue to not like Microsoft because they don’t make a single piece of software that I like, and they’re ruined the ones I did like. I am just dead tired of Microsoft getting zero credit for the changes they've made since the antitrust verdict. they haven't done EVERYTHING that EVERYONE wants, so lots and lots and lots of people still shit on them like they're still mad, and it still gives nerd cred to shit on them for any reason. as if we're perfect in comparison... if the same people speak poorly of Facebook, Google, or Amazon in the same ways, the reactions observed are very different. Very different. People pick on Microsoft because they once earned it, yet we let so many worse things slide, today, because those things are done by companies which are not Microsoft. flippin' pick an opinion, and stick to it. Happily picking an opinion: I won't use phone apps; I won't buy new computers or devices, though I infrequently buy refurbished; I won't buy new cars; I carefully vet all vendors for quality, politics, and ethics (and have largely automated the process); I won't buy new kayaks; I eat a plant-based diet and try to source it locally/ethically; I won't knowingly vote for or endorse rapists or murderers and do my best to vet candidates. I abhor Google, loathe Facebook, have considerable contempt for Musk (as a person) and the products his companies offer, use Amazon sparingly and grudgingly, and only started shopping at Walmart during the Great Recession when a) they became the employer of the majority of my neighbors and b) other vendors in my area closed. @naikrovek may not now be aware of what happens when one makes assumptions about other people and their actions, beliefs, etc. > Linux succeeded and defeated Microsoft in every single way that matters to open source people I don't know, hardware still regularly does not support Linux, the Linux desktop has a risible fraction of the world's user base, popular apps still only exist for winmac. It won on the server, sure, but that's hardly every single way that matters, at least from what I remember as an open source user in the '00s. About 80% of the smartphone market too.. Android is Linux to about the same degree that Chrome is Windows. Android uses the Linux kernel, which, to old-heads like me, is Linux, whereas the combination of userland and kernel was GNU/Linux. ChromeOS also uses the Linux kernel. If you are talking about Chrome-the-web-browser, well, then the parent comment is nonsense. > yet the same people use google, and facebook, and AWS That's a weird assumption to make, and definitely incorrect in at least some instances. > Just admit your motivation for saying things like this: you hate Microsoft because Slashdot told you to ... Huh? This seems like a really bizarre path to be going down. Not sure how you got there. :/ > Linux succeeded and defeated Microsoft in every single way that matters to open source people So when someone gets a job where they're required to use Windows and spend all day in IE11 and Microsoft Office, let's say at a very large semiconductor company, they're hallucinating? The fact that there exists people who don't get to use X is not proof that X is not the most common, or that they are hallucinating. I think I got whiplash from this episode of HNers Moving the Goalposts. Yet you moved the goal post from "Linux defeated Microsoft in open source" to "there isn't anyone using Windows at all". Or are we all hallucinating your comment? Try undertaking to recognize the actual premises of ∀x-style statements and ∃x-style statements that you encounter in the wild, scooter. The goalpost was set at "Linux succeeded and defeated Microsoft in every single way that matters to open source people". Ignore the record of what was actually claimed if you want (the quotes you're throwing around are works of your imagination), but that is not merely a claim that either Windows or Linux is "the most common". The observation that that there exists some X where P does not hold is precisely the way to counter a claim that for all X, P is true. First let me say that talking with you is really annoying, way worse than anyone else I've interacted with on this site. Removing the "must be hallucinating", "try undertaking to recognize", "scooter" would turn your comment more into thoughtful discourse than sneer (you might want to check out the HN guidelines). I am trying to politely argue, pointed out that your comment didn't make sense to me hoping for a clarification, and got shit on, twice. Nice. To attempt to answer you (I might regret this): what did you take "every single way that matters to open source people" to mean? By gazing through your insults into your comment, trying to find the "∀x statement" you think you saw, I have to assume that you think that means "every one developing open source code isn't using Microsoft?" Is your made-up acquaintance working at a semiconductor company primarily developing open source software there (rather than, say, semiconductors)? Is the "∀x statement" about "x: way that matter"? Am I to take "non-free software is being used for making semiconductor at a large company" as an example of something that should matter to open source people? You seem so convinced that there is a logic statement written right there with quantifiers and everything that you don't hesitate to use ridicule on complete strangers and question their grasp of logic. It's funny but couldn't really count as an argument (assuming you were trying to argue rather than just get some bile out). > First let me say that talking with you is really annoying, way worse than anyone else I've interacted with on this site. This is not an uncommon reaction from people who are accustomed to bullshitting their way into arguments and who typically "win" those arguments by being just enough of a nuisance—and where the stakes are just low enough—that the likeliest outcome tends to be that the other party decides to move on rather than exhaustively refuting the bullshit. When you find people doing this with you, you are not "winning". In fact, that it happens is a consequence of how annoying others find it to interact with you. An example (of someone who was similarly incredulous that everyone didn't just let the lame contrarian quips go unremarked upon—and of the sort of company you're in): <https://news.ycombinator.com/item?id=27906289> > what did you take "every single way that matters to open source people" to mean?[...] I have to assume that you think that means "every one developing open source code isn't using Microsoft?" No, you don't have to assume that. The only possible reason to assume something so self-serving is because you refuse to engage in a good faith resolution. On the issue that is under discussion, there's no ambiguity here—at all. The original statement—which you distorted twice—is a for all X statement: "Linux succeeded and defeated Microsoft in every single way that matters to open source people": For all X, where X is some thing that "matters to open source people", the condition C is true, where C is the claim that "Linux succeeded and defeated Microsoft [on those Xs]". The fact that you are unable to parse this out of a very straightforward passage like the one that appears in the part of the comment I quoted, but that you _are_ able to just, like, make some shit up about where the goalposts were set suggests very much that you are not making any attempt whatsoever to actually understand the issue, and you are just in the business here of issuing low-effort quips (like your first two here[1][2]) that don't actually track the discussion. But setting that aside, let's move on to the actual issue of the claim. The following scenario results in a contradiction of the statement that I responded to: It is 1997. Ned is working for a very large semiconductor company. At Ned's company, they are required to use Microsoft products. Ned instead wants to use Linux. He is an open source people, and this is what matters to him. Now, it is 2022. Ned is still working for that very large semiconductor company where they are still required to use Microsoft products—he's not allowed to use Linux. This still matters to him. It is therefore undeniably refuted via proof by contradiction; the statement that "Linux succeeded and defeated Microsoft in every single way that matters to open source people" is false. Having now dealt with that, let's read back the transcript and look at your multiple attempts to move the goalposts, e.g. from "every single way that matters to open source people", to something that you plucked out of the air entirely—that is, whether one of Windows/Linux is merely "the most common". That is absolutely _not_ where the goalposts were set, and that's just the first instance of distortion. The second is where you go on to manufacture a quote: '"there isn't anyone using Windows at all"'—which appears nowhere except in your comment where you present it as a quote. Not only is this bullshit, it is the kind of bullshit that is against the rules here on HN. Do not do this kind of thing. And certainly don't do this sort of thing while making a big deal about how annoyed _you_ found yourself over the course of carrying out this (entirely avoidable! excruciating!) back-and-forth that you alone were responsible for foisting onto the discussion. No. It won't die. In the same way that memory of the Holocaust won't die (hopefully), or that memory of Putin's invasion of Ukraine won't die (hopefully), or that the memory of the corruption of Donald Trump won't die (hopefully), or the memory of the enslavement of Black human beings in the US won't die (hopefully), or the memory of corrupt policing won't die (hopefully). “Those who cannot remember the past are condemned to repeat it.” – George Santayana You have no idea of what tools I or anyone else on here use or don't use. You only know that I use git, which I could be using independently of everything. I could be using git to track my personal thoughts every day and nothing more. I make an effort to source everything I use as ethically as possible, though it is ridiculously difficult in the world of modern technology. Microsoft bundling a browser is not the same as the Holocaust. Not the same magnitude, not the same offense, not the same violation of any moral code, at any magnitude. No one died because Microsoft included IE with Windows. no one was enslaved because Microsoft included IE with their OS. that you think these are all events of the same magnitude has completely invalidated your opinion, and placed you firmly in crackpot territory. I'm not at all saying that they are equivalent, just that they should not be forgotten. it's all about the lesser of multiple evils. Like voting for president What are you using now? I've only heard of GNU Savannah. try gitlab or sourcehut Or Codeberg, or the self-host parent project, Gitea, or self-hosted cgit That’s a pretty neoliberal attitude to effecting systemic change, do you think that’s enough to mitigate those behaviors considering they crop up elsewhere and ongoing If refusing to do business with a company whose business practices you don't like is "neoliberal" then the term has officially been stripped of any meaning, significance, or usefulness. Thank you for your input but I am not commenting on the decision to stop purchasing. I'm responding specifically to the final demand: "Vote at the ballot box and with your dollars." The OP is encouraging us to take action against this behavior specifically via choosing how they allocate their dollars as consumers (and also voting), which is core to the neoliberal mission. They're not just saying it's worthwhile to stop purchasing, they're saying that this is the solution "Voting with your dollars" is not only a neoliberal attitude, it's also a liberal attitude, but moreover it is also very common for leftists. Unless you're buying the rope from capitalists to hang them, realizing how power flows in our current economic system is essential for all. Waiting for bad corporate behavior to punish individual actors for individual behaviors with dollars requires such an extreme level of widespread activity that you might as well be talking about something more radical and mass-organized than promoting care over simple purchasing decisions, to be meaningful at a systemic scale. These negative behaviors are a feature of capitalism not aberrant fringe behavior to course correct. It’s fine but leaving such a strong command to, paraphrasing, “fix this world by choosing between democrat and republican, and through carefully considering your purchases” (not even a clear call for organized boycott, just the individualized ethics) does not satisfy me I only care about the individualized ethics. You do you. Neoliberal has always been an meaningless slur. It was also my experience that it takes GitHub several months to fix issues and inform compromised users. Does anyone know what type of audit logs or actions we need to look for? Yep. As I mentioned elsewhere on the pyright/pylance issue: Microsoft are scumbags.