Heroku CI and Review App Secrets Compromised
Just got an email from Salesforce: "Action Required: Heroku security notification".
Looks like the database that stores pipeline-level config variables for both Review Apps and Heroku CI were compromised.
Per Heroku, "...any secrets you set in Review Apps and Heroku CI config may have been compromised and should be rotated".
This...is really messed up :/ Text of the email: At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business, and we take the protection of your data very seriously. We value transparency and wanted to notify you of an issue affecting your account. Based on current progress, we plan to complete our investigation by May 30, 2022. We are continuing with remediation activities and plan to publish additional information about the incident once it’s resolved. As reported on status.heroku.com, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. On that same day, the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. This was identified on May 16, 2022, after further forensic investigation. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022. As a result, any secrets you set in Review Apps and Heroku CI config vars may have been compromised and should be rotated. In addition, any Heroku tokens stored in these pipeline config vars would potentially have allowed access to your Heroku account between April 7, 2022 and May 5, 2022, when your passwords were reset, invalidating all Heroku tokens as a result. Please note, these pipeline-level config vars are different from standard app config vars. App config vars were not stored in this database and we have no evidence to suggest app config vars were compromised. > At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business [...] Hey Bob, why didn't you tell your customers a month ago to rotate their creds just to be safe? This is flat out insulting. What's more - the public status page of this security incident (https://status.heroku.com/incidents/2413) doesn't mention that these secrets were compromised. They chose to send this notification privately instead. But… “We value transparency…” Give me strength. The true masterstroke though is shutting down Heroku so that the negative press of this doesn't affect it. "What is dead may never die!" > At salesforce.com, inc., trust is our #1 value Their legal pages[1] are filled to the brim with those ridiculous statements. I never understood why they'd even bother making it sound nice, especially not for B2B. Customers won't trust the message and likely can't use them in court, and they themselves must surely know they're creating expectations that they can't guarantee to meet. Regarding values I like to ask myself if another company would defend the opposite for smelling emptiness. Reminds me of Jakob Nielsen's rule for writing a good "About" page. If you can insert a "not" into a sentence and get something that no other company would ever put on their own About page, the sentence is worthless. Is Salesforce potentially in violation of EU law regarding data breach notifications? It seems like they either knew the scope of the breach was likely to be much bigger (based on the fact that the investigation was ongoing) or flat out had evidence that it was already. But that said, I don’t know how that all works. So I’m genuinely curious if there’s a possibility this is illicit. I spent the last two days migrating my company to Render from Heroku, and now I'm glad I did. Render is a little rough around the edges; Heroku is far more polished. But it's probably to Render's credit that, in my opinion, the most annoying thing about Render is that it's impossible to google about Render because "render" is such a common word in the tech world! Their support is good and responsive, and the developer experience was good enough. It has some warts, and there were definitely times I missed Heroku, but their speed of improvement gives me confidence in their future. Sad to leave Heroku after almost a decade with them. They were far ahead of their time. (Render founder) Thank you for the support. You might have shared this with our support engineers already but I'd love to hear about what you missed from Heroku (email in profile). We're building Render for the very long term and every bit of feedback helps, even if we can't get to it right away. The bit about the unfortunate name of the product should be taken more seriously. I understand you got that sexy .com domain, and you are here for the long run, but it is currently doing a disservice. My 2c :) Not sure I agree with this, hear me out please :-) People who don’t know about Render won’t be googling “render” - instead they’ll be googling something else (along the lines of what render offers), and then perhaps discover render in the results. And of course people who know about Render won’t ever need to google it, because of that “sexy .com” :-) Edit: perhaps you meant googling about Render’s features/docs/how-tos? Granted this might be trickier! I'm almost certain they were talking about googling docs etc as mentioned in your edit. That would definitely be my concern. I agree with you, that's what they meant. Just like Go is referred to as golang by the community, maybe render can be referred to as rendercloud. I'm not a power user but including the string "render.com" instead of "render" seems to help. I noticed that because I saw some youtube vid titles referring to it that way. (Not parent) I started my migration last night. Overall, great experience! The heroku addon worked great. I was very impressed that the build succeeded and I had my site working in a very short time, with sendgrid and all that working. Only minor misstep for me was when pg_restore didn't work because by default Access Control has no entries and the migration doc didn't mention having to add one. The much more significant issue for me is that I honestly have no real clue what to make of Jobs. In Heroku I use Scheduler to run rake tasks. And in Render there's an API Explorer (and my rake tasks fail when I attempt to run them through that) and then I'm supposed to add crons to... my repo (?), and the Jobs I create go... somewhere. I am very confused. I've read the Jobs and Cron Jobs docs like 40 times. Good to hear you liked Render, and sorry for the confusion between Jobs and Cron Jobs (we have an open task to improve docs for both). Heroku Scheduler = Cron Jobs on Render. Would you mind emailing me (see profile) or support@render.com with details on your Rake tasks so we can take a look? I emailed support and Alan was super helpful. I think I'm all set (or at least close). For me, the key was discovering that while both my native and docker builds fail, my Dockerfile.render builds (which use the heroku buildpack) magically work AND I could use that Dockerfile.render build for the Cron Job (I don't think I would have figured out that I could just plop in that Dockerfile Path in the Advanced section). I have no experience with Docker (someone set up docker-compose stuff for us like 4 years ago as a student project, but I haven't used it). So it was a little overwhelming for me when the Render migration tool thrust Docker on me. Still testing out some things before configuring my DNS. I'm sure in a week you'll never hear from me again (because things will just work). I enjoy how snappy the Render interface is. Maybe more of a walkthrough (with screenshots) on the cron jobs doc? We should definitely add more tutorial/walkthrough type docs. I haven't had any issues finding docs via Google e.g. "render postgres" or whatever specific thing I'm looking for It's possible my google search is now biased because I've clicked on links for render? That's fair. The current workaround is to use the search in our docs, or add 'render.com' to Google queries. Of course, I'd love examples of frustrating searches. I love the look of everything about Render, except its native lack of PHP support. I really don't want to have to use Docker. Do you have any ETA when that will be available? No immediate native PHP plans, but we're actively hiring to move faster on all axes. I don't use Heroku, nor Render, and I definitely think anyone using Heroku should be moving out, but... Do you have any evidence Render actually takes security seriously? Not shitting on their platform, I actually never used it, I just think as an industry we should be way past the point we trust platforms by default. I asked about this too. Everyone meme'ing these alt platforms essentially assumes they are safer than Heroku by virtue of the fact that Heroku had a pretty severe incident. I haven't actually seen these platforms prove that they're safer than Heroku, they could be as bad or worse in security. I wouldn't move off of Heroku because of the incident. I would move off of them because of their response to the incident. They plainly lied. Responses take weeks and are very incomplete. They have so few people they can't possibly run a secure, stable system anymore. They don't have a plan or backing from sfdc to get back to a solid foundation. I can't speak to competitors but I can say with certainty that Heroku is simply not an option for you anymore. Whether that means you use another PaaS or fire up an EC2 instance yourself you must move away at this point. I had to switch from Render to Heroku a year or so ago because Render had no security documentation at all. I asked them about it at the time and was told security docs were perhaps six months out. There's still none, so it's clear that demonstrating security is not something that's a priority. I agree we haven't focused on demonstrating security, but obviously, internally, we are quite paranoid about it. Still, this comment is well-deserved because as much as we'd like them to, our customers can't simply read our minds. Update: we now have https://render.com/security > I just think as an industry we should be way past the point we trust platforms by default. That's a great point and I fully agree. I'm struggling to come up with reliable ways of checking security of the companies I'm not familiar with. It's not like I can rely on their landing page. And they are likely not on the market long enough to see how they responded to past security incidents. The only thing I can think of is checking how they handle registration and logins - but it's not that strong of a signal anyway. Does anyone have other ideas? (Render founder) Security and uptime are the only two existential threats to Render, so you can imagine we lose enough sleep over it. We regularly fill out security questionnaires that lead to successful migrations. Still, we can do a better job explaining and documenting our security posture publicly. We just started planning our transition, as well. This was handled so poorly I can't imagine anyone would ever plan to start a new project on their platform. What happened? Oh, sorry, I thought the comment was about how the transition was handled poorly. I’m well aware of the slow-motion train crash that is the Heroku incident! How did the pricing compare? What was the Render server needed to match what you were using with the Heroku Dyno? I run a high-traffic nonprofit edtech app that runs on 3-5 Heroku L Dynos, and I'm curious how well Render will perform at this level. Off the top of my head, I'd say Render is a little more expensive at the lower tiers and less expensive at the higher tiers. I also intend to deploy some new services on render (having previously used Heroku). I was debating between render & fly, which I've also had my eye on and may still try for something else in the future. I think Fly will be great, but it's not there yet. I migrated to Render. I am considering moving, could you share why Fly wasn't adequate for you? Mind sharing why you chose Render over other competitors? Considering the same Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situation. If you're still on it, make your plans to move away now. Time is ticking until a major outage or another security incident like this one. See my comment history and related threads for more. Specifically this summary: https://news.ycombinator.com/item?id=31374048 I would not say that they lied about the env vars. The stated line is still "env vars in apps were not compromised, but env vars in CI pipelines and review apps were". For some applications there may have been shared data in these vars - in our case (N=1) our CI pipeline and review apps had a dramatically smaller and less critical set of variables. It still sucks that they are parceling out the information, but the claim that they outright lied is not true. The lie was: > We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets. https://status.heroku.com/incidents/2413 Nowhere in that did it clarify it was speaking of app but not pipeline env vars. They had plenty of time to author that post too. Make sure you rotate those app env vars anyways as this somehow appears to be getting worse by the week. I would like to move but there are really no good alternatives that are even close to Heroku. YMMV, but I quite like Render for the sorts of things I'd have used Heroku for ~5 years ago. Plenty of folks I respect absolutely love fly.io--I have less hands-on experience there, but they've got a fantastic crew, too. Who are you using for a database provider? Render doesn’t have auto backups and HA Postgres right? Those are table stakes for me. We've had a number of folks migrate over to Crunchy Bridge [1] from Heroku, the only main feature we're missing at this point is dataclips which is coming. We can also help with pretty much no downtime migration even for larger databases. And the team over here is pretty much same team that built the original Heroku Postgres. Render has automatic Postgres backups, and you can also click a button to create new backup. We're working on HA (ETA late summer/early fall). One thing that I have really come to like with Heroku is the pipeline. fly.io doesn't have one (I don't think?) and render isn't the same; it rebuilds for different stages and there's no concept of 'promoting' the same slug. We're planning to release the notion of build promotion on Render late summer/early fall. Well hopefully once it's gone the competition will be able to get more market share to build quality product. Heroku has been starving the entire ecosystem for years. I don't have experience with any other PaaS's so I can't recommend one, but what you say is what I commonly hear. So, just to be clear. Your solution is:
1. move off to the competition who are offering a subpar service; 2. then wait until they eventually catch up in (how many years? who knows..); 3. then profit? Heroku is around, because there is no other service that offers the ease and convenience. I looked at Fly.io and Render and they are nowhere close and mature to Heroku at the moment. For example, here is Fly.io's "Solution" for Redis: > Setting up Redis requires launching it as a separate app.
.. Or if you want something as commmon as Sidekiq.. have fun messing with configuraiton files: https://fly.io/docs/app-guides/multiple-processes/ Now let's compare this to the Heroku experience: How to use Redis: Step 1. Add a Redis addone Step 2. There is no step 2. How about Sidekiq? Step 1. Add a worker Step 2. Update your Procfile Step 3. There is no step 3. Fly.io tells me to "Just Use Bash".. So while I kind of see where are you coming from, unfortunately all these alternatives fall short. Not to mention that Heroku has hundreds of integrations built-in. I'm simply telling you Heroku is not a stable platform that you can trust. It's up to you to figure out what to do. I haven't offered any solutions. Just because it's a slick product to get going doesn't mean that you can trust it to be a reliable and secure host—or be around for the long-term. I think it's obvious an ecosystem without a Heroku will help the upstarts. I understand that doesn't help you get a new host today. I'm not telling you to just go to another PaaS and expect them to be the next Heroku in a couple of years—chances are they won't be. This makes no sense. Heroku have had no competition because nobody has built a better product. They’ve not been starving anyone or anything. Given the biggest and most common complaint most lay against Heroku is that it’s too expensive, if anything the lack of innovation for years has created a huge window for a competitor. And yet here we are. Still. tl;dr: Heroku is taking customers away that if competitors had it: they would be able to receive more capital It's not unlike Google Search. Google Search has atrophied over the years but because it's still the best in the market, it's used by almost everyone. Competition is hard to build because it has to be better than Google Search in order to bother using it. Heroku competitors have struggled in part because Heroku is a fully featured platform. It's relatively easy to build a platform that ticks a couple of boxes really well but building something that matches Heroku in feature parity is a daunting task. In order for competition to get there they need customers and funding, and funding is way easier to get the more customers come in through the door. Once Heroku dies (perhaps already since this incident) we'll start to see real competition in this space because their competition will be getting used. The PaaS space needs that oxygen Heroku is taking up. Heroku hasn’t been starving the ecosystem. They simply haven’t had real competition on their caliber of (zero-)devops. Have you explored Aptible? https://www.aptible.com/heroku-alternative/ I agree although I literally just started using heroku again after many years. I haven't seen alternatives that support the idea of multiple buildpacks. Which seems sort of a must have for any non-js backend because a frontend js stack is going to exist no matter what. All of Render's native environments include Node by default. I read this when looking at migrating: "We are working on supporting migration of Heroku apps that use multiple buildpacks." -- https://render.com/docs/migrate-from-heroku#fnref-2 I guess I would need to place the extra node steps and python build steps into a single build file and then point the render.yaml at that custom script? I wanted to tell from the docs if it was possible before even starting on a prototype to demo it. Is there an example app that fulfills this. Thanks for your time. Exactly, you can use any command or script to build your app, and you would select Python as the native Render environment and run your Node build as part of your build script. Here's a quick example: https://render.com/docs/deploy-django#create-a-build-script We are looking to move to either fly.io or render. Leaning towards render currently but I haven’t spent a ton of time on the effort. Why not fly.io? Take a look at platform.sh 11 days ago they said "While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets." I guess that was a lie?! the subsequent blog post (https://blog.heroku.com/we-heard-your-feedback) says: > Additionally, we have no evidence that the attacker has accessed any customer accounts or decrypted customers’ environment variables. which, as pointed out in its HN thread, means "we now know they got access to encrypted vars, and we don't know yet if they could have decrypted them." in BS-speak. The title "We've Heard Your Feedback" is also a red herring, usually means "we know we fucked up bad and we still have no idea of the whole impact of the breach". HN with the quality security advice, with all the recommendations to rotate config vars just to be safe. I can't think of a reason to not rotate credentials and variables the second you see a security incident of this scale even when it was in the very earliest stages. Better safe than sorry, and also a good time to review just how easy it is to update all the variables (is it automated & scripted, where are they stored and generated, etc.) For those looking for a great alternative to Review Apps - Livecycle is great (https://livecycle.io/).
It offers the automated per-PR ephemeral environment and much more. It also includes a rich layer of built-in collaboration and annotation tools that allow all collaborators to join the PR review and leave their comments visually, on top of the product UI. The comments are maintained in Livecycle and also synced back to Git as review comments so that developers can see the issues faster, understand them better and address them sooner.
There are setup templates that make it easy to simply copy over your docker file and get started within a few minutes. And the team is eager to help if you have any questions or issues. When Salesforce bought Heroku back in 2011, it was pretty clear Heroku would become yet another dead product that once was an absolute great piece of software. Why? Commercialism. Founders sell to the highest bidder to make their exit worthwhile for themselves, not caring about the future of the product (and customers). It's a no-brainer that a commercial company like Salesforce (it's in their name!) doesn't have what it takes to build AAA software, but focuses on maximizing their profit. They drove away their best staff, focused on the wrong features, and are seemingly overwhelmed by maintaining their purchased software, all while probably not even realizing their demise. We should all come to the agreement that takeovers of fundamental software by incompetent companies should be seen as a hostility towards every current user of said software. > When Salesforce bought Heroku back in 2011, it was pretty clear Heroku would become yet another dead product that once was an absolute great piece of software. That feels like a angsty-tinted view. I recall the day it happened. The Ruby dev shop I was at was optimistically nervous. As Heroku had been a shiny new thing and only deployed Ruby. Acquisition allowed them to expand and support other languages. They didn't even have pipelines! https://techcrunch.com/2010/12/08/breaking-salesforce-buys-h... Yup, that's game, set, and match. I really feel bad for all the herokai still left holding the line, but damn am I glad I got out when I did. Slightly off-topic, but can anyone tell me how you’d know that your database has been accessed by a threat actor? Should I be periodically reviewing all my logs for something unusual? Yep. Quick and dirty you could alert on large or slow queries, and check the logs periodically. I know it’s probably not effective but I grep logs and watch the terminal looking for aberrant shapes. I believe AWS offers a ML solution to watch your infra and alert for things that are out of the usual, and I’m sure (haven’t built it, but talked to people that worked in the systems) the big companies have sophisticated systems looking for threats that use everything above and far more. Sounds like a customer's canary token triggered this based on the current reporting. "Trust is our Number 1 value." For folks employed by Salesforce, the phrase "Trust is our number 1 value" only comes up in two contexts: 1. company all-hands meetings, which are basically pep rallies with no actual content 2. when someone working at Salesforce brings up a glaring problem and says "if Trust is our number 1 value, why don't we do something about this huge problem?", which is usually met with either silence and bureaucratic obstacles or with excuses, usually something like "customers trust us to spend the money they pay us building the features and products they want", which is like...exactly not the definition used at any of the pep rallies. that and dreamforce (over and over again) Has anyone done a load test comparison for Heroku vs. Render.com? The "Pro Ultra" on Render is $450/month for 32 GB RAM + 8 CPU. The Heroku Performance L Dyno is $500 a month for 14 GB RAM. The Render server seems like a much better offering. Has anyone done a security review of Heroku alternatives? I see lots of comments about moving there and using them instead of Heroku because of this incident. But that's hardly a justifiable reason alone, since those new platforms are probably worse off in security. Fly have https://news.ycombinator.com/user?id=tptacek working for them so it's hard to imagine they would get security wrong. Is render SOC 2 certified? ISO 27001? Those are fairly standard. If they don’t have either it’s going to be a hard time getting any serious business to migrate. Render will have these certifications late summer (we're currently being audited for both). Aptible has always prioritized security, and is a competitive alternative. https://www.aptible.com/heroku-alternative/ Any guesses as to how the "threat actor" got access to the databases? I understand most guesses would be conjecture (unless someone here has an inside scoop). Just curious about how stuff like this usually gets compromised. This is why we at vulert.com never access the customer's codebase or any installation, who doesn't know vulert, it's a service that notifies you for security issues in your software dependencies. I completely stopped getting their update emails when I deleted all of my running apps. This is the second one I've seen on HN that I haven't received (though I received the others). Is the impact limited to specific customer accounts, or are they just not updating me anymore? From the email: > We value transparency and wanted to notify you of an issue affecting your account. My guess is they sent it to users with pipelines that have env vars. It's funny since this sentence demonstrates they don't value transparency by not telling the other users more information about the hack. They updated Heroku Status but surprisingly failed to mention anything about CI or pipelines. I think it's specific to apps with Review Apps or Heroku CI (specifically the Config Vars). If you didn't have those, based on this email, you may not be impacted. I had both of those, and I had config vars for my pipeline apps. Still havent had any notification on this. A+ for Heroku oh shit. I'm surprised we haven't heard about major services getting hacked to oblivion right now, so much is stored in environment variables are there any mystery hacks occurring yet? is this database known to have been spread anywhere? Hopefully most people didn't have production creds stored in pipeline config. These vars were for review apps and ci. My sides, what a shitshow.