Signal president Meredith Whittaker says they had no choice but to use AWS
theregister.comPeople get very reductionist about language in these things. I think she is right that the most logical choice was to partner with an existing hyperscale backend, they had choices of which and went with AWS for reasons which made sense to them. "no choice" isn't a literal statement of fact, its a choice they made, sure. There may be price, or jurisdictional, or complexity, or understanding, or willingness to sign a specific kind of contract, all kinds of reasons. The alternates all come with their burdens too.
They had choices beyond just other hyperscalers. Rolling their own probably would have meant both capex and opex, which reduced to opex in AWS and so made both logical and financial sense. In risk terms you might have said (before the incident) it was also the best way to lay off risk, but it turns out "too big to fail" actually doesn't mean what it says on the label.
I still back signal over all the other choices. I wasn't looking for an excuse to leave, and as a strawman if you leave signal because chosing AWS as a backend "was unwise" or "was the wrong choice" I think you're reading the signal wrong (sorry)
I would add that "the register" has a house style, and it's not tending to damp down. It likes to be inflammatory, it's tagline "biting the hand which feeds IT" rings true. I enjoy reading it, and I've had work repeated in it, but I also read it with a jaundiced eye. I don't like the comments section it's a minefield of in-group language, memes, bad behaviour.
I always assumed Signal favored AWS because AWS is valuable to a lot of foreign entities that might just as well block all signal.org and any IP network advertised by its ASN.
The advantage of bundling your service in a hyper scaler is in persuading censors that they’d rather tolerate Signal than lose AWS. This doesn’t work in China which has sophisticated alternatives, but it can help Signal hold on in other countries.
In reality it's probably the same reason as why they used Chromium Electron for their desktop app - it was easy.
Couldn’t we try to rely more in other topologies other than client server. I see technologies like wireguard and Tailscale could reduce the dependence on data centers now that we have considerable compute on our homes.
Anyone can choose to not use AWS.
It’s ok, the world won’t end.
You might get systems that are reliable and cost a great deal less if you exit AWS.
Lose your fear, have courage, find a better cheaper faster more reliable alternative…. well pretty much anywhere.
Putting it another way…. AWS is slow, complex, VERY expensive, unreliable and underpowered.
They have convinced you this is your only choice. It is not.
All true except it being expensive though. I'm yet to find a major and reliable cloud service provider who can beat AWS on pricing. In fact, most either use or resell AWS under the hood themselves, even Vercel does as it turned out recently; and they're supposed to be one of the most competitive in the industry.
There are some rare exceptions who have their own large scale infrastructure and don't depend on AWS like Hetzner in EU, Alibaba Cloud in China or Ananta Cloud in India, but this market is still emerging.
Self-managing databases is radically cheaper than using RDS which is often times priced at 2.5x - 3x the price of the underlying EC2 instance.
So do it?
You are making the claim you see a clear massive global untapped market, for lower price and higher quality cloud/global compute, that you know how to provision and serve.
Apparently Signal will be happy to hear from you.
Why will they be happy if they're convinced their current option is the only one possible?
Ah. You don't believe you have an alternate solution they would find convincing, without even trying.
Well then. I believe you.
1. It might not be unsafe, but it's still fragile: American government can decide anytime what to do with AWS servers, locking (non-USA) users out of the chat
2. My donations to Signal apparently also go to Bezos
Aww, they grow up so fast. It seems like it was only yesterday that Moxie was saying they had no choice but to use Doubleclick Play Services.
You can tell that Whittaker isn't an engineer.
WhatsApp grew to much larger scale than Signal: self hosted, not on cloud. Running Erlang and FreeBSD.
Telegram grew to much larger scale than Signal: self hosted, not on cloud (dc IPs here: https://docs.pyrogram.org/faq/what-are-the-ip-addresses-of-t...). They set up their datacenters carefully to make it hard for governments to access data via legal mechanisms, something Signal didn't bother with.
Threema, similar concept to Signal: self hosted, not on cloud.
Every other messaging app before these bunch? AIM, ICQ, MSN Messenger, iMessage... self hosted, not on cloud.
The idea there is no choice should be hyperbole but it seems she might really believe that. It says a lot that Signal is run by such a person.
>make it hard for governments to access data via legal mechanisms, something Signal didn't bother with.
No legal mechanism can access proper encyrpted data, something Telgram has to bother
Governments can order the company to push out a new client that turns off decryption for that user, or give them metadata logs (which encryption doesn't help for). Both legal and cryptographic strategies have to be used together.
Signal is not a project led by strong engineers in general, I realized as much when they had a class of bug that should not have been possible in a well-architected app (UI and state were disconnected resulting in random private images being sent to random contacts without your knowledge, an unforgivably bad incident that revealed slop code in the UI).
There are interesting questions to pose - what are the differences and why not follow those paths. Signal is a different application and organization, which has different requirements and resources - for example, they can't optimize performance by sacrificing security, and they have limited resources. It's hard for me to imagine Signal, with already constrained development resources, devoting people, time, wealth and attention to building a private cloud - they would have two businesses, cloud and private communications app. And to match AWS would be pretty difficult - how about scalability for the days when Signals load shoots up? I wonder how these other organizations do it - some clearly have far greater resources than a non-profit with almost no revenue streams.
But you're kidding yourself and everyone else to state an answer. It's amazing how HN commenters love to use leading FOSS projects, like Signal and Mozilla, as targets for their performative takedowns - it causes real harm to the most important projects around. Taken seriously, the parent comment's arguments contain no engineering, and their foundation is a lot of assumptions and arrogance:
No engineering is required to understand those arguments. No competent practicing engineer would offer a serious opinion about an organization and technical issue that they haven't directly examined.
The assumptions are a long list: The totality of reasons that Signal has, as an organization, to choose AWS. The people who made the decision:likely others at Signal were heavily involved, and the CEO's role is unknown to us - maybe just approval - and possibly it was before Whittaker was there. Signal having unlimited flexibiliy in requirements and resources to optimize for this issue.
The arrogance is that we know better than Signal's CEO and team members, who are intimately familiar with the project, the organization, its requirements, its resources. The parent doesn't address most of those essentials.
But maybe the parent is performative - that's not illegal, but ugh, pick on the big guys; punch up, not down.
We do know better than Signal's CEO because Whittaker's statements are false. She said:
> The question isn’t "why does Signal use AWS?" It’s to look at the infrastructural requirements of any global, real-time, mass comms platform and ask how it is that we got to a place where there’s no realistic alternative to AWS and the other hyperscalers
> Which is why nearly everyone that manages a real-time service–from Signal, to X, to Palantir, to Mastodon–rely at least in part on services provisioned by these companies
Which is both dishonest and stupid. She's claiming it's impossible to run an app like Signal outside of public cloud despite all her main competitors doing so. That's why she lists a bunch of non-competitors to try and support her argument.
So it's ironic you say it's arrogant for us to judge their requirements, because we know their requirements. Signal's design is fully open and the requirements of such platforms are well known. It's rather Whittaker's thread which is the height of arrogance. Her response to criticism of downtime is to be "concerned" at the ignorant users who don't "understand" the "concentration of power" and to "explain" to people why it's impossible to do better even as her competitors all do it. It's practically gaslighting.
You're picking apart a few comments made in a moment; that's indicative of nothing (except the person picking them apart).
> we know their requirements. Signal's design is fully open and the requirements of such platforms are well known.
You're kidding yourself. 'Open' doesn't mean you understand them on a level to draw real engineering conclusions. Smart people would wonder at the questions you raise, and ask people who do know. Maybe someone from Signal is around here - but who would respond to someone that calls them stupid?
Whittaker's comments weren't "made in the moment", they were posted in a big thread on Mastodon where she had all the time in the world to write them. But I guess you see that her comments are embarrassing to Signal, otherwise why invent excuses.
The Mozilla/Mitchell Baker vibes are strong here. It indicates a lot about Signal that its leadership doesn't understand who their competitors are (she thinks its Palantir?!), nor the basics of running a messaging service, nor even what her imagined competitors really do. X runs its own datacenters, Palantir is running in every cloud. They don't support her argument.
And I've worked on two web scale products with billions of users, both of them had 5 9's uptime. HN is full of people who have. This stuff isn't rocket science. The first reply on Mastodon points out that even Tor has better uptime than Signal.
She's saying this stuff because of her social background, not technical reality. It's just AWFL activist buzzphrases strung together, the sort of rhetoric that served her well in the past to climb the ladder. She's "concerned" and "surprised" that angry users don't understand the "power" of Amazon which "bodes poorly for our ability to craft reality-based strategies capable of contesting this concentration". She acknowledges "the high stakes use cases of many who rely on Signal" but has no interest in meeting those high stakes by driving the execution of an ordinary HA/multi-cloud/multi-region project, of the type that happens all the time in any bank. That's impossible, literally "there isn't really another choice", and also unnecessary because Signal is a mobile app so it depends on Android and that's the same thing as a depending on a cloud (what?). Her conclusion: "my silver lining hope is that AWS going down can be a learning moment", by which she means a learning moment FOR OTHER PEOPLE.
Can you imagine Mark Zuckerberg or Pavel Durov crying about Amazon and demanding that their outage be lesson to their users? It's unthinkable. They'd be in the conference rooms with the engineers figuring out how to ensure it never happens again. They might publish a public post mortem to build confidence. That's because they are engineers. Whittaker isn't so she publishes heart emojis and expresses concern at how little sympathy she's getting. No, this is 100% bad news for Signal. It's got totally feminized leadership that responds to the orgs own mistakes with demands for empathy, not fast paced engineering.
> The Mozilla/Mitchell Baker vibes are strong here.
> She's saying this stuff because of her social background, not technical reality.
> Can you imagine Mark Zuckerberg or Pavel Durov crying
> Whittaker ... publishes heart emojis and expresses concern
The gamergate vibes are strong here.
> She's saying this stuff because of her social background, not technical reality.
Always look first in the mirror.
This is yet another red flag from Signal.
Telegram was not disrupted during the AWS crash, so they probably were not using it (or had a decent fail-over mechanism to a backup system). Telegram's user-base is two orders of magnitude larger than Signal, so 'we use AWS because we have to' argument clearly is bogus and nonsense.
That flag is tiny compared to the one telegram has been sailing with for years.
Despite there founder crying on twitter[1] how horrible and distopian chat control client side scanning to bypass E2EE would be, telegram is still only offering hidden and limited opt-in E2EE instead of making it global default like signal.
E2EE is nice to have, but not the magic cure Signal advertises it is. The #1 most authoritarian governments access chats is by forcing people to unlock their phone. At which point Signal's obsession with phone numbers becomes a huge liability. You can't claim security while tying a phone number to each and every account.
>The #1 most authoritarian governments access chats is by forcing people to unlock their phone
How would you know this? If they access the data from the platforms server you would never know unlike with obvious forceful physical seazure. The point of E2EE is that the weakest link, the server, is removed. It increases the required threat model from simple dragnet surveillance to high effort targeted attacks. If the client is insecure nothing can protect your data and signal has said that many times.
I don't see how the debate about requiring a phone number is relevant to this discussion since telegram does too.
Because I live in a very authoritarian government. That is how I know.
The weakest link is not the server. The weakest link is the user device. There is no security without anonymity.
That makes no sense. If you don't trust your government anything but E2EE is compromised from the get go. "But they could seize your device" is not an argument against but for mandatory E2EE because it moves the responsibility from the server you have no control over to your device that you do.
>There is no security without anonymity.
You don't understand what these words mean. You can be surveilled 100% by bodyguards and cameras to be secure but have 0% anonymity (or privacy).
Telegram vs Signal is really a moot comparison in the technical sense.
It is more of a question, who would you rather read your messages ? USA or Russia ?
Because even if there is E2E encryption and an open source client, unless you review it and compile it yourself, there is nothing to say that your messages are relayed to some agency's datacenter after decryption. The USA has all the legal framework necessary to achieve that with the tremendous power of the "intelligence" agencies, and Russia.. well.. doesn't even need that.
Telegram's founders are in exile from Russia after the Russian government took over their previous venture (Vkontakte). It is misinformation to associate Telegram with the Russian government.
Nikolai Durov lives in Saint Petersburg and works for the Russian Academy of Sciences. Pavel Durov had visited Russia multiple time since his "exile".
The public-facing story around Telegram is performative PR, which could be explained by the exact reasons listed in the parent comments: association with the Russian state had hindered VK growth besides the CIS region.
I can't verify this anywhere. The latest information appears to be that Nikolai Durov lives in Dubai.
Wikipedia quotes https://www.bbc.com/russian/articles/cdjwm9eew0xo , which uses this article as a source: https://www.kommersant.ru/doc/6920516
Telegram is not related to the government of Russia.
It isn't that it's impossible, but that it would take significantly more resources to accomplish the same thing.
Why? AWS is neither cheap, nor reliable (as we saw last week).
What does Telegram run on?
Their possible DB-related single points of failure are located in on-prem DCs in the Netherlands.
not AWS apparently