Vercel's CEO offers to cover expenses of 'Jmail'
threads.comWhat would this have costed had they used render or railway? Maybe, what, $200 a month?
Vercel's pricing is so ridiculously convoluted that you can't even cleanly compare usage. With render/railway/(insert provider of choice) you can at least predict that you're your biggest cost is going to be egress.
edit: I just saw that it gets 450m pageviews. I'm guessing on the upper end this costs ~$1k with railway + cloudflare?
Railway is getting so good I'm not sure what Vercel brings to the party anyway.
While I used to think Railway was an amazing service, I had a production workload get broken because they removed a feature without any depreciation period or warning. I now struggle to recommend it for anything more than a hobby project. Vercel has the benifit of being big enough they have to do things properly. For reference https://station.railway.com/questions/smtp-connection-failur...
SMTP is gated behind the $20/mo Pro plan to reduce spam on the internet.
It sounds like you were running a production workload on the Hobby plan
450 million pageviews on Vercel = $46,000
450 million pageviews on a single 16c/32t OVH box with nginx and a 3 Gbps connection = $245
Why is everyone using Vercel and the likes anyway?
Setting up a VPS with Node takes ten minutes and is miles cheaper. And it's not like you never have to debug issues with serverless configurations, which can even occasionally be harder to debug because of their proprietary natures.
This is the Dropbox problem. People are willing to pay for convenience, and tech folks tend to underestimate how much convenience comes from seemingly simple solutions
Just note this guy took a selfie with netanyahu. So whatever his motives. That will never be unseen.
The post said 450 million pageviews, likely since November. If we make very generous assumptions and assume that each pageview is a megabyte (very generous based on my own experience scanning billions of websites), then that's 450TB total in traffic. If you really did 450TB per month in traffic, you would need slightly more than one gigabit line (and hence VPS), but not more than two. With Hetzner the traffic would cost you €450 or $535.
Did I get something wrong?
Well, https://jmail.world/jacebook-logo.png is 670KB by itself and loaded on initial load, so I think they might have blown your suggested traffic budget and still have some optimization to do.
Fair enough, I just loaded some pages and some of them are even bigger than 2MB. But then again those static resources would be cached client-side. So unless you have 450 million unique visitors who only ever go to one URL on your site, you are looking at significantly less per pageview. I reloaded the frontpage with caching enabled and it was ~ 30kB of data transfer.
Isn’t part of Vercel’s value proposition a robust global CDN in front? Seems quite a bit different than one sweaty VM in Helsinki.
Yes, and I didn't mean to imply that a single VPS is all you needed. But I wanted to put things into perspective for the other posters who claimed that you couldn't possibly serve a site like this from a single machine, purely in terms of performance.
well each view of an 'epstien file' is a pdf with images embeded so i think your 1mb might be not so generous.
Isn't Jmail a static site? How could the bill be $47k?
There's a searchbar, and the amount of content you search through is far more than you could do with clientside search.
Definitely not just static content.
There is still a helluva lot of data to transfer to the client, I'm sure it's being stored somewhere.
And with the entire world perusing this archive, I'm sure the costs will be very high, regardless of provider.
That's a good question. As someone bootstraping a few projects on Vercel this post has me looking over at the pricing sheet more closely.
Bandwith costs
It would be $0 if they spent 10 minutes just throwing Cloudflare CDN in front. They don't even need to move off Vercel.
Tech bros and VCs need to eat, that's how.
1) Covering the ~$50k hosting bill for Jmail on Vercel sounds generous, but a self-hosted VPS on Hetzner could serve the same purpose for ~€30/month, which is orders of magnitude cheaper and avoids vendor lock-in.
2) This comes as the CEO of Vercel, Guillermo Rauch, is already facing community backlash for publicly supporting Israeli Prime Minister Netanyahu, a move that’s led to boycotts and migrations off the platform among developers. All my homies hate Vercel.
I'm new to webdevelopment and was using Vercel because people told me it was good, but I was unaware that the company supported the genocide. What other similar services there are that you would recommend?
Do you really think a $30 hetzner host can sustain that level of traffic performantly? Don't get me wrong, I love hetzner, but I would be very surprised if the numbers work out there.
Isn’t it just serving static content and the content fitting in RAM? If so your laptop can serve it just fine even.
A laptop would have a hard time serve thousands of people hitting a single endpoint multiple times a day.
It shouldn't. The issue is that most developers would rather spin up another instance of their server than solve the performance issue in their code, so now it's a common belief that computers are really slow to serve content.
And we are talking about static content. You will be bottlenecked by bandwidth before you are ever bottlenecked by your laptop.
To be fair, computers are slow if you intentionally rent slow & overpriced ones from really poor-value vendors like cloud providers. For people who started their career in this madness they might be genuinely unaware of how fast modern hardware has become.
With a 2025 tech stack, yes. With a 2005 tech stack, no. Don't use any containers, no[/limited] server-side dynamic script languages, no microservices or anything like that.
Considering the content is essentially static, this is actually viable. Search functions might be a bit problematic, but that's a solvable problem.
Of course you pay with engineering skills and resources.
SRE here, Containers are not causing any performance problem.
Maybe the perception comes from all the Mac and Windows devs having to run a Linux VM to use containers.
Is there any feasible way to implement search client-side on a database of this scale?
I guess you would need some sort of search term to document id mapping that gets downloaded to the browser but maybe there's something more efficient than trying to figure out what everyone might be searching for in advance?
And how would you do searching for phrases or substrings? I've no idea if that's doable without having a database server-side that has the whole document store to search through.
Theoretically, just thinking about the problem... You could probably embrace offline first and sync to indexeddb? After that search would become simple to query. Obviously comes with it's own challenges, depending on your user base (e.g. not a good idea if it's only a temporary login etc)
There are several implementations of backing an Sqlite3 database with a lazy loaded then cached network storage, including multiple that work over HTTP (iirc usually with range requests). Those basically just work.
A 6 core server or laptop can easily serve 100K requests per second, so 259B requests per month. 576x more than their current load.
No it won't. This is static content we're talking about. The only thing limiting you is your network throughput and maybe disk IO (assuming it doesn't fit in a compressed RAM). Even for an "around the globe roundtrip" latency, we're still talking few hundred msec.
Some cloud products have distorted an entire generation of developers understanding of how services can scale.
I think it’s more helpful to discuss this in requests per second.
I’d interpret “thousands of people hitting a single endpoint multiple times a day” as something like 10,000 people making ~5 requests per 24 hours. That’s 0.5 requests per second.
It all depends of course, but generally no, a laptop could handle that just fine.
There may be a risk of running into thermal throttling in such a use-case, as laptops are really not designed for sustained loads of any variety. Some deal with it better than others, but few deal with it well.
Part of why this is a problem is that consumer grade NICs often tend to overload quite a lot of work to the CPU that higher end server specced NICs do themselves, as a laptop isn't really expected to have to keep up with 10K concurrent TCP connections.
If it's mostly static, just cache it at the http level e.g. cloudflare which I believe wouldn't even charge for 450m requests on the $20 plan at least
I would use a $100/mo box with a much better CPU and more RAM, but I think the pinch point might be the 1Gbps unmetered networking that Hetzner provide.
They will sell you a 10Gbps uplink however, with (very reasonably priced) metered bandwidth.
yes
and if it doesn't spawn up another $30 instance and add another RR entry to the dns
serving static content scales horizontally perfectly
For sure, even cheaper if you cache effectively.
Lol yes? It's all reads. If it can all fit in ram, great. Otherwise an SSD will do fine too.
You could probably serve it from the quad-core ARM64 inside the SSD controller, if you were trying "for the lulz".
No . Hetzner would terminate your server as you are not a profitable customer.
We handle 200x their request load on two Hetzner servers.
A profitable customer? How would Hetzner know if you're profitable or not?
I've hosted side projects on Hetzner for years and have never experienced anything like that. Do you have any references of projects to which it happened?
Because you are using an incredibly large amount of bandwidth for €30 a month.
They offer unlimited bandwidth with their dedicated servers under a “fair usage” policy.
The bandwidth costs would be higher than what you pay monthly, so they would simply drop you.
You are probably using very little bandwidth, so it doesn’t matter in your case.
However, I assume Jmail consumes a very large amount of bandwidth.
I have heard of hetzner terminating customer relationships if too many legal complaints are filed against your VPSes.
But not because of being "not a profitable customer". Mind sharing some links here?
I am not sure how one even gets 250TB/mo through a 1Gbps link. In any case, completely saturating your networking for the full month is outside most people's definition of "fair use".
Yeah but they still advertise with unlimited traffic. "All root servers have a dedicated 1 GBit uplink by default and with it unlimited traffic" https://docs.hetzner.com/robot/general/traffic/
2nd point resonates with me, how come he wants to cover expenses, while being connected to Israeli PM and Epstein is connected to Israel through Ehud Barak.
Isn't he going to ask for a "favor"?
Barak and bibi are political enemies (or at least we're when Barack was a relevant political figure) and besides that I haven't seen anything suggesting that his connection with bibi is more than the one meeting that was publicized.
Even before the Vercel CEO supporting a genocidal maniac. Vercel as a platform has been silently giving open source projects a "fuck you, pay me" when it comes to renewing benefits.
Have seen it happen to smaller projects and even pointed it out when Vercel took static sites down.
So they have always had a bad rep in my opinion.
$50k and €30 are of the same order of magnitude.
This is offtopic honestly, but I'm curious if I've been using this phrase wrong for my whole life. Doesn't "order of magnitude" refer to steps of powers of ten?
$50000 vs €30. (or €42066.30 vs €30 if I normalize the currency) 5x10^4 vs 3x10^1.
You have it right, perhaps the original poster was referring to it in a more colloquial manner, in the sense that against 200 million in revenue, 50,000 and 30 are in the same ballpark?
> in the sense that against 200 million in revenue, 50,000 and 30 are in the same ballpark
I don't understand how those are in the same ballpark? I thought saying something is in the same ballpark suggested that they are similar in scale, and the implication is that little-leauge does not play in the same ballpark as a NBA team. They are in the same category (baseball), but not at all the same level.
At a big enough scale, previously large differences are effectively 0.
50k/mo is 600,000/yr vs 360/yr at 30/mo. Thats existential for a 1MM/yr company. Neither register on a balance sheet for a 1B/yr company. They are both closer to 0 than being a major cost.
But saying that 200 million and 30 are in the same ballpark is not true in 99.99% of contexts.
Even 50k and 30 I would not say are in the same ballpark. I've worked for major corps and of course a cost saving of 50k/month would not register for the overall company but it probably would for my team. A saving of 30/month is probably not worth spending any considerable amount of time on in most non-personal contexts.
I took it as a joke about the USD/EUR exchange rate ;)
Unreadable without an instagram/threads account
no account needed -> https://www.threads.com/@qa_test_hq/post/DUkC_zjiGQh/media
It's also dumb because the original interaction happened on X https://x.com/rauchg/status/2020984434338693622
Say what you want about Elon but X is where all the investors and tech execs are. Nobody is going to sign up for threads because they saw it link to a picture in a HN post
With noscript active, I was able to see most of it.
>Vercel's CEO offers to cover expenses of 'Jmail' as it has become the number 1 site for tracking the Epstein files
and the expense is 46,486 USD. He said he is happy to cover expenses and that Vercel worked good for your needs.
Rather than linking to a threads post that is a screenshot of the x.com post with little to no commentary, we should be linking to the original x.com post
or to a nitter instance, where you can actually read responses without signing in:
https://xcancel.com/rauchg/status/2020984434338693622 or https://nitter.net/rauchg/status/2020984434338693622
Considering that Twitter doesn't show the original post for non-logged-in users, the screenshot on Threads actually provides a better reading experience for most people!
It seems like sometimes it does and sometimes it doesn't, and I can only imagine popularity is somehow the reason.
Only if you already have a threads account. On mobile it makes you sign in
You're missing a large part of the conversation and context if you don't at least link to the source.
Agreed, was slow to load and I just had to find the source post on X to view the real conversation "thread".
This is the first Threads link I've ever seen here. Is that what Threads mainly is, reposting X screenshots and starting a sidechain conversation?
for fucks sakes, don’t link twitter
Use nitter or xcancel
Boo is the only think i can imagine when I hear about Vercel
How does this work from an accounting perspective? They write off a bad debt, but the actual loss is likely multiple orders of magnitude less. Do they only get to write off up to the actuals?
It's simply discounting the fees for that one user to zero.
(It's not writing off a bad debt, which is technically different)
So: your costs are still X but now your revenue is Y instead of Y + (that one user's fee which likely wasn't going to get paid anyway)
You pay taxes on Y - X (profit).
So, really, their costs just increased by whatever it cost to deliver that data (likely zero depending on how they're billed for it), and their revenue didn't change at all.
Turning a no-collect situation into a PR positive.
To be fair: it really depends on their datacenter environment; if they're physically hosting, this is probably a rounding error. But, if instead, they're actually running on top of AWS or another hyperscaler and paying 9 cents per gigabyte for traffic, then their bandwidth bill could actually be quite substantial and they're just passing that along to the customer. In that case, this could be actually quite generous of them.
You deduct the expenses you paid, not the income you hoped to earn.
Marketing probably, unless thew CEO pulls out his credit card
I don't really understand why he'd say he'd cover the costs personally... like, Vercel can just write it off, what's the significance of him paying for it?
Yes, because accounts payable are valued at recognized revenue, and aren't being revalued at cost when written off.
Alternatively, bill the costs under the PR department as a marketing campaign.
I suspect this sort of thing is some of the best marketing money can buy anyhow, so it's a bit of a no-brainer.
threads.com wrapper for some reason. The actual origin post is on x.com: https://x.com/rtwlz/status/2020957597810254052
Guillermo reply: https://x.com/rauchg/status/2020984434338693622
Would have been much cheaper in the first case on Cloudflare
I'm not the first to point this out but the website in question, which is mostly static, could easily be hosted on a VPS for at most a couple hundred dollars a month.
Directly to CDN. Put it in a CloudFront distribution and it would be a fraction of a fraction of that Vercel bill.
Remember kids, they're incentivized to get you to build something to burn as much compute as possible.
Or dump the EML for everyone to import into their own clients.
That feels gross. I'd hate to have the risk of something that was meant to be redacted and not and now you have that on your own client.
It's a screenshot of this Twitter thread [1] for those who can't view Threads on mobile because it forces you to sign in.
I know there's a lot of questions why it's so expensive, but can I just extol the work done by Riley and team?
Since the Epstein files dropped they've cloned gmail, gdrive, gmessages, amazon orders, transcribed court proceedings (yes with AI), fights, facebook, and imessages.
It's an insane amount of work. They added the latest batch of files, photos, videos in like 2 weeks. And he's keeping up files that the justice department took down.
jmail has made it so much easier for everyone to explore the files.
I don't know how Riley has planned to monetize this or if it's simply for the public good. I can totally understand not wanting to optimize for cost from the outset. And I see a lot of abject criticism on every social media platform rather than constructive.
This was there (way) before the last documents drop.
But your point stands, the amount of work they've put into this is remarkable.
Is that good PR?
Doesn't seem to be a good idea to be associated with that.
Why wouldn't it be good to be associated with publicly exposing pedophiles, cannibals, murderers, and rapists? That seems to be a very good thing to be opposed to them.
Its good PR. He had some pretty bad PR recently that caused a lot of people to boycott the service. I assume this is him trying to regain trust or something?
Garbage engineering begets garbage bills
$46k for 470m page views.
That seems extremely expensive. What the heck?
Is he using Vercel Functions as well?
I think this is where some SPA + a few instances of a Node.js server + Redis would be much cheaper.
I'd say you can probably serve this much on $1k/month? It's simple content. It's not like it needs to do complex business logic in the backend.
Public files needing to be distributed to a huge population of interested persons? Sounds like the perfect situation for an oldschool torrent. That's how large data leaks were handled back in my day. 450TB is peanuts for perhaps ten thousand peers on fast residential connections.
Insane to me a bill that large for what is effectively hosting static content. He could dump the entire thing on S3 and even with cloudfront it would be fraction of that.
in other words, "we know our product is overpriced as hell, so i will pay for it to avoid further exposure of our pricing model".
this seems like an unreasonably unchartiable reading of a relatively chill and nice situation
I'm not sure I would describe the discourse around Vercel and its CEO as "relatively chill and nice". Things are perceived in context.
It's common to hear rumors about SF CEOs and their NDAs with young (but legal) ladies. I hope there's no irony here, g.