How Netflix really uses Java
infoq.comThanks! Macroexpanded:
How Netflix uses Java - https://news.ycombinator.com/item?id=37829395 - Oct 2023 (196 comments)
(Reposts are fine after a year or so, but when a story has had significant discussion within the last year or so, that's what counts as a dupe on HN. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.)
I never worked with micro services, but I have this question. Do micro services require more and better management?
I can imagine that you need to keep track of all these services, along with information about dependencies, code version, API version and others.
Of course, monolithic services also require these this sort of management, but it should be less complex.
The short answer is yes. The longer answer is that you will spend a bunch of engineering time on building systems to help manage these things, but if you do it well and you're at a large enough scale, the investment pays off.
Some companies spend a bunch of engineering time making their monolith easier to manage (Google/Facebook) and some spend it making microservices work (Netflix/Amazon). Neither one is better, just different paths.
Where you get into trouble is trying to go either route without spending the resources to make management better.
> I can imagine that you need to keep track of all these services, along with information about dependencies, code version, API version and others.
This is the software Netflix wrote to do that:
> making their monolith easier to manage (Google/Facebook)
Google has Borg.
They literally invented Kubernetees to make their microservices easier to manage.
---
Don't confuse mono/microservices with mono/micro repositories.
I'm aware of that. I was talking about their modified version of Subversion.
Right. Google has microservices, in a monorepo.
The whole microservices thing is a little crazy.
Someone just looked at what devs came to realize worked well, slapped a name on it and it blew up as a hot thing.
Since then, I've seen so much wrong done in the name of 'microservices'. It shouldn't be some aspirational goal to work with them.
Also, the fact that the word 'micro' is in the name makes the cargo culters split atoms to reach some tiny nirvana. The size isn't the point. The real point is in having the system/component boundaries closely match the problem domain, and that a single group 'maintains' it.
The “micro” moniker is very unfortunate. I have seen architectures where endless strings of lambdas are strung together like function calls, with the developer not realizing or caring that each is a process, and each call is a network call. Literally one service per function, with the completely expected result of horrible performance and high error rates.
They should’ve been called domain services or something.
After few years of development, current project has around 30 microservises which are dependent on each other, communicate through Kafka and have individual mongo databases each with lots of data duplication.
We call it distributed monolith. It has the worst things of the both architectures.
At Netflix scale, there most certainly is a service discovery registry where each service registers itself. The registry can be used for other services to find out how to reach each service, what regions those services are deployed in, how many instances of each service there are, what operations that service supports, etc.
They have Zuul: https://github.com/Netflix/zuul and Eureka: https://github.com/Netflix/eureka
In many ways, micro services are an engineering solution to a management problem, which is: how do we keep feature teams etc. unblocked and constantly deploying without stepping on each other?
Some famous (Google, Meta, Microsoft IIRC) went into the "one repo to rule them all" approach, with requisite tooling, culture, and expectations to enforce that.
Netflix, Amazon (and I suspect a great deal others) went the micro services route.
As far as I can tell, its about whether or not you can reasonably maintain a good mono-culture around development practices and tooling or not ends up being the main question.
Now, there are aspects of micro services that cross these boundaries, but I strongly believe based on my industry observations that this is where it grew out of. If you think about how these organizations approach developing software it can give you a pretty good idea of why they ended up where they did on this frontier.
You may find this video interesting, https://www.youtube.com/watch?v=5IUj1EZwpJY It's about Conway's Law. I think that's worth looking into based on your question. You can look it up on Wikipedia if you don't wanna watch the video and would rather read about it.
I've deployed multiple open sourced netflix java services and each one of them was archived and no longer maintained according to the repos.
Are they still being developed internally and the foss repos are basically without the secret sauce? Or are these projects "done"? Did Netflix move on to replacement tools?
I'm referring to things like hystrix and .. a queue system I can't recall right now. This was a long time ago but that queue system was a bit of a rough roll out and IIRC I had to make a fair amount of code changes to get going.
Not biting the hand that feeds me, just curious about how netflix treats projects like those. I'm usually extremely apprehensive about deploying archived things but a few teams wanted to use them. I don't know if things are archived because they're done, perfect, terrible, cooked, deprecated, insecure..
edit: Oh I hadn't looked at it in a long time, the hystrix repo explains it well on the readme. I'm not sure if the other project I used did that.
I have been out of the loop with Java. Is Virtual Threads the answer to asynchronous I/O? (Much like Go/C# async/node.js?)
That looks like an interesting solution to support asynchronous I/O without breaking all the APIs, and having the async/await mess that C# created.
What do you find messy about it? Seems fairly straight forward, IME.> ...the async/await mess that C# createdNot sure what the poster above was thinking of, but it seems kinda the same as every other language that’s adopted it - powerful, but footguns abound. I ran some async C# in a debugger - in Rider - the other month, and the debugger just goes off the deep end.
Does C# have the same issue Python does with accidentally calling blocking code? In async Python - which I mostly quite like actually - it’s terrifying to bring in a library because if you’re unlucky it does some blocking network call deep inside that stalls your whole app..
.NET uses threadpool with hill-climbing algorithm. Each worker thread has its own queue and can also perform work stealing. However, if that is not enough and the work queues fill up faster than the items in them get processed, the threadpool will spawn additional threads until the processing lag is counteracted (within reason).
This historically allowed it to take a lot of punishment but also led to many legacy codebases to be really shoddily written in regards to async/await, where it would take some time for threadpool to find the optimal amount of threads to run or devs would just set a really high number of minimum threads, increasing overhead.
In .NET 6, threadpool was rewritten in C# and gained the ability to proactively detect blocked threads outside of hill-climbing algorithms and inject new threads immediately. This made it much more resilient against degenerate patterns like for loop + Task.Run(() => Thread.Sleep(n)). Naturally it is still not ideal - operating system can only manage so many threads, but it is pretty darn foolproof if not the most foolproof amongst all threadpool implementations.
As of today, it is in a really good spot and with tuned hill-climbing and thread blocking detection you would see the .NET processes have thread counts that more or less reflect the nature of the work they are doing (if you don't do any "parallel" work while using async - it will kill most threads, sometimes leaving just two).
This can't really happen in C# except maybe if you are working on a GUI thread and you make the mistake of running blocking code on a GUI thread.> Does C# have the same issue Python does with accidentally calling blocking code?For APIs, console apps, and such, it's not a concern for actual parallel-concurrent code. Of course, if you write a standard non-parallel `for` loop that has blocking code, it's going to block in a console app as well if you don't run it on a thread.
But I think that once you do enough JS/TS or C#, `async/await` doesn't feel very onerous if you have the basic concept.
Python's async works on a single thread, C# uses a thread pool. Calling a blocking method is not ideal, but doesn't ruin everything, and it's easy to hand that work off to a separate thread by using Task.Run.
Agreed. The task system in C# is pretty clean imo. Same with Rust (sans the type system implementation of Futures) and Go's goroutines. Especially compared to CompletableFuture in Java.
Specifically, function coloring (C# and Rust in your examples) is not the same as coroutines in Go or virtual threads in Java.
Sure function coloring can be a problem, but the gp just spoke about async/await being a mess.
Function coloring can be handled by just blocking on an async function. Though the reverse takes some planning.
Meanwhile, if you're building something smaller than Netflix, I'm writing a book just about that (https://opinionatedlaunch.com/).
It's about mobile apps, but I talk about backend at great length, especially since my background is Java. The book is called "opinionated" because I cover Quarkus (https://quarkus.io/), monolith, Fly.io, and no K8s. No fancy stuff to pad your CV, just to get things done the simplest way possible, with the least headache in maintenance (I'm lazy).
Why Fly.io if you want to build a monolith? Compared to Heroku or Render, to me Fly.io seems to be more aimed at edge computing.
I want Heroku experience without paying Heroku price, so Render and Fly.io were an option.
But when I evaluated them, Render didn't support loading my own docker image, so it's incompatible with my build process and CI/CD.
The book is a living document, and I always try to find easier/cheaper/faster/better ways of doing things, so I'll definitely check Render again!
Render supports loading your own Docker image now: https://docs.render.com/deploy-an-image
Has anyone else used Netflix DGS outside of Netflix to build GraphQL servers? I'm wondering what the experience was like. Full disclosure: I work at a GraphQL company. But still, I'm genuinely curious. I was a longtime back-end Java developer, and I'm wondering what the experience is like these days.
always wanted to work on a system that needs hystrix, I haven't needed such strict uptimes just yet.
Hystrix isn't only for uptime. It also provides some really great observability. Even if you don't have the uptime requirements, the observability can help with debugging (and having better than expected uptime can never hurt!).
It's in maintenance mode now, FYI.
In theory... would Go be (1) as performant for Netflix's scale/speed needs, and (2) be practical in 2024 from a platform/ecosystem perspective?
I think that depends. One of the advantages that the jdk offers is jit. I haven't seen the inside of netflix, but I am guessing it is similar to other big tech companies where they have a large portion of there instances on all the time, and receiving a lot of traffic. This can be an advantage for the jit.
I think there is a great ecosystem for building large systems with go (check out cncf for a ton of go projects targeting the cloud), but a lot of big tech companies open source their java frameworks, tools, and systems. If you are looking for something really specific there may not be an equivalent in go.
I think go is great, and it is close enough for me to continue to choose it for this kind of work.
Java reached somehow parity with Go performance when using GraalVM and virtual threads. For a Java shop, this will minimize the appeal to move everything to Go as the gains are marginal.
Netflix uses that, too.
I mean...PHP/Hack works for Facebook's scale, so it's not really a black+white question.
Go will give you lower memory usage, less abstractions, and the ecosystem has fewer observability tools.
Netflix Hollow is pretty great: https://hollow.how/
…let's say, your TV, or your iOS device will just do 10 network calls to these different microservices. It will just not scale at all. You would have a very bad customer experience. It would feel like using the Disney app. It's just not ideal.
That’s some shade.
Lol the Disney+ app UX is honestly terrible. Not to mention the regular app crashes on apple devices.
Disney has _how_ much money to spend on their streaming services technology stack and they just... don't?
Just feels like they'll piss on the customerwhen given the chance because people will still pay for that sweet Disney juice regardless.
What I find funny is that with that kind of money you could staff 10 app teams, develop the app 10 times in different ways and just pick the best one in the end.
I've worked in streaming before and something many don't realize is there are 10-15 different app platforms you need to build for (web, ios, android, roku, android tv, tvos, fire tv, xbox, playstation, etc, etc). So 10 app teams only get you ONE app across all of the platforms.
Who picks? Likely that is the real problem.
I don't think the experience is that much different between the different services, honestly. I vaguely have some recollection of Max being a bit bad, but don't remember the details. They are all fine.
Except for YouTube, that's just leagues ahead of everything else.
YouTube is the only thing I've ever interacted with where I can hear the audio of videos that I shouldn't be able to hear. And I've had times where two were playing at once overlapping eachother.
It only happens a few times per year for me, but I've never had it happen on another service.
Small price to pay for videos that load almost instantly. It's certainly faster than my Jellyfin setup despite Jellyfin's massive home field advantage (SSDs and 0ms LAN latency)
I don't feel that's true.
For example, Disney is actually pretty terrible compared to Netflix + Prime. Worst part of the service is when it frequently loses where you were in a program. They also made it really hard to get to 'continue watching' for a long time.
Another one that has annoying UX issues is BBC iPlayer. If you click on the latest episode of a show that you're watching in 'popular', it won't resume where you were in the episodes, it will play that episode. Very frustrating. While technically correct, that's not what most people want to happen.
Nah the Disney app is really poor, it crashes / refuses to load way more than the other ones in my experience. On multiple different devices / networks.
This adds absolutely nothing to the conversation, but the Disney app on LG tvs are rock solid
I do find most are very similar.
The service type is a bit different, but IMO Peacock is the worst. None of the others are as consistently slow for me.
the tvos app for youtube is pretty bad
What problems are you experiencing?
One of my peeves on Disney+ on Roku, is scrolling down to ~third row for Continue Watching, takes another 3 seconds to load that list while I stare at the silhouette thumbnails or whatever. I can (barely) understand that they want to advertise their new content prominently above, but it’s just sloppy to make such a common flow (continue watching) feel so slow and irritating.
My peeve with Roku is that the Roku Express' processor is too slow for a smooth UI experience. It's always a few button presses behind.
You shouldn't have to buy the more expensive Roku 4K just to have a device powerful enough to run its own interface smoothly. Granted it's only $10 more but had I known the UI was sluggish I would have paid it from the start even though I don't need 4K. But now I've bought both.
Disney app works really well for me (Android).
But now I'm gonna hear the comment every time I wait for it to load :P
Would hit a whole lot better if the Netflix app wasn't equally as bad as the Disney+ one....
This seems really immature as well.
Focus on your own house. At the end of the day both Disney and Hulu work more than fine for me( I'm guessing it's the same team ).
Netflix had what, a decade head start. I'd expect them to of perfected streaming.
I don't consider making many network calls necessarily bad for experience. My https://hackerer.news page load makes more requests than I care to count and it runs for 70 seconds (looking at the 'date:'s in the network tab). But the page renders very quickly. It might be bad if you're in AU/NZ idk, but all the requests are cached via CloudFlare and being individual units is highly cacheable which probably offsets a lot of other downsides.
Based on my own experience its just frank observation
I'm interviewing at Disney Streaming, lol
There is a bug in the Roku app where if you close it and then re-open quickly it will hang on the profile login screen every, single time. Have to go back to Home again and re-open the app.
Seems dumb, but I have 2 little kids and they both like to "pause" whatever show when it's time to turn the tv off. So after the first one does it, I have to do this dumb little dance to get the app to load again so the second one can do it.
> they both like to "pause"
My son also want to do this. Pause. Then turn the TV off. Maybe they believe the show will keep running with the TV off unless paused?
I have sometimes opened YouTube on my Roku to find it skipped part of a video I was watching, maybe because the Roku took time to close down after I turned off the TV by IR.
Better safe than sorry. ^S
Fix it!
And then quit. Reason: My work is done here.
I bought into the "RX Java/JS/etc.." years back. Everyone I showed the code couldn't handle it, and we just backed off to other methods, and things worked just fine.
RX has some interesting ideas, but from a practical standpoint, at companies not netflix, it just doesn't work
Even at Netflix RxJava is not fully accepted. When I was there I wrote a service which called many downstreams. It orchestrated test user creation. I used RxJava because it was being pushed by the platform team, and coupled well with our internal GRPC api handlers.
I shared with the team and people complained about the complexity… so I ended up ripping it out. What people wanted was a test user creation service that worked well and could be maintained… not one that was the most efficient.
Looking back, do you think choosing practical reasons over efficiency was the right call?
Yes. Code is read (and debugged) way more than it is written.
I was using RxJava initially because it was being pushed by the platform team and coupled pretty well with the GRPC interfaces we used internally. However, the rest of my team couldn't/didn't want to wrap their minds around the concurrently functional tasking that it brought (which was understandable).
I didn't want the code to have a bus factor of 1 (me).
Is there any reason they don't adopt kotlin? It seems a lot simpeler with suspended functions and coroutines?Instead of RXJava.
In growth we had a failed experience with groovy… but Netflix wide? I think they just wanted to stick with basic Java.
Adopting RxJava means fixing all the bugs in everything up front. That is why it feels complex. In my opinion and in the opinion of other adopters, it is a more expressive way to define correct behavior.
This is essentially the same tension between static and dynamic typing.
No matter what, all bug free code is complex. Correct code can be written in any paradigm.
But productful code ships with bugs.
Does test user creation need to be bug free? I don’t know. It feels like something where productful test user creation can ship with bugs, even if in reality your test code harnesses should not be buggy.
You can step even further back and ask: when you have essentially unlimited resources, why ship buggy code at all? Netflix has unlimited capital, and it isn’t in a hurry anywhere. Engineers should never choose productful over bug-free. You learn nothing (Netflix’s product problem will never resemble your startup’s product problem) and you forfeit the opportunity to learn (ReactiveX is a good, expressive paradigm).
This is a startup forum, is this true for startups? IMO, yes. Your startup is going to fail. You will not learn valuable product lessons from the journey. I mean people certainly pretend and say they do, but even in the situation of success, Brian Chesky still hasn’t become Walt Disney last I checked. But once you have enough money to be employed for a year, as an engineer, you know, fucking learn how to do something the right way, bug free and performantly.
So I agree your coworkers are fundamentally making an error.
> Adopting RxJava means fixing all the bugs in everything up front. That is why it feels complex.
I disagree with this. It mentally changes the thought processes when debugging, which was the biggest issue that was brought up.
> Does test user creation need to be bug free? I don’t know. It feels like something where productful test user creation can ship with bugs, even if in reality your test code harnesses should not be buggy.
It should be bug free at Netflix scale. My service created more customers daily than Netflix signed up monthly. It also sat in the middle of Growth. People use it to gate deployments. It needs to work flawlessly (or at least consistently).
This was part of some “testing in production” type scheme?
Are there any talks or blog posts on how y’all wired this up, lessons learned etc?
No this all happened in preprod. Every team in the growth space ran their own tests across the various permutations of users (country, payment type, plan, ... at the time free trial, feature flags) to determine if their code was valid and worked to go to production. Each team continuously deployed independently. Given that we also were continuously integrating as well... you get a lot of customers created.
Sorry no talks.
I tried using an API based on RxJava to implement some async code... It was buggy as hell, as using the API correctly was nearly impossible for me - that's the opposite of a good API which makes it very hard to use wrong. I am in the process of replacing all that shitty code with Virtual Threads, and all bugs just immediately disappear as the code is so much more straightforward to analyse.
I had a close look at RxJava and decided I wasn't smart enough to understand it. I intentionally didn't tell my colleagues about it, they would have loved it and I would have been in big trouble.
Yeah, I'm in the same boat. I'm coming fresh to the team and I haven't written any Java in anger for 8 years, and there's Rx in here. I'm sure I could sit here and study it over and over until I get it, but why? My service is running at <10 QPS. I cannot believe the runtime value is worth the SWE onboarding time.
Promises are all over the place too. I understand why someone thought this was a good idea ("I'm doing the scalable thing!"), but in the end it appears completely unnecessary (e.g. doing a blocking get call in all uses).
The problem with Java isn't Java. The problem with Java is that it lets so many people hold it wrong, and the Java ecosystem encourages it.
Yeah the problem with Java and a language like Kotlin is it has so many language features. Meaning programmers can do lots of hard to understand things.
I've been coding in C recently and really enjoy the limitations. Usually there's only one way to do something.
We use RxJS in two of our video streaming products. In one of them we have a spaghetti bowl of streams. You can follow most of what’s going on, but not 100%. And it hurts. Architecture, patterns and naming conventions hamper readability and understanding a lot.
In the second codebase, things are a bit simpler. The architecture allows for more readable code and far fewer cases where multiple streams might introduce loops or race conditions due to being in separate files and initialised at different times. We felt the benefits of a v2 build ;-)
I’d say RxJS is a great tool to help respond to multiple concurrent data streams (e.g. user interactions happening at the same time as video playback). If you’re not dealing with a lot of realtime behaviour, you probably don’t need it.
In my opinion the complexity ramps up massively if some consumers want a reactive streams pushing data to them and other consumers prefer to request a current static value of the same data. Watch for this and experiment to find comfortable solutions or acceptable compromises.
If you think you need RxJS, take a look at Redux + Sagas and consider a choice between the two. I prefer the rigour and dev tools for Redux + Sagas over the freedom to be hairy with Rx. But observable streams are a lovely abstraction.
RX definitely requires some non-standard thinking to do well, but any competent engineer can get it if they want to dedicate the time to it.
But you're right, the same things can be accomplished in other ways with less up front effort. But where it really shines is when you hit massive scale. Having RX and its concepts in place ahead of growth makes the growth easier.
I think this is where things go wrong. I used RxJS plenty and tried even spearheading its usage at times in some of our apps since it could cleanly solve some problems in what I thought were elegant solutions. I quickly abandoned this after realizing that the upfront ask in terms of dedicating time to learning it for some people is just too much for them to handle.
I'd say a decent chunk of engineers just were not competent enough to get it in the first place and then there were those who just refused to put in the time to learn.
i.e. there are many engineers who I truly believe do not even want to put in the effort to be competent in the first place. That or false competence takes precedent as well.
If you're doing worker threads with multiple callbacks to listeners whose lifecycles are properly managed to avoid memory leaks then Rx will probably be simpler. If you're not doing threading or if your listeners only receive one callback then Rx might be overkill.
This is mostly off-topic I suppose but I recently noticed that Netflix, and Netflix on my TV(WebOS) specifically is absolutely spamming my router with DNS requests sometimes several times a minute, most of them for nrdp-ipv6.prod.ftl.netflix.com. I'm not blocking them btw, so it's not some buggy retry thing going on.
Even if I do a hard reboot of my TV and don't start the app, my TV is still happily resolving away. I'm gonna have to set up a local cache or something just to save on my NextDNS quota.
Edit: I just asked Netflix support about it. Worse than useless, just kept telling me to either try a different internet connection(no idea why), then they told me to take it up with my ISP and immediately closed the chat. I'm gonna wait an hour or so, open another chat, and tell them my ISP said this was purely a Netflix issue, I suppose.
Not my area of work, but pretty sure that is expected behavior to resolve an ever changing set of services being called out to. Better to do dns resolution than for a call to fail and have to re-resolve while you wait on a loading screen right?
It looks like the TTL on that address is 24 hours, so likely a problem of the dns resolver, which would point to webos. Have you tried a factory reset recently? (Or root and diagnose the system internally)
My system is rooted actually, but no idea how to disgnose this. I can SSH in but it's a very limited busybox setup and everything app-related seems to be encrypted.
> absolutely spamming my router
Ah, how many k/second?
> sometimes several times a minute
WTF?
We're talking 0.00003 thousand requests/second? Most apps do several DNS queries per minute. Otherwise every DNS update would require >1 minute to propagate. Slow DNS updates is a massive pain in the ass.
FWIW, ISP DNS servers and cheap home routers are both capable of caching DNS responses, updating the cache when they get a query for something that's about to expire, and serving thousands of queries per second.
> then they told me to take it up with my ISP and immediately closed the chat
This is the correct response. DNS is usually provided by the ISP via DHCP. If their server can't handle "several times a minute", the ISP would be at fault 100% of the time
My DNS service isn't having any problems, that's your imagination. Everything works. It's just filling up my quota. If "every app" is doing this, why does Netflix account for close to half of all DNS requests on all my devices?
I explained to them that I'm not even using ISP DNS. They still told me to talk to the ISP. Still think it was the correct response? It was obvious they didn't even know what DNS was, they were just slavishly following a script that said DNS resolving issues -> ISP. They didn't even grasp what I was reporting to them.