Adopting Microservices at Netflix
nginx.comNetflix doesn’t have an HR manual. There is a single guideline: “Act in NetFlix’s best interest.” The idea is that if an employee can’t figure out how to interpret the guideline in a given situation, he or she doesn’t have enough judgment to work there. If you don’t trust the judgment of the people on your team, you have to ask why you’re employing them. It’s true that you’ll have to fire people occasionally for violating the guideline. Overall, the high level of mutual trust among members of a team, and across the company as a whole, becomes a strong binding force.
Christ on a bike, that sounds horrific.
All this approach can do is replace codified guidelines with unknown, arbitrary ones. Who defines what "NetFlix's best interest" is? In an organisation of that size, conflicts of interest must happen all the time.
Is this just a fancy way of saying "If we don't like you, you're out of the door?"
G'Day, I work at Netflix, and we hire what's been called "fully formed adults". Here's a quote that can explain it better than I (also see the full article):
Source https://hbr.org/2014/01/how-netflix-reinvented-hr/ar/1:
"Hire, Reward, and Tolerate Only Fully Formed Adults
Over the years we learned that if we asked people to rely on logic and common sense instead of on formal policies, most of the time we would get better results, and at lower cost. If you’re careful to hire people who will put the company’s interests first, who understand and support the desire for a high-performance workplace, 97% of your employees will do the right thing. Most companies spend endless time and money writing and enforcing HR policies to deal with problems the other 3% might cause. Instead, we tried really hard to not hire those people, and we let them go if it turned out we’d made a hiring mistake."
I trust myself, and everyone on my team to act in Netflix's best interest. Throughout my career, I've worked with many professional engineers who could have been trusted to work in the company's best interest. But there have been many times when we've been prevented to do so, due to process. So I've been fascinated at how Netflix is operating, and it's been great working here.
It's important to know that a key role of management is to provide context to employees: what problems exist, what challenges we are facing, what opportunities might exist, what's important to Netflix right now. So that we know what to do, and how to exercise judgement.
I've written about working at Netflix earlier this year on my blog (someone else posted a link).
I think rather than "fully-formed adults," what's described here is mutual compatibility. Which is kind of unfortunate, as weird as that sounds.
We'd all like to work with people who operate with the same understanding, ethos, sense of humor, etc. It makes the day better, right? No office drama, no awkward meetings. But I would contend doing this tends to put you at a genetic disadvantage, and that some types of confrontation are essential for growth. Introducing only like-minded people will produce linear results.
What codified HR rules do is put some workplace guidelines around humanity - that allows a dissonant group of varied people to, you know, be themselves without necessarily worrying about conforming to avoid ruffling feathers.
Perhaps it's more nuanced than that, but it sounds more like "hire people you like" expressed somewhat condescendingly as it is.
Fully-formed adults does not mean like-minded people, which is a different topic (groupthink).
Expressing competing professional views is indeed healthy, and it's part of our culture deck: http://www.slideshare.net/reed2001/culture-1798664
slide 13: "You challenge prevailing assumptions when warranted, and suggest better approaches"
slide 14: "You say what you think even if it is controversial"
I got something very different out of it - it reminds me of Auftragstaktik, or leading by mission. Instead of adding explicit restrictions on employees, give them freedom of action. It takes trust that they're the kind of person that would make the kind of decision you want. In other words, having a shared "adult" background.
My employer treats us as 'fully formed adults' also. It's fantastic, especially after the years I spent in government, where we were all treated like children who required policies on every damned thing to guide us.
An obvious example is about how you use your computer. In my old job, it was highly regulated. Want to install Firefox on your machine? Sorry, that's against IT policy, we only support IE. Your machine is locked down to prevent unauthorised installs, and even if it weren't, you'd be up for a warning for violating the policy.
In my current company, the sysops still limit what they support - for example, they're not willing to support OS X Yosemite users yet, and recommend users stay on Mavericks. But, if you want to upgrade to Yosemite, you can, just don't ask the sysops for help fixing it. And if your broken Yosemite install is impeding your work, well, we trust that you'll resolve it one way or another.
Or another - alcohol. In the government, alcohol at work was banned. A beer at lunch could get you fired. When we had Christmas parties, alcohol was provided, but on a token-per-drink basis. Everyone got two tokens. My current company, the beer fridge is a treasured perk. There are no rules about how much beer you can drink, or when you can drink it. So far, no raging alcoholism has impeded work.
What I really notice is that when you treat people like adults, they respond like adults, and when you treat them like children, they respond like children.
>In my current company, the sysops still limit what they support - for example, they're not willing to support OS X Yosemite users yet, and recommend users stay on Mavericks. But, if you want to upgrade to Yosemite, you can, just don't ask the sysops for help fixing it. And if your broken Yosemite install is impeding your work, well, we trust that you'll resolve it one way or another.
Sorry, but as someone who works in IT security, this sounds like an absolute nightmare. Even if you have a small company comprised only of intelligent developers, those developers do not necessarily understand the latest malware threats or what sorts of software can introduce risks. Wide-open FTP servers and Tomcat servers with default passwords are a major issue. I would actually say developers probably introduce more threats into our environment than any other demographic.
Whitelisting software installs from specific domains (google.com, mozilla.org) is okay, but a carte blanche policy is usually a very bad idea.
But I would contend doing this tends to put you at a genetic disadvantage, and that some types of confrontation are essential for growth.
Fully formed adults have confrontations all the time. The difference is that the result of the confrontation is not threats, backstabbing or whining later. We've all worked with the engineer who thinks they are the smartest thing around and acts like a 2 year old every time they are challenged. That's the person I assume Netflix says they will not tolerate, and I agree with them.
I have heated discussions with co-workers all the time and at the end of the day we are all still respectful of each other and our ability to get the job done.
> Fully formed adults have confrontations all the time.
Sure. But without some HR codifications the line between "fully-formed" and "childish jerk nymph" becomes wholly subjective, and controlled by the majority. The end result is an organic uniformity.
Nobody gets along all the time. But how you define "fully-formed adult" can vary from one person to the next. Essentially you're defining "good and bad" or "right or wrong" by a feeling. I can understand the appeal of that, but it would worry me quite a bit as even a loose policy for behavior and interaction.
Presumably, if one person called another a "childish jerk nymph", then you have at least one person to fire (the person resorting to name-calling), if not two (the person maybe deserving the label) or more (the people taking sides rather than trying to dissolve the dispute.)
"Fully-formed adults" are people who I would expect to know how to mediate themselves. If people aren't able to successfully mediate their own disputes, then your failure in hiring came long before your latest addition.
Examples of people likely to be fully-formed adults: a mother with multiple children. A schoolteacher. A military sergeant. A nurse. A bartender. In general, people who have been exposed to enough pointless complaining and dispute that the object-level arguments don't matter to them any more, relative to the issue of figuring out what will allow everyone to continue working together optimally.
Note how most of those jobs are service jobs. People with natural talents rarely have to grow up in this way, and the people with the most natural talent are frequently the least grown-up (rock stars, career academics, startup founders). You only see this "fully-formed adult" trait re-emerge at the highest levels of accomplishment: astronauts are frequently fully-formed adults, for example.
Even with codifications it is still subjective. The obvious things of no physical abuse, etc... are obvious. How do you codify don't be an asshole? In fact, simply saying act like a grown adult is probably the most clear I've ever seen an HR guideline.
> What codified HR rules do is put some workplace guidelines around humanity - that allows a dissonant group of varied people to, you know, be themselves without necessarily worrying about conforming to avoid ruffling feathers.
I hate to be the one who says this, but that's not what those guidelines do. They don't allow varied people to be themselves. They make everyone be a self-similar group of other people. Other people who are not themselves.
Codified rules also mean that we can figure out who is responsible, rather than this where the management can't be held accountable, as there is nothing to hold them accountable to. It's all on you.
Even when we account for HR being an organ doing the bidding for the parent org.
While this isn't nearly as bad as companies like Valve which go around claiming not even to have any managers, let alone any policies, it still means you're going to run into this:
http://en.wikipedia.org/wiki/The_Tyranny_of_Structurelessnes...
Having worked in similar environments with similar polities before, those sorts of policies tend to mean, "whoever has the most shininess in an argument" is the person who is considered to take the best policy, and frankly? innovation becomes stifled as the most political adult wins or stalls out the innovator who thinks further than others. Because in general, the members of the team are all adults, and we all understand the company, and we are all competent - so it comes down to who has the most appeal in the rhetorical wars. I find it appalling - extroverts and manipulation tends to win.
IOW, I'd rather work for a structured hierarchical firm, where politicians brown nose managers and the managers firewall these people from the rest of the team.
edit: Hum, your name seemed familiar. I attended your LISA talk. You're brilliant and well known (witty to boot, as I recall. :) ). You, sir, will be listened to by default, so you will tend not to encounter the failure modes I described above. But do think about them, okay? Tyranny of Structurelessness really is a great read.
Yes, in prior jobs I've seen the extraverts and manipulators win, by use of their charisma over technical facts. Most engineers aren't like that, but a few are, and we try not to hire them.
It helps to have technical management, who are able to understand all sides of a technical argument, regardless of how those sides were presented. It also helps to have staff who all can communicate effectively, at least at some basic level.
Tyranny of Structurelessness is great. But I'd like to see a more focused analysis of tech culture and its failings.
You realize your company has an HR policy that is illegal in most civilized countries, because it is considered unethical for an employer to treat employees that way?
The issue here is that some (many) people need more guidance, because it's not their company, and they have no idea what it considers "the right thing". Should you hire such people, you can't just dump them like thrash because you can't be bothered to manage them properly.
Netflix is going to get into a truckload of legal issues if it tries to scale this practice internationally in countries with decent employee protection laws, which is most of the Western world outside the US.
Netflix sounds to me like a Randian cult.
I'd be curious where this is illegal in.
I don't think they are saying "we have zero policies whatsoever", but exaggerating the level of trust. For instance I am sure that new parents are given a fixed amount of paid leave.
I don't think it is illegal anywhere. It might be illegal for Netflix to fire people over "violations" of such a "policy" -- but I don't really think that'll be an issue either. It's not a policy that lends itself to firing people because some manager didn't like them (which, incidentally is illegal many places anyway). But if someone were to actually "work against the company's best interested" repeatedly, that'd typically be a firing offence anyway. That'd be stuff like not actually doing work, stealing, sabotaging and/or leaking company secrets etc.
I think it sounds great to not have a lot of pretend-policy that does no good other than pretend to cover the ass of inept management.
Sounds exactly like Github's policy that got them in trouble. Hopefully it works out better for you but somehow I doubt it.
I'm not sure what Github's stated policy was, but it's clear from Julie Ann Horvath's description of how things operated there that quite a few of her co-workers were far from being fully-formed adults.
This other quote seem horrific as well.
> A single specialist in Java distributed systems is managing the entire configuration without any commercial storage tools or help from engineers specializing in storage, SAN, or backup.
So this person is either a genius who has specialized knowledge in these areas, or it's been abstracted by the platforms people to the point where it's only Java knowledge. Either way, it sounds like this person never gets to take a vacation or be not on-call. Hopefully there's not just 1 person responsible for all of the configuration, it just takes 1 person to run it.
> Process prevents problems. At many companies, the standard response to something going wrong is to add a preventative step to the relevant procedure.
The linked section goes on to talk about HR process with the "Act in NetFlix's best interest". It totally avoids the concept of technical processes. I'm betting "best interest" is a bit more formal on the tech side. Bad process is process for process's sake. Good process serves a very important role for offloading what people have to remember, as well as knowledge transfer. From time to time, steps of a process become irrelevant because of other changes and it has to be pruned. Sometimes companies forget about this step.
If Netflix has ever seen a problem with their builds/deploys and added a test to check against that problem, guess what--they followed the standard response to update the procedure. It's not necessarily a bad thing.
Not to speak for Netflix on the storage part, but Cassandra can run against ephemeral disks in AWS, and is designed for it. As such, I think what they are saying they don't have to think about storage as much.
Yes, it is pretty arbitrary. It's not very different than how Valve works, other than valve firing and hiring less people.
Now, what you have to take into consideration is that it's written as a contrast to all kinds of other places that run a command structure where it's your boss' job to figure out what is good for the company.
I don't know about you, but I have worked in those environments before. They attract bad people, as doing what seems better for the company, as opposed to what is better for your boss, is discouraged. I think of one example in my past, where the team worked for two months on a feature we knew our in house customer did not want, and would refuse to use. However, it helped our boss' internal political status, so the team tolled, the feature was unused, and in the end, half the team quit. Our former manager, however, is still doing well, because what is good for the company was not what was good for his boss either.
I often get recruiting calls from that company. They have trouble retaining good talent. Imagine that.
So the Netflix approach brings in people that are more dedicated to the business than the chain of command. There's a lot of very good engineers there, and it's a pool that is often untapped outside of startups. It's not a bad strategic decision, IMHO.
Interesting how we're happy to talk about companies in a Positive light that do good things, but don't want to name names when it comes to the shit-holes we've ejected ourselves from.
Is that really interesting? I would be surprised if people were more eager to make enemies and potentially make themselves look petty than to make friends.
We expect freedom and transparency in all aspects across the world irrespective of personal/social costs associated with it. Isn't it? So why exception in this case? Just because it is unwritten and some sort of convention inducing hypothetical fear? Won't it help future employees from joining such toxic work places? Just a view point.
I don't know how life is at Netflix. But I don't read it like that.
I read it like I would want to work there in a heart beat. Because it seems that Netflix trust you and doesn't (micro)mange you on bullshit KPI's or something.
It seems to me that at Netflix, you are trusted. You are trusted that your work improves the environment. Your work is evaluated afterwards. There is a risk of things going wrong, but with good, responsbile, knowledgable people, this is not so high.
Part of this is that you take the initiative to set expectations and clearly discuss what you are planning to do and what you'll be working on so there is no misunderstanding. It's all about setting expectations and communication.
I'm actually quite curious how life would be inside this culture. Since you have so much responsibility and control, life should be quite bussy, but manageable.
The ideal would be that you just have a bunch of adult, professional people who manage themselves just by talking to each other, agreeing on how they want to work with each other to get things done and they manage / steer this process themselves just with open communication. No (micro)managers needed.
The key is the people and their values / culture. Very simple things may make all the difference. Being on time. Being prepared for meetings (reading the short memo upfront, maybe lookup some things or make sure that you have done the tasks required of you). Starting the meeting on time. Respect each others time.
A willingness to to share and help people understand how tings are setup and why (reasoning), document the stuff that is important. And you know what is important or are capable of just asking what is important for other people to know so what you document is relevant.
I can only dream.
http://www.brendangregg.com/blog/2015-01-20/working-at-netfl...
This is precisely what I was picturing.
If I can ever get my own business off of the ground, I hope to have it work similarly to what you described.
To be fair, the line about "Act in Netflix's best interest" seems to apply to expense account reporting (http://voiceglance.com/netflixs-anti-establishment-pro-cash-...), which seems like a reasonable rule. On the other hand, your basic critique rings true: everything I've heard about Netflix's internal culture -- or, to be precise, how Netflix likes to position its internal culture -- seems a little too self-consciously "high speed, low drag," to the point where even the laudatory article I linked to admits that "workers feel legitimately afraid of losing their jobs at all times." Yes, Netflix is full of amazing engineering challenges, and I'm sure compensation is excellent, but I doubt you can underestimate the psychological toll of working in an environment that makes stack-ranking look like a union shop.
This would seem to echo Hastings' Netflix culture slide deck published in 2009[1]. It's worth reading even if you disagree. The deck was the first time I had heard of a major company having a "no vacation tracking" policy. It was presented in the same context as the HR manual. The policy of having no policies is built on employee trust. Employee trust is built on the standard of only hiring top quality employees.
I've never worked for Netflix or any other company willing to make such bold statements so can't speak to the reality. It would be interesting to read a first-hand account from someone who has been at Netflix since the time such non-policies were enacted. At the time, it sounded like a fantastic experiment. I would love to see the results.
Kind of a tangent, but having worked at two companies that have a "unlimited" vacation policies, I can say that I detest the concept. It's basically an anti-policy.
There are two major ramifications of unlimited vacation policies:
1) It changes how you feel about taking time off. Rather than paid time off being something that you have "earned", you evaluate your PTO in comparison to your teammates - specifically most people try to make sure they are not taking off more than the average. The end result being everyone is subconsciously competing with everyone else to consume less PTO than average.
2) Since you have not accrued vacation time, your company does not owe you financial compensation for vacation time not spent when you leave. As someone that changes jobs every 3 or 4 years, this is more painful than it sounds.
If I ever decide to be in a position that affords me the opportunity, I think I would enact a "minimum vacation policy". It's still unlimited, for whatever that's worth, but employees will be required to take a minimum amount of vacation every year.
> It's still unlimited, for whatever that's worth, but employees will be required to take a minimum amount of vacation every year.
It's worth noting that in some sectors (including the one I work in), minimum chunks of leave are considered an anti-fraud measure. A non-trivial amount of internal fraud is uncovered when someone is required to take 1-2 weeks off in a block, which is enough to require someone else to stand in for them. At that point, oddities get called out.
There's an important related point there: regular vacations are the business process equivalent of the Chaos Monkey. Repeatedly having to do without employees for a short amount of time forces you to make your processes robust.
I've previously asked co-workers and even bosses who work too hard to take time off for exactly this reason.
Seems like #1 would be a symptom of having the policy without the culture to back it up. Was that your experience?
For #2, I've worked for several companies, only one of which had a compensation package for unused vacation. All the rest have been use-it-or-lose-it by the end of each year. Getting paid for unused time sounds like a luxury.
But I also recommend negotiating for more time off when you sign on, especially if you have several years of experience in the industry. Even if a company says first-year employees get X days of vacation you can still negotiate. Especially for cash-strapped companies, vacation may be the only part that can change.
#1 sounds like it could be solved with culture, but actually good culture works against solving it. Good culture means you hire considerate people and considerate people don't take more than their "fair share" (i.e. the median).
Yes, most are use-it-or-lose it, however, legally in some states you MUST be paid for the time unused (if you leave before the end of the calendar year.)
Re: #2: in some US states, if you are laid off you are entitled to be compensated for vacation days you have accrued but not used.
This goes to the point that time off is something you have earned. It is literally part of your compensation.
So, having a policy of "take only what you need when you need it" would be akin to giving everyone access to the same bank account that runs the company and trusting them to use only what they need and/or feel is fair.
The so-called "unlimited vacation" is manipulative to the extreme. It would be illegal in most countries outside the US, because it's effectively taking worker's rights away.
I don't disagree, but I thought this was an interesting tweet on the subject
"The more talent density u have the less process you need. The more process u create the less talent you retain.
-- Reed Hastings (@netflix)"
The second sentence I agree with, but not the first.
An 'untalented' developer is one who is perhaps slow to complete features, needs work on their design patterns, produces buggy code, etc.
Process isn't needed for these people, rather it is for people who do or try to get away with things like: expense excessive purchases, frequently lie about being sick to get out of working, speak or act in a lewd or unprofessional manner, etc.
Nobody is under the impression that mandating wearing a business suit (process) makes one suddenly better at programming (talent), the converse is equally absurd.
edit: punctuation
Process isn't wearing a suit to work. Process is having to fill out paperwork and present to a change review board to do things like modify a DNS record or install a service. Process is to keep morons from breaking what everyone else is doing correctly. If you have a team of people who know how to manage DNS in a way that won't explode, the request process be pared down to an email asking them to do it and trusting they will, them trusting you're not asking for no reason, and management trusting it won't jeopardize the system.
Similar to Scotsmen, a true process(tm) is whatever successful people do to get work done.
I agree this sounds on the surface like it's very liberal but in effect it's stifling. Since there are no concrete outlines the onus is on you to figure things out. That sounds great but then like you said, who decides "best interest"?
It's an ideological wet dream. It's like saying, you can make any choice, you have the freedom, just make sure you make the "right choice".
Not to mention what happens when netflix's best interest conflicts with outside interests.
This is exactly what happens in the situations I've been in. It's miserable when you don't turn your attention to greasing the political game daily.
I work for a company with a similar HR Manual. It simply reads "Use good judgement."
The company is Nordstrom, a retail store that has been around for more than 100 years and comes up often in "Best Places To Work" articles. It's pretty great.
I don't think so. This is similar to HubSpot's policy (and probably influenced) of "Use Good Judgement". You can read more about it in the culture deck[1], but it's incredibly liberating and means there is a culture of trust rather than rules and regulations. And in most places, employment is at-will anyways so outside of illegal reasons, you can get fired for any old reason anyways.
[1] http://www.slideshare.net/HubSpot/the-hubspot-culture-code-c...
The logical conclusion of your codify-everything approach is this: http://en.wikipedia.org/wiki/Work-to-rule
Every functioning organization requires faith in the people you work with.
This is similar to what people in those despicable countries are doing. In the name of "best interest for the country" people in power can do whatever they want.
"Best interest" is hardly a measurable or even agreeable term.
This seems like something where "when it works, it works great", but when it doesn't...
I can easily see just a couple rotten apples spooling that who could damage that whole system very quickly.
And the rotten apples get sacked, according to this post:
Would love to hear from any Netflix employees lurking around here.
> Is this just a fancy way of saying "If we don't like you, you're out of the door?"
I'm sure it is, but is there any company where that isn't the policy?
If Netflix's goal is to attract more easy-going employees and avoid paranoid engineers, it's working for them.
"If you ask any developer whether a slower development process is better, no one ever says yes. Nor do management or customers ever complain that your development cycle is too fast for them."
I've never known a developer not to speak up when the development cycle is too fast or a user who is not averse to the resultant buggy software.
You're confusing 'development cycle' and 'pushing code'.
Pushing code too fast is clearly a problem that causes bugs, and neither developers or users are a fan of.
The 'development cycle' includes the total time to implement something, including time taken to fix resultant bugs.
Where you can speed up the development cycle without introducing more bugs (eg. by implementing comprehensive testing and QA practices), then no one is going to complain.
I think you draw an important distinction, but I don't believe that either of our comments are incorrect. I was explicitly referring to a development cycle that is too fast. After all, developers are humans and not machines. Perpetually fast development cycles lead to mental exhaustion, missed vacations--bugs. Unless, of course, you exist on a team that is staffed over-capacity such that the absence of any single developer has an immaterial effect on development cycles. I suspect that is not the case for most of us.
I've had a customer complain that I was being too productive for their team of business/design folks to keep up. I literally got pulled aside and asked to slow down. I think it was making them look bad.
Faster dev cycles can be safe, even safer, as long as the changes are accordingly incremental. See the whole "continuous delivery" line of thought.
What I'm missing is actual examples of services split in micro services, with particular focus on the db (originally one db, then split in multiple dbs, one in each micro service).
From the talks I kind of inferred that sharing is a good example of micro service, but that's also a good example of something that one could outsource to addthis, sharethis and similar (at least, I'm speaking from the perspective of a relatively young startup).
It's interesting that this is an advertorial for nginx ... I'm currently working on implementing a microservices system, but have instead opted to use ZeroMQ for the connections, and utf-8/json for the request/response interfaces... Which in my mind seems a bit more sane once you reach behind the publicly accessible API (though also means that you have to wrap anything the public/website needs.
I don't understand the dichotomy between speed and efficiency. What's speed is the author talking about, and whose efficiency?
The submission title should include 'Lessons for Team and Process Design', as it's a part of a series.
I agree, the title was shortened after I shared the link here.
This looks like a thinly worded ad for Nginx Plus.
I think it is thickly worded. Entire sections have nothing to with the technology you choose. I particularly liked the culture section.
Sure, it's called content marketing and there is nothing inherently bad about it.