What New Twitter Employees Do For The First Week
quora.comMy "new hire" treatment as government contractor was so bad I almost quit. Even though it's a huge organization there's no process at all in place; you're just thrown to the wolves. Basic things like where and how to get my security badge and car tags were not covered. I was not given any input about what was expected of me, but I found out about things I was supposed to do only when I was getting yelled at for not having them done. Everyone I work with are of the baby boomer generation, while I am a millennial, and instead of mentoring me they excluded me. Often I would get in trouble for missing meetings because the location of the meeting was spread by word of mouth but no one would come by and tell me. My first project was to port some code written by a much older engineer, but this engineer believed that he had job security if he were the only one who knew the system. Our boss told him "You WILL let him shadow you and read your code," and he shot back, "He will NOT shadow me and he will not see my code!" For the first few weeks, the only copy I had of the code I was supposed to port, I got from snooping in his trash when he threw away a CD (old hacker trick :) ).
Things got better eventually but it was definitely the worst way to treat a new hire.
Huh. I had a government job in highschool; I went to a poor school with a big 'occupational program' where you get one or two periods off and some school credit to go to work, you know, to help the kids who are thinking of dropping out get a diploma. It was pretty sad because there were mandatory class days where they showed you videos about how to be a better retail employee; I can still tell you the steps they taught us to spot short-change scams and shoplifters. It's sad because they could have just as easily spent some time talking about various trades one could learn by becoming an apprentice. I had a job at the local computer repair shop the summer before[1] where I obtained some contacts, so I became the IT guy for the local county health department.
It was a really great experience for me. I was about 16, and I think 40 was pretty young for the office. I don't know if I got much by way of mentoring as an IT guy (well, later on they hired another guy who was way better than I was and I did get some mentoring, but for most of the time I was there it was me and the office manager, who probably knew less than I did about most computer things) but there was a lot of /very important/ social mentoring I got. I learned how to (at least kind of) act like an adult, or at least how to interact with adults in a healthy manner. On a social level, the semi-technical guy who hired me was one of the best people managers I've worked for.
I think it was an all around win/win; I got treated like a human being (which feels /incredibly special/ to a high school kid.) and I learned how to more productively deal with less-technical adults, and I got out of some school. They got a reasonably skilled IT monkey for minimum wage.
After I got out of high school, of course, it was .com time, and I have not had reason to look back at government work since, just because I don't value stability that much and there's little chance I'd have the patience to wait it out long enough to get the pension. I still look back somewhat fondly on my time working for the state. (Or rather, the county.)
[1] my stepmother made me quit that job 'cause they had no workman's comp and I was being "exploited" for slightly under minimum wage. I was 15! it was far less dangerous than anything else a kid that age might reasonably do for fun, and they gave me discounts on used computer parts for the next three years, so I thought it was great. Anyhow, it worked out okay, but it's another side to "exploitation"
Contractors are the lowest scum in the pond in the Government (they all know your making twice what they do). "Real Employees" get treated a little better, but you'll often find the same indifference.
I work in a French administration. We have a special day, the "journée des nouveaux arrivants" ("newcomers' day") for new hires, where we are taught the history of the service and learn about the different departments. The only problem is, since there are few new hires and organizing it requires the animators to be here, that this event takes place about twice a year.
I was lucky, my manager thought about it and managed to get me to do it after a bit less than a year. Some people have been working here for years without the "new hire day" (in all fairness you don't learn a lot of practical things, but still...).
Gov jobs are great! When I was a contrator I saw ladies browsing internet, playing computer games (like Sims), streaming video. It was fun!
I did some work for the government while in college. While I'm glad I did it, as it gave me a purpose and venue for my essentially self-taught advanced coding, I would never do it again.
I did work for the city gov't too while in college. It was a strange experience, being the only computer intern working for the Dept. of Agriculture, giving my end-of-summer presentation about JS and Access forms to a room full of other students who spent the summer hiking around farm fields taking measurements.
Wow, how strange.
How can over 100 people be needed for basically a scaled blog system that consists only of 140 character titles (not even any article content) and RSS feeds? It doesn't even have a search feature (past the previous week) or html, or tags, etc.
Doesn't WordPress.com do it with like a dozen people and their system is way, way, way more complex?
update: no, wait, in December 2010 they announced it's over 350 people now
http://blog.twitter.com/2010/12/stocking-stuffer.html
What on earth are they all doing?
Handling on average something like 76000 status updates per second (much higher when something interesting happens in the world), each of which must quickly become available to anywhere from 1 to a million clients.
Wordpress.com is trivial compared to Twitter.
You're an order of magnitude past their peak load - New Years Eve they set a new record of almost 7000/second.
http://thenextweb.com/twitter/2011/01/06/new-years-eve-set-a...
I don't believe Twitter is trivial, but I think they're perceived as more complex than they are - WordPress + Reddit + Heroku only have like 10% that number combined. Apples to oranges, but those 3 companies would be doing more everythings per second than Twitter when you combine them.
As has been pointed out elsewhere on the thread, complexity doesn't scale linearly. It's far easier to write ten sites that each gets 30 qps than it is to write one site that gets 300 qps, and it's easier to write ten sites that get 300 qps than one site that gets 3000 qps.
Twitter's fundamental problem also is a harder one to scale than something like Heroku or Wordpress. For those hosted sites, you can shard easily by host, so that each of the 100,000 Heroku-hosted sites can get its own EC2 instance(s) and behave pretty much independently. You can't do that when the point of your site is that any action might instantly be broadcast to thousands of followers. High-fanout writes are not an easy problem to solve.
But if you design it well enough, it does. Here's a copy-pasta from a reddit comment of mine from a few months ago: http://www.reddit.com/r/programming/comments/b2u6t/twitter_o...
------------------
Color me unimpressed.
At some point, I was collecting 40GB/day of financial data (and that's after bzip2ing them .. probably 200GB/day before); This was done on hardware costing $30K (which was two equivalent machines with 4GB, each having 20*1TB in a raid configuration -- this was a hot-backup configuration) and the operation run (coded, supervised, administered) by 2 people.
I'm extrapolating from your numbers: Let's say you have 70GB over 14 days = 5GB/day; Let's assume Twitter has 100GB/day of text twits (which, incidentally, means ~1 billion tweets per day which I highly doubt, as they took few years to get to the 1 billion mark, and last I heard they were at less than 100 million twits/day)
Then, at this day and age (numbers selected for 2 years ago, when they had their last infrastructure revision), what you do is buy 20 servers with 8GB of memory each (for, say $5K each), plus a little redundancy, and store all the latest twits in memory, and the most popular user's older twits as well; everything else on disk. Throw in cheap web front-ends that don't even need a local disk, load balancing, and a gigabit ethernet backplane. You're still under $200K in equipment.
Yes, the code is not going to be trivial, but for $100K and 3 months you can get a stunt programmer (I know a few who can do it and won't charge as much).
A run-of-the-mill RDBMS is the wrong tool for this job; Basically, run of the mill anything; but that does not make it incredibly hard.
I think $300K for hardware and software, you can get a Twitter clone that performs as well.
Twitter is successful, but that's not thanks to good engineering.
Be careful extrapolating success scaling in one domain to success scaling in another.
I've also done the 40 GB/day of NYSE TAQ data financial analysis thing, and the 1000+ trades/second real-time financial analytics thing. And I work on Google Search, and have a passing familiarity with how other Google products scale.
The scaling challenges of batch financial models vs. real-time financial processing vs. information retrieval vs. email vs. social products are very different. Even going from a model of the web where it's static and changes every few months (like Google of 2004) to one where sites get update every few minutes and users expect to see the updates immediately in search results (like Google of today) requires vastly different technology.
The main thing about scaling that I've learned from working at a couple places that require it is to go into it with a fresh mind each time, and really pay attention to what the requirements are and what you can cut corners on. There're some general principles you should know (eg. Jeff Dean's "Numbers you should know", memory is much faster than disk, cut out layers of abstraction that you don't need), but in order to apply them effectively, you really need to pay attention to the details of your problem domain.
If you think you can solve Twitter's scaling problems, they're hiring, they're pre-IPO, and they're probably giving out decent chunks of stock.
> Be careful extrapolating success scaling in one domain to success scaling in another.
I agree about that.
> go into it with a fresh mind each time, and really pay attention to what the requirements are and what you can cut corners on. There are some general principles you should know: memory is much faster than disk, cut out layers of abstraction that you don't need, etc - but in order to apply them effectively, you really need to pay attention to the details of your problem domain.
(slightly edited) - This is golden.
However:
> If you think you can solve Twitter's scaling problems, they're hiring, they're pre-IPO, and they're probably giving out decent chunks of stock.
I know I can solve Twitter's scaling problems (I don't think the solution I posted is the end-all-be-all, and for all I know that might not be where their scale problem is -- it is just perceived and argued about this part, which is not very hard).
However, Twitter's abysmal uptime (for the kind of sevice they are providing) had no bearing on their growth in 2008-2009. And even if by re-architecting Twitter they can save $2M/year on operations, it would be dumb to do that before they're in the black for a while and can identify their real profit and loss centers.
Also, those stock are not worth quite as much as people think when you take everything into account. (I've got a successful exit as a non-founder behind me; I'm intimately familiar with all the gory details including taxes, dilution, etc -- Whether options or RSUs, if you are granted anything of value, you have to pay full taxes AT THE TIME OF THE GRANT).
My point was only to show how non-impressive the problem twitter is (supposedly) facing. It's a repeating discussion:
Don't know why I even bother anymore. A company that had (maybe still has?) their millions-of-views-a-day pages created in Ruby doesn't care about solving scale issues.- Twitter sucks - No they don't, they do xxx and it's damn hardAt Google, you guys throw out closing paragraph tags from the main page when it is clear it renders fine without them.
Yeah, the number I had was for total API calls (6 billion a day), not just updates. http://techcrunch.com/2010/09/17/twitter-seeing-6-billion-ap...
Handling updates to 7000 user accounts per second isn't what makes it hard for twitter. It is the propagation of these updates to a gazillion followers that taxes the system. Their primary overhead is the messaging update to followers. Ashton Kutcher has 6 million followers. Sending instantaneous notifications to all those million twitter clients is what makes it a killer.
TLDR; It is not the vertices in the twitter graph but the number of edges in it that is unprecedented.
Unprecedented, except is it really unprecedented or are we just looking at it from the sexy social startup whatever must say graph and also disrupt angle, where everything is new and super hard and nobody has problems that have ever been solved before.
RSS strikes me as an almost identical process other than the time subscribers wait to check for new content, and there are feeds with millions of subscribers. So now the problem is reduced to doing it in a timely fashion.
This is why I wonder if they're really that immensely complicated - isn't Google doing much the same thing, and possibly even at a bigger scale at one point, with their reader and FeedBurner?
"isn't Google doing much the same thing, and possibly even at a bigger scale at one point, with their reader and FeedBurner?"
Another company with a ton of employees.
Probably not that many on the FeedBurner team itself, but a lot of people at Google working on the general problem of keeping things unreasonably responsive at massive scale.
This is exactly the reason why every IM network has a limit to the number of contacts that you can have. I recall reading several years ago that this was an incredibly hard problem for IM providers to solve, and AFAIK none of them have allow you to have an infinite number of contacts.
Yes but that's handled by hardware, not by people, and I am sure only a few critical coders handle the software performace.
WordPress.com is not as trivial as you might think http://en.wordpress.com/stats/
Certainly it's just a matter of scaling once you hit a certain level of volume, you just have to be able to bring more servers online into the grid.
Scaling from 10,000 users to 1 million is probably very hard.
Scaling from 1 million to 100 million, well you better have a pattern down that works with easy hardware replication (like google does).
Understand that Google is continuously rewriting their infrastructure to handle increased scale. That's what they have 25k employees for.
Jeff Dean's rule of thumb is that you should build a growth factor of 10 into the design, but any more than that and you will probably have to re-architect anyway. So going from 10,000 to 1 million and 1 million to 100 million are probably roughly equivalent in difficulty.
How many of those 25k employees actually handle any of the infrastructure scaling?
A fairly large percentage of them, and many of the ones whose direct job responsibility isn't infrastructure (like me) frequently have to deal with the consequences of building for scale as they develop features.
The big difference between blog hosting and things like Twitter (or Reddit, or Digg, etc) is that blogs are independent, so adding servers scales you up linearly. When you are looking at X's blog, basically everything you see is coming from one server.
You will have to have something that deals with mapping URLs in the unified logical name space of your site to the individual servers that the particular blogs on--that's the part that you can't just throw machines at and get good results.
With the social sites, you can't isolate things easily, because what a given person sees at any time is drawn from an ever changing set of content from other users, with each viewer drawing from a different set.
Based on the stats on the site, I calculate that Wordpress.com is averaging about 10 writes and around 8800 reads per second.
edit: Correction, ~880 reads per second.
Looking at the posts per day detail I think it's more like 30 per second with bursts that probably go to twice that, while nighttime is idle depending on timezones.
But I agree it's a fraction of 7k/sec peak for twitter.
However, twitter does not have to parse html, has no plugins to execute or templates, and has a max string length of 140 characters which is trivial.
Each post/comment published on wp.com takes many, many more cpu cycles than twitter.
I agree that Wordpress requires far more CPU time (I run a network of Wordpress blogs).
My figures: 900,000 transactions (500,000 posts + 400,000 comments) / 86,400 seconds per day:
~= 10 transactions per second on average, though like twitter, activity probably has a power law distribution corresponding with US daylight hours.
Reads: 2.3 billion / month Suppose a 30-day month, that's 86,400*30 or 2,592,000 seconds per month.
2,300,000,000 / 2,592,000 ~= 887 pageviews per second.
I was incorrect by an order of magnitude, though the same uneven traffic patterns caveat applies.
Just to put that into perspective, right now, you can buy off the shelf from Tibco an appliance for routing 10 million messages/sec (at an average message size of 50 bytes).
What Twitter does is not hard, and they do it badly.
Having used Tibco and having written high-performance message routing services for investment banks, I'd like to say yes it is hard, and Tibco does it badly.
I'm not sure where you got your numbers, but based on it, there are 6.5 billion new status everyday. I don't think it's quite accurate.
Twitter.com is probably one of the great engineering feats of todays social Internet.
You tweet from your mobile phone app (maintained by twitter) or twitter.com, and that tweet gets posted on your public timeline. The tweet is parsed for mentions, and it is also copied on to the mentionee's incoming timeline and their mentions timeline.
The tweet is parsed for hashtags, and active searches (i.e. ones open on twitter.com, or via the APIs) and is also copied onto those timelines.
Also, you have followers. So Twitter also copies the tweet onto the inbox timeline of each of your followers.
Not only is all this information published in real time on Twitter.com, but it is also made available via JSON API, Streaming API and Firehose API.
All tweets (on all timelines) are stored forever.
Twitter's scale is hard to fathom - all of this processing is way beyond what your Rails / Django app could process with a MySQL backend. MySQL replication wouldn'e even come close to keeping up with the sheer volume of events to be processed.
To make matters more complicated - Twitter is expected to scale to meet the demand of emerging world events (eg: Egypt, Iran, snowstorms, hurricanes, earthquakes, bushfires). These events don't evenly spread traffic across Twitter's network, but instead provide "storm surges" of localised intense traffic.
Oh, and Twitter haven't just launched an analytics product?
I think for sheer engineering at scale, there are maybe only about half a dozen other companies in the same league.
(Edit: grammer changes for readability)
A lot of companies deal with traffic of this magnitude. UPS perform 12 billion transactions per day through their systems - DB2 on Sysplex'd mainframes. Finance companies are currently dealing with tens of thousands of messages per second with much more complex handling and forwarding rules than Twitter. And, of course, in terms of page reads, Google and Facebook leave them well behind.
Twitter has become impressive, but let's not overstate its achievements.
Sure but UPS, Google and Facebook have a lot more than 350 employees.
Not an apples-to-apples comparison. UPS, Google and Facebook have much less concentrated product/service offerings (ie, they each have >1).
> Not an apples-to-apples comparison.
So why did you make the comparision?
I am just pointing out that ck2's summation of Twitter understates the complexity of the problem they solve.
> So why did you make the comparision?
My point was that twitter's scale and problem complexity are not unprecedented and that small, motivated teams have solved them before. Parallel queries and transactions for DB2 on z/OS were developed by small teams in Poughkeepsie, NY and Perth, Australia. Google's infrastructural software was designed by a small group of very bright people, and so on.
If we compare the groups actually working on the problems then I expect that twitter will have comparably small groups of engineers directly facing the scaling problem. And I also repeat the point that bigger problems have already been solved. Twitter's issues are not unprecedented if you are prepared to look outside the Journal of Stuff I Remember Seeing on Highscalability.com.
I don't disagree with any of that, except that no one has claimed that Twitter's issues are, "unprecedented".
Once again I will reiterate I am just pointing out that ck2's summation of Twitter understates the complexity of the problem they solve.
Great "social internet" Engineering feat, eh maybe. Compared with other real-time & data intensive platforms, not even close. Electronic finance assets exchange platforms are a order of magnitudes more complex.
That's a good breakdown of the processes but I am positive all that is done in stages and queues.
So with a good design you have groups of servers doing the different stages in the queue.
One you've got the pattern down for 100 tweets per second, the pattern should be reproducible by scaling servers in each queue to 1000 tweets per second, and eventually 10,000 tweets per second.
The database requirements may explain why it's all done in one datacenter instead of trying to do replication across the country/world.
In the research literature this is called SEDA: http://en.wikipedia.org/wiki/Staged_event-driven_architectur...
Once you hire over 15 people, you need to start hiring people to manage the HR issues that it brings, and then once you have 20 why you need middle management, and at 40 you need someone to write internal tools and manage hardware etc... then for every 40.... you get the point. Past a small team, employee count snowballs.
Quora to the rescue:
http://www.quora.com/Twitter-Inc-company/Why-does-Twitter-ne...
Wow, very nice find. I took a stab at the question via Google but gave up.
I also discovered they have their own (but singular) datacenter:
http://engineering.twitter.com/2010/07/room-to-grow-twitter-...
But they are planning on adding more, which leads me to believe maybe that's what some of the new hires are for.
Wow, very nice find. I took a stab at the question via Google but gave up.
Exactly why quora is compelling. If you didn't know now you do
If you ever wonder about why a company needs X developers, please come back to this list.
Huzzah, I asked this question!
One thing that CEO Dick Costolo has said is the amount of technical debt we have (http://kara.allthingsd.com/20110117/full-dces-interview-vide...). The scale is simple tremendous, and you've got to remember: Twitter isn't just a website; it's an SMS service, clients and an API.
I initially upvoted you because I agree - it is mind boggling but really I think the reality is that scaling at levels like that is a much bigger job/problem than you/I can imagine. Also I doubt all of the people are technical?
I just have to start wondering when it's over 100 though and now they are at 350.
They don't have an ad sales force right?
They don't have a billing department right?
Maybe they have housekeeping and food prep people included in that 350?
I honestly don't know and my instinct could be very wrong, but 350 seems crazy high.
They do have an ad sales force. http://bits.blogs.nytimes.com/2010/08/24/new-hires-at-twitte...
iPhone, Android, and Mac clients. Maintaining as well as improving the web interface and the API. Lots more scaling - people check Twitter way more than they do Wordpress-hosted blogs.
I'm sure there's more, this is just what came off the top of my head.
Wasn't their iPhone stuff a single guy who had a nice/leading client they bought last year?
Yup, Loren Brichter, who wrote Tweetie. Twitter bought his company, Atebits, and hired Loren to continue working on the iOS Twitter client.
That client would then touch a lot of other people though, even if it was primarily the one guy. Changes that are rolled out on the platform would need to be considered in terms of this app along with the all other platform apps.
Sure when independent a single guy can sit there and create something like that solo, inside Twitter though there would be a lot more process and interaction to achieve the goals.
I upvoted you because I have no technical proficiency in the (engineering) challenges they face and would love an explanation by someone from Twitter about their apparently ridiculous employee numbers.
Well you need 100-200 engineers just to try and make Cassandra scale. The rest do the production site with more proven technologies.
Despite the facetious nature of the comment, there's a kernel of truth there: Twitter actually did hire 3 people just to work on Cassandra.
Quite a contrast from the purported Honda "onboarding" process (or lack thereof). They basically let you figure out where you'd be most useful, and no one really tells you what to do. It seems to scale well, I suppose.
A funny thing happened on the way to the solution. Try and picture this. On my first day of work, no one told me what to do. On the second day, the same thing happened, and on the third. That’s as much as I could take. I decided to meet with everyone I was coming in contact with to find out more about their individual talents and personalities, and to find out what was going on. Before I knew it, I was developing a picture of how things really were, and who needed what, and I became creatively involved in defining my own participation in relation to the skills I could bring to the table. In the process of doing this, I had complete access to everyone in the company, from other newlings to the President. Nothing but open cubicles no higher than 3 1/2 feet. I was allowed to learn, interact, and find solutions to every problem and need I recognized. I always found something important to do, and it became natural to provide effective solutions as needed. I am not a very unique individual, but I am effective because I am allowed to be. I also know it may be different for some people, experiencing this kind of freedom. I know that some people are petrified by this kind of freedom, and equate it to abandonment, and it drives them crazy not knowing what to do. I also know that even under the best of circumstances, people become sedate sometimes and settle in to patterns of repetition for false comfort. The answer, then, is to have them all switch places every few years, no matter how well they may be doing their job, because it is just as important to let everyone see their own position from someone else’s position. It also allows for the surprise of finding how much fun change can be when your creativity meets a new challenge. See what you end up with. It’s either this, or that.
More here: http://aditya.sublucid.com/2008/11/20/let-your-employees-fig...
My first software dev job entailed building and configuring a LAMP stack. It wasn't much to deal with: some perl, some bash, some administration, IP filtering, code deployment, the basics.
On my first day, the person responsible for the project went on vacation and for next 10 business days, I was told to do research. That's about it. It was the oddest startup experience I have ever had. Other members of the group, asked me questions about how everything was going, but couldn't answer any of mine. So I indeed did research.
Seems like kind of an overwhelming amount of attention.
Many people would muddle there way through things trying to work stuff out that would be easily explained to them. If no one has engaged the person to explain it to them, they will assume that they should just know it and be afraid to ask. This ensures everyone gets up to speed early, probably looks like a lot of time spent at the time but probably saves a lot of time down the track.
for some reason this feels a bit skimpy on the details compared to the "first month at facebook" (or may be 6 weeks) item that came about a few weeks ago...
What does the engineering onboarding involve?
Talks about the codebase, about the system as a whole, also about specific sub-systems, about how to think through some things and, maybe, about recent design decisions.