How We Make Trello
blog.fogcreek.comI'm more interested to know the actual technical details of the build and testing process than the "Task-List" software development based approach because that can be done by any software project management tools (even JIRA).
What's missing in this article is the whole Continuous Delivery technical aspect of it.
What do you guys use to build the NodeJS app?
What do you guys use to test the NodeJS app?
What do you guys use to check the code coverage of the NodeJS app?
What do you guys use to test the front-end?
What is the automated testing strategy?
How do you store artifacts of builds, schema migration (if using RDBMS) or handle different model versions, how do you rollback (what's the rollback strategy)?
I hope Doug writes a more technical post about the multi-client build/release process. We have unit tests for the whole API and very few automated tests for the front-end. Client builds are stored in S3. We use mongoDB and will do backfills if necessary (pretty rare). Rolling back the client is just pointing the stable branch to another build.
Bobby, you said every app is a client of the API. I notice Trello.com consumes API from https://trello.com/1/xx while an OAuth client (from one of your Jsfiddle examples) consumes from https://api.trello.com/1/xx?key=xx&token=xx. I suppose the former just passes through to the later? If so, does the former needs to pass over the key and token (I suppose you can generate on the fly based on auth cookie)? I'm trying to get my head around the fact that your app provides OAuth for other clients and at the same time (from what you're saying) your app is also one of such clients. Not sure how it really works.
If you control both the OAuth client and server (that is, the client is always trusted and doesn't need manual authorization), you can dispense with request tokens and just issue access tokens directly, and forget about the whole authorization flow.
Trello dev here.
The web client (trello.com) has a cookie that has the auth information. The actual code executed for both examples is identical.
api.trello.com is just a CNAME to trello.com, in case that difference was confusing you. The authentication part is taken care of differently depending on the kind of tokens we get (the web client uses a cookie) but we turn that data into a standard authentication object and the rest of the route uses the same code regardless of request type.
Got you. This is very helpful, thank you!
Please ask Doug to write that ASAP :D (just kidding).
NodeJS community (and to some extend, the RDBMS/NoSQL crowd that wonder how to support different version of the model) would probably learn greatly from that type of article.
er, by stable branch, i mean stable _channel_. or whatever channel.
I've sent this to a couple of people around our multinational already. While I get what you are interested in, I have to commend them for this excellently written article. Would that our actuarial models were maintained with anything approximating this level of sophistication.
disclaimer: my entire life is run via trello.
Is it actually NodeJS even?
Yup. More about the tech stack here http://blog.fogcreek.com/the-trello-tech-stack/
Yes. Check out the interview the Trello team did on the NodeUp podcast: http://blog.fogcreek.com/trello-server-team-on-nodeup-podcas...
Fog Creek are the _kings and queens_ of dogfooding. Spolsky, you sure have nurtured a group of very loyal team players. I applaud you all. It must be really nice to work at a place where the love of the process and the product are both so strong.
In my opinion if there's one thing a reader should take away from this it should be that Single Page Apps and separation of server and client are The. Best. Thing. Ever. From the start, design your system this way.
Good post, and an entertaining read.
While I don't think client side apps are the only way of doing things, I will say that I think client side apps will become much more popular once browsers get a little better.
I love the separation of concerns that is possible with JS apps - you can have one team working on the API and one on the interface and the only place they really need to communicate is in the API documentation. Once it's all done, you've already got a fully functional and secure API (because it wasn't an afterthought) that can be used for other clients.
That is, in my opinion, the larger point. If you can do solid 'interface driven design' then you enable rapid evolution on both sides of the API. One of the things Jon Wall and I did at NetApp was prototype a better split of system across the RAID and Filesystem layers, that split achieved 50% better performance across the same spindles and it allowed for innovation in the file system layer that was currently hindered by "all the hooks into the RAID stuff".> I love the separation of concerns that is possible with ... one team working on the API and one on the interfaceThe key though is picking the right layering, and not having too many. Like hashes in perl you can go crazy and suddenly everything is an API and simple things go through n layers and bog down.
When people tell me they want to be architects I ask them questions about layering, that is where you separate the good systems thinkers from the not so good ones.
> I love the separation of concerns that is possible with JS apps - you can have one team working on the API and one on the interface and the only place they really need to communicate is in the API documentation.
Yeah, me too. Apart from the actual, observable benefits you mentioned, I just find something really satisfying about the separation. Fewer hacks, easier to modify things.
I think the rise in the popularity of doing things this way is largely thanks to the increase in popularity of test-driven development. Creating a client app without an API is so easy given all the tools available for mocking or faking APIs, and creating a standalone API is easy given that most of the time, you're just testing the JSON or XML output of a bunch of functions.
>you can have one team working on the API and one on the interface and the only place they really need to communicate is in the API documentation
That's already how you should be working anyways. Unfortunately, most language don't seem to have support for a decent template system that makes this natural. People need to start making heist clones in their language of choice so doing things right becomes more common.
I agree, but it's often too easy to just add another variable in your controller then modify the view, and now there's undocumented functionality. IMO the full separation is more helpful to stay DRY.
What does the article say to advance "single page apps" as the "best thing ever" over other application models that use a clean interface and separation of concerns between client and service?
>In my opinion if there's one thing a reader should take away from this it should be that Single Page Apps and separation of server and client are The. Best. Thing. Ever
Those are two orthogonal things. All my sites have a complete separation of presentation from code and access a nice API to get data. Including the ones that are purely HTML and have no javascript at all.
Single page apps are good for things that are actually apps. Except that I want to leave the app open, and several other apps, and not have it interfere with my normal browsing. Until browsers realize this, it is actually pretty irritating to use browser apps.
Not completely orthogonal. The way Trello is set up, client and server releases are completely unrelated and don't have to know about each other. The web app is almost as separate from the server as the iOS/Android apps, which isn't really possible if you're rendering html in the server.
That said, I agree that making something into a single page app just to get this separation isn't going to be useful.
> which isn't really possible if you're rendering html in the server.
You can have a front-end web server that is also a client of the API server and keep the separation as if it were any other API client. This is the approach I'm taking in my current project since I have to support some obsolete browsers, but more generally it lends itself to a cleaner decoupled architecture
That type of double-server architecture isn't one I'd considered. Very interesting.
It is, let me know if you want more info
>which isn't really possible if you're rendering html in the server.
It's not only possible, it's pretty trivial in most cases. Just endow each user facing object with a .ToHTML() and .ToJSON().
This is actually one of the core benefits of REST, you send me a request for a resource along with some desired media type(s), and I send you back a representation in the media type of your choice (or as close as possible).
I dunno, if you've got a good API, server-side HTML rendering is just another client, just like client-side rendering would be.
The trap a lot of developers fall into is going around the existing API for server-side applications though, thinking they'll get more performance by (for example) going directly to the persistence layer. That's how most server-side apps are written, actually; api and front-end tightly coupled.
> That said, I agree that making something into a single page app just to get this separation isn't going to be useful.
I agree, but there are other benefits too :)
>which isn't really possible if you're rendering html in the server.
Yes it is, that's my point. Our designers push new templates and our developers push new binaries completely independently. We literally have an API doc that the designers put "I want this" into, and the developers implement it and update the doc to reflect it being finished. People just use template systems that make this unnatural. See heist for an example of a system that makes it natural and straight forward.
Thanks for the name of 'heist' - never heard of it before and it looks interesting.
Yeah, you're right, not the same. I was generalising and lumping them together as the same thing though for the sake of praising the Trello way of doing it and encouraging more people to keep the data and presentation separate. I personally like the single page app approach though for reasons other than the fact that it just creates the separation.
It's almost comical how nobody at my company would ever take a project this seriously. Good for them, Trello is awesome.
Everything I've ever heard about Fog Creek indicates that they take their people seriously. Which probably makes them more likely to take their work at Fog Creek seriously.
Funny how that works, huh? A lesson many, many other companies could profit from.
I agree with your point, but it requires that you hire correct right from the beginning. Spolsky (like pg, Atwood, Fried/DHH , etc.) has his pick of the litter because of well thought out essays and large base of followers. You can't simply take this attitude without having a large pipeline of people who (1) agree with you and (2) who are good.
Lesson learned - creating a community or following of people (i.e. talent marketing) is a very powerful thing.
The essays and followers surely helped, but there are lots of basic pragmatic things you can do to make yourself a desirable employer even if you aren't a great essayist. Just some obvious ones off the top of my head:
- Have sensible working hours (30 to 40 hours a week is optimal).
- Either let people work from home, or give them private offices.
- Don't have idiotic hiring criteria like buzzword matching or college degrees. (Meta: don't have the personnel department doing the hiring.)
- Get at least the basics of tools and process right. You don't have to let people code in Lisp or Haskell, but when a candidate asks you about version control, the answer better hadn't be "oh we don't have time for that here".
- If you are requiring people to work in the office, don't quibble about things like high-spec machines and good chairs that cost a small fraction of the cost of hiring people.
Hit everything on that checklist and even if you're still not quite as sought after as Fog Creek, you'll be well out in front of most of your competitors at little cost in either time or money (and infinitely far in front of Fog Creek for any candidates who won't or can't live in New York).
> and infinitely far in front of Fog Creek for any candidates who won't or can't live in New York
Fog Creek actually hires remote people now. We made the change almost a year ago and its been a great source of new candidates.
Ah, then I happily stand corrected on that point!
Having essays is not the only way to do it. The other way is to just be willing to wait for the right hire. At my company, we have a small engineering team so far, but we have taken a long time for each hire with the basic idea that they need to fit some key characteristics of our team. At the beginning this was actually harder to identify what was important, but now that we have a larger team there is more diversity (ideas, genders, backgrounds) and the core important tenants to us are more defined.
I am curious about what characteristics you have uncovered. And do you think they are accidental?
Well if you read what Joel says about working conditions and look at the office pics where you can see that in practice it just makes sense that fogcreek can attract the best people. Look how everybody has is own office.
I still dont get it why people keep thinking that to cram puzzlers on a noisy heap with all possible (social, visual, audio) distractions saves money in the end. If Joel can offer programmers sane working conditions in NY, so can every office around the world.
> > The Trello API is well-written, has no bugs, and is totally rock solid. Or at > least it doesn't change very often. That means we can put out new > clients all the time without having to update the API. In fact, we can have > multiple versions of the website out at any given time.]
A very counter-intuitive result: most people would not consider a stable API to let you iterate quickly!
I think the article says that the API is stable unless you have access to "Alpha" and "Beta" channels.
Alpha and Beta channels only affect what client you get. All client versions talk to the same API, so it has to be very stable/backwards compatible.
I've used Trello & FogBugz over the years and we've even modeled some of our software after some of the practices they've written about. Amazing stuff!
So they create a new branch for each fix? And all developers are supposed to merge in their branch every few hours or so? Isn't it a bit of a drag?
That's how I work as well. Not a huge problem if you're using git, since branches are cheap and merges are easy.
We use Subversion mostly, I guess that's why the merging is a hassle.
In SVN, "merging" is a bad, scary, no good word. In Git/Mercurial it's an everyday operation that happens so seamlessly 90% of the time that you don't even notice. The other 10% of the time, you have great tools (i.e. 3-way merge tools) that make it easy to reconcile conflicting changes.
I was confused about all your "we avoid merging" comments, because I can't imagine how you could possibly work without merging, but this explains it.
Agreed. Amazing how much the right tool can change habits - for the extremely better. It has barely occurred to me that people might struggle with merges (at least, small feature merges) anymore. Switch to git at a minimum, if possible.
On the Trello Android team we have a similar workflow to the web client and server developers. We merge in to a shared development branch when we have a complete feature or bugfix. With git's --no-ff this allows us to see when a new feature was implemented, and when bugs pop up we have a clear list of intersections where they could have been introduced. Our workflow is roughly based on an excellent post by Vincent Driessen, http://nvie.com/posts/a-successful-git-branching-model/.
I'd love to hear more about the Android team's CI tools. What do you build with? What do you test with?
If you can build some kind of SaaS Android CI system, you'd probably make heaps of cash. Android CI is just enough of a pain to do yourself that people would pay for a one click type solution.
Another Trello Android dev here.
We use gradle for building. The actual app is split into two, trello-app and trello-lib. Anything that can be done w/o Android like SQLite caching, API calls, etc., is done in trello-lib which is a plain Java library. It makes the dev cycle much faster since you can write a unit test and run it in a second.
We don't have CI setup yet. For the server we use something custom but will most likely be moving to Buildbot and with that we'll also setup Android CI.
With Subversion, you have a working copy with changes and then you do "svn update", and Subversion merges the upstream changes into your working copy. (But you don't normally call this merge.)
With Git, you have a working _repository_ with changes and then you do "git pull", and Git merges the upstream changes into your repository.
From the user's perspective, it looks about the same. But the Git merges are safer than the Subversion updates, because if the Subversion update messes up your working copy, you're stuck. But with Git, you always have (1) the commits you made locally, (2) the commits you just pulled from upstream, (3) the merge that was done by "git pull". And (1) and (2) can't get messed up, only (3) can get messed up. But you still have both your version (1) and their version (2) to go back to, so you have lots of chances to fix it.
Think of Git branches and the corresponding merges to be like Subversion updates with backups of the previous local working copy.
> Subversion updates with backups of the previous local working copy.
Which is a practice I got into the habit of doing manually when I used to work with Subversion.
That's how we work on Brackets[1], and I think a lot of projects work that way. It depends what kind of version control system you use. It's pretty easy to do this with the "GitHub flow".
Yes, more or less a new branch for each fix. But with git and Kiln, it's really not a drag at all; the merges are generally really easy and mostly performed by our release manager when the card makes it to 'Ready to Merge'.
Do I understand correctly that your stable 'master' branch is always ready for release? Basically it means that changes (fixes) should be relatively small, otherwise they are released via channels, correct?
How do you test the fixes? Which branch gets deployed to the test environment?
Everyone runs dev environments locally and tests. A dev environment is really easy to set up with mongo and node. Occasionally, we’ll put stuff on a staging environment if we want to test a database or infrastructure change. Big, experimental client changes go to the alpha channel on production.
So when you have 2 features/bug fixes in the same area, built in 2 different branches and deployed to isolated dev/test environments... the first time you might find out that they dont work well with each other is in Staging? Why wait for that when you could deploy to a common environment and catch issues sooner?
Btw... I am trying to get some teams in my company to get out of branching. So trying to understand your view (which is exactly what these teams are doing), hence these questions...
That rarely happens in our experience. With our internal board, we know who is working on what and in what area of the code. Plus, all code is reviewed by someone working in the same area.
The alternative, having everyone work on the master branch, is a bigger drag, because it introduces all sorts of coordination problems like "is the master branch in a state to be deployed? are your changes done?".
You are right, that would be the opposite extreme. What we do is that we create a branch and push our code every few hours. After a preset time, the branch is merged with the level 2 branch for code review/testing/regression etc., and after that to the main branch and then released to live production.
This saves us some merging.
Is merging painful enough for you team that it's worth avoiding?
It can't be avoided, but we do tend to try and minimize it.
My experience has been that maximizing merges works best. If you're constantly merging, hardly any of your merges have any friction. The longer you're separate from the master branch, the more likely you are to get painful merges with conflicts and subtle bugs.
Elsewhere in the thread, you'll see that 'sidcool is on SVN, where merging is not a fun experience — which explains his team's aversion to merges.
I recently switched to this approach and it works very well. No more "I want to push out this quick fix, but I have to wrap up this other thing first so I don't break the build".
With github it is especially nice. I push to a remote branch, which I can see when I go to the project page. When I click merge it shows whether or not the TravisCI build passed. If it looks good, then I click to auto-merge and it's all set. It is possible to even automate away that part as well and have the whole thing merge and deploy on push if the build passes, but I am not quite ready to take the plunge yet with that (too easy to botch a production release IMHO).
Yep. My team follows a similar workflow with Github - every feature, bugfix, etc is developed on a branch off of master, then merged back into master via pull request when it is ready for deploying.
It's a nice workflow, changes are very visible to the entire team and well summarized (by the commit history and any comments/discussion on the pull request itself). Making a new branch is a one-line operation (two if you count hooking it up to the remote), so no, I've never personally felt that was a drag. Sometimes it feels a bit silly to create a branch for a one or two line fix, but the visibility to the team is worth it.
Developers in my team have quite a distaste for merging. We try to keep it at minimum. The trade off is less visibility. I agree with you.
I agree. I would pick 1 dirty branch with continuous builds and tests any time over several branches. I have worked with several branches too and it is always messy, lots of confusion and too many overheads (including the Release Manager guy).
I love trello... but I don't like the branching model... :)
That's how my team at work does things as well. It actually speeds things along much more quickly than doing a bunch of fixes in one branch, because if one of your fixes is held up by QA, all the others can still be merged independently.
I love to hear about workflows, what's your ideal one?
Commit to 1 dirty branch; assume it is broken and let tests prove otherwise. When tests pass, you know it is a working branch, tag and release. I skipped a few steps for simplicity. Here, merges are an exception, not the rule.
This workflow won't work without automated tests... is lack of automated testing the reason to have individual branches for features and bug fixes?
What we do is that we create a branch and push our code every few hours. After a preset time, the branch is merged with the level 2 branch for code review/testing/regression etc., and after that to the main branch and then released to live production. This saves us some merging.
What I'm interested in is the mechanics behind how they know were to send a user based on their channel (beta/stable/alpha). We wanted to do something like this, but we couldn't figure out how to route users to the right app server either using AWS ELBs or nginx proxying ... admittedly we didn't really spend a lot of time thinking about it though.
We do that decision-making inside the web servers. It only affects the client you get, so when someone requests a page we do a lookup on the logged-in member to decide which channel they get. API reqs don't care what channel you're on. No need for any fancy nginx proxying/etc.
I'd love to hear—maybe I missed another blog post—why they went with the single release manager, where only one person can merge and deploy. What happens if Doug is sick or on vacation? Or even just in a meeting? What is a typical amount of time for a change to sit in "ready for merge" or merged but not deployed?
When it's nobody's job, nobody does it. Any developer COULD do it, though.
The primary benefit is that it creates a single point of communication for the developers, the designers, and the QA. It also helps with prioritizing changes that could be conflicting and, in the same vein, helps prevent bad releases or merges. And, of course, anyone can do this job. If I'm not available, someone picks up the slack. I just volunteered because I was interested and available.
All that said, I think that letting every developer deploy would not be a bad idea at all. The problem is that our team is too big to do that without creating more robust deployment tools and too small to dedicate enough time to doing so. My hope is that one day we can get there, though.
You should definitely work on the Kinect-to-deploy integration, then everyone would want to learn how to do it ;)
This task used to be more distributed over the server-side devs, but having a single release manager who does this most of the time and other people who do it occasionally and/or when the release manager isn't working seems to work well.
I don't really see why a developer couldn't also merge changes into "the official Build repo". Or is the "release manager" just a term for the person deciding what gets released when?
I like the lighthearted approach to cross-selling in the article. ☺
Are they on AWS or hosting your own hardware? What OS?
AWS, which came about when Hurricane Sandy took down the data center's backup generator fuel supply and much of the team spent days bucket brigading diesel fuel up 17 flights of stairs.
How difficult is it to make the backend work with both new features and old features at the same time?
To add to what bobby said, changes to the API are far more likely to be additions rather than actual behavior changes (except for bug fixes, which of course all clients handle fine). This makes it easy; old clients just ignore new routes/arguments.
Thank you for this. I hope more people come to understand this more nuanced understanding of change. Please talk about it more!
Another Trello dev here.
Basically, we have many clients that are consuming the API, including our mobile apps. We never change the published interface so that if you're getting some board fields that will always work. This means that we don't change the types of fields (for example string vs object) or the names of fields since that changes the published service contract.
What we do instead is add more fields. A very good example of this is emoji. We added emoji support a while back and instead of changing the "text" fields to be objects or embedding HTML (gah) into them we added another field called "textData" that has the extra info.
This is copy/pasted from an actual call for getting card actions:
This may not be sustainable in the long term and if it's not we'll version the API if we have to at that point. So far though, the API has proven itself to be very well designed and adaptable (thanks to @d_lec)"textData": { "emoji": { "daft": "https://trello-emoji.s3.amazonaws.com/4f820a995a03e8e82d134ac4/ae846ca70bf7aa95dd3330b8fd1c70cb/art-daft-punk-steampunk-951631.gif" } }, "text": "Custom emoji! :daft:"+100
If there are necessary API changes, we just need to push updates to the API ahead of time. The API needs to be stable and backwards compatible anyway for the mobile apps (and all third party apps).
Does anyone know how the server determines which channel the client should be using? Are they doing this check at the apache/nginx level, or on the server right after the user is authenticated, and before the client code is sent?
Because we are only delivering multiple clients, it means we only have one server version running at a time. Then, after authentication, we decide which client version you will be receiving based on your chosen channel and hashed member id in the case of multiple distributions within the channel.
Good question, I wondered about this as well. Channel switching would certainly create issues on data model changes in the underlying store.
From what I understood, the API doesn't have channels, so the store only stores a single data model. The channels are only for the clients.
Right. The there is only one version of the API. Multiple clients.
This is great and all, but I really wish I knew why they won't make the audit logs accessible to users. If someone has a hack for this, I'm all ears.
Need it for compliance documentation.
> The Trello API is well-written, has no bugs, and is totally rock solid.
The no bugs claim indicates to me this wasn't written by a technical person. Or at least, not very technical.
It was a joke. =)
I reread that paragraph. In context, it doesn't seem so.
Read more than just the paragraph. The entire article is written in a "fun", joking style.
> The revelation could result in a rippling shockwave that knocks you off your seat and may have troubling, unpredictable consequences for the time-space continuum. Possibly.
> You’ve probably downloaded our iOS, Android, Kindle, and/or Windows 8 apps, and are saying to yourself, “These are very well polished apps which I have rated or will rate favorably in their respective app stores
> The Trello API is well-written, has no bugs, and is totally rock solid. Or at least it doesn’t change very often. [Goes on to only reference the fact that it doesn't change very often]
I like the passion for the product. Shines through very well.
You figured it out. They secretly don't have any bugs. Shhh.
These tech intros are pretty cool, yet can I ask how does the team acquire users? That seems to be harder than the tech challenges for Trello IMHO.
I don't get it. Why is having an API in the backend so that many clients can use it a big deal?
In trello's case they have a browser app, ios, and android. Having one well designed api makes these easier to build and maintain. It appears to be working pretty well for them. I use trello in browser, on an iphone, and on an ipad, and they all work together very seamlessly.
Anybody using FogBugz and Kiln from Fog Creek? Both seem to be a bit outdated...
In what sense are they outdated? UI-wise (which I will agree with) or feature wise?
I'd like to be able to write like this. It made me LOL.
Trello using Trello,(Inception) awesome!
Trello is using Trello to develop Trello. So meta :)
they're using Trello to write a Trello post on how they're using Trello to work on Trello.
So interesting!