Deploying branches to GitHub.com
githubengineering.comI really enjoy reading how organizations have implemented continuous deployment.
One question I have which is not addressed by the article is how to deal with database changes. Every database has difficulty with schema migrations to one degree or another, but MySQL (which IIRC is what GitHub uses) is particularly bad. In my organization, we are VERY careful with any deploy that contains a migration.
(I suppose this is where GitHub's staging environments with real world datasets come in.)
One workaround I've considered is automatically deploying code that doesn't contain a migration (which is the vast majority) and forcing a more manual approach to database migrations, to make sure people are on hand to roll it back if necessary.
At GitHub we take a fairly manual approach to migrations, basically the migration is done in its own PR ahead of the PR that has the code.
That's awesome! Seems like a really solid setup. Bufferapp took a page from your book and have the same deploy setup with hubot... it works out really well for them, too.
I have three questions to ask about DB migrations (which I can guess the answers to but would love to hear directly), if that's okay:
How do you handle a DB migration with a staged rollout (two 2 of N production servers)?
How do you organise timing between a migration deploy and code deploy if one is done before the other?
How do you handle rollbacks?
Migrations stay pretty small. Most of them just add a column or index. We also have to make sure tables aren't too hot to migrate. See a previous post for more on that: http://githubengineering.com/using-mysql-performance-schema-...
It's also really important that migrations don't affect the running app code. New columns shouldn't be used yet. Removed columns or tables need to have all references removed first before running the migration. We confirm this with a deprecation helper that sends metrics to Graphite.
That's about all I can answer from the app side :)
Sweet, that makes sense. Thanks for answering!
Was assuming that a column rename would duplicate the column during the first migration — so that the old and new codebase would work correctly. I guess the only complication is that you need to keep track of which branches have been successfully rebased/merged to master so you can run the second cleanup migrations.
Thanks again!
Runnable.io makes it very easy to run all your Github branches, using containers. Here's is how it works https://www.youtube.com/watch?v=_uwPKIH990E&feature=youtu.be
Do you know how well it works with an SOA setup with multiple repos?
We run 10 repos in our sandbox
The 'merged once deployed to production' thing, yes, I know even if advocated by GitHub, seems extremely weird to me. It does seem they have a staging check first, which is good.
It seems you'd want to merge it first, so that you know it when merged with "all the things" on master, so it more closely mirrors what you are going to get once it's merged in.
So they could just merge first, and then if staging passes in their CI system, automatically deploy to prod, which is the way many orgs do it.
My point is though, you'd want to deal with the merge fun (if any) first, else you are deciding to test branches (pull requests) that only have ALL of the commits from master (rebased, etc), so it's easier to just make sure they hop on master first, else you might "remove" something from prod for a while until it's merged in. Not good.
They may have some things to deal with that, but in this case, it doesn't seem like something I'd recommend for most people, and feels weird and organically evolved. One branch may not have the commits another has and both could be deployed without merging, leaving the github deployed code state fluctuating back and forth as one commit drops out and another drops in, before finally both are in at the same time.
I wonder how this is handled?
I'm guessing that the master branch is first merged down to their develop branch or directly to their release branch. That way their release branch does have everything in master already.
Indeed that is the case:
> Since master is our stable release branch, we want to ensure that any branches being deployed are caught up with the latest code in master. Before proceeding with a deployment, Hubot will detect if our branch is behind and automatically merge in master if required.
We've adopted some aspects of this flow, and our take is that we test feature branches on stage thoroughly including CI test run and code review.
Then we merge to master and let all the CI run again while we manually verify. Any troubles and you revert. All green? Deploy right away. We try never to deploy more than 2-3 changes to production at a time.
The main bottleneck for us is the speed of our CI runs. It's tempting to merge in a lot of changes to master and let them accumulate on QA. Reducing the test run time is an ongoing goal and should make this system pretty scalable for our team.
Usually, you won't have merge conflicts if you deploy early and often and keep feature branches deliberately small. For larger stories, consider breaking it into discrete feature branches that implement part of the functionality (ex: behind a feature gate).
The deployment system automatically merges master into the branch being tested on deployment, and will not deploy any branch that does not contain master. In fact, there is a check that nags the main app team if master is not deployed to production for some reason (usually someone merging docs changes while someone is testing a "real" branch). It's considered an abnormal state, and I will often block all deployments until we figure out why master hasn't gone out.
Even forced deployments (which ignore CI and a few other checks for emergencies or maintenance mode) won't deploy a branch that's 24 hours behind master.
If one of your developer's Campfire (assuming y'all still use this) accounts gets popped, does the attacker now have the ability to deploy to production, or is there some other mitigating factor not mentioned here?
I'd be surprised if they don't have additional security. Here's some info on what Box did to add authentication and authorization to Hubot commands.
Thanks! This is exactly what I was looking for.
VPN would be my guess.
If I read this correctly, only 1 dev can test on production at a time. How does this scale as the company grows?
The "lab" environments have enough capacity to allow plenty of parallel testing. When deploying to production, it's expected that you already know your branch does what it's supposed to do. A production deploy is when one makes sure a branch has not introduced regressions, so holding it for more than 15 minutes is rare.
I imagine deploying a green environment[0] to test before merging to master would be one solution. It is interesting that Github seem to not do that.
I suppose you could have a hierarchy, with team leads pulling from their devs and deploying the whole team's stuff when it's ready.
That would defeat one of their major wins, ownership/agency/responsibility.
"spider-skull-island" can I get access?
That's just a special staging server for Venture Brothers nerds (no one else wants to type all that out).
We're following github flow and are currently doing these steps manually on deployment (i.e. merge branch with master, check out branch). Then roll back to master if something fails, otherwise merge master with the branch. It would be great with some tooling to help this. Has someone done open source work on this already?
I've been trying so hard to move my organization towards a chatops approach. We are making progress, it's just slower than I want.
Deploy Locks looks interesting. does any one has more info on how this can be achieved in a git pull based deploy flow ?