Launch HN: MergeQueue (YC S21) – Automate rebasing and merging for your codebase
Hi HN, we’re Spriha and Ankit, founders of MergeQueue (https://mergequeue.com/). We enable automatic merges for your Github Pull Requests, based on configurable rules you can set for each repo. These automated merges ensure you never have failing builds on master/main and save time on rebasing your feature branches.
If you have a monorepo where a big engineering team is regularly merging changes, the stability of the main branch degrades considerably. This happens because more engineers working on the same codebase introduce incompatible changes, causing builds to break even though their commits pass independently. Here’s an example: https://blog.mergequeue.com/managing-github-merges-for-high-....
Github has a setting to restrict branches to be up to date before merging, but turning on that setting ends up forcing a rebase-athon within the team. This results in wasted time, and all too often, a stressful scramble to figure out what changes broke the build.
We had this problem in our previous company where we looked for a solution to automate the process. We found this paper [1] published by Uber to manage monorepos at scale and we built a lightweight version of that internally. It immediately eliminated the overhead of keeping builds healthy. After that, we decided to build a public version to save others from re-inventing the wheel.
We spoke to engineers at Airbnb, Stripe, Uber, Shopify, Quora and other large companies who have internally built similar tools, but teams who need such tools the most often don’t have the bandwidth to dedicate developers to building and maintaining them.
MergeQueue (MQ) is a FIFO queue of Pull Requests (PRs) that works as a Github app. To use MQ as a developer, instead of merging manually, you just add a Github Label “mergequeue” to the PR. MQ then takes care of the rest: it sequentially updates or rebases the branch, monitors the configured CI checks and eventually merges the changes. If the checks fail, it will dequeue the PR, add comments describing the reason and move to the next one. For high output teams, MQ also offers batch mode to run CI in parallel batches. If you’d like to learn more, there’s a lot more here: https://mergequeue.com/documentation.
Currently, we are also piloting a way to manage “flaky” (i.e. unreliable) tests through MQ. This integrates with your CI provider (we currently support CircleCI), analyses test results and flags the tests that fail inconsistently. When flaky tests are identified, MQ reruns the test depending on the configuration set.
We charge by usage in an organization, so for instance if your organization has 100 developers but only 20 of them use MQ, only those 20 will be billed. You can sign up for a free 14 day trial without a credit card. We also support single tenant or on-prem deployments, have webhooks to connect to your other apps, offer multi-queue support, and are SOC2 certified!
We’d love for you to try MergeQueue and give us any feedback you have. If you've used something similar in the past, we're also curious to learn what problems you faced so we're better prepared for them ourselves :)
[1] https://eng.uber.com/research/keeping-master-green-at-scale/ - discussed at the time: Keeping master green at scale - https://news.ycombinator.com/item?id=19692820 - April 2019 (115 comments) Really dumb question while I'm trying to decide whether to use something like git vs. CRDTs to handle version control for user changes made in an app I'm working on: why do we even use git anymore for source code version control if we want behavior like this? Nobody likes merge conflicts. We all want versioning. So long as we have versions at all why isn't the ideal interface for developing in teams something more like editing in Google docs? Why aren't we just doing that? Why are we still using systems that produce merge conflicts? Hoping for insight from folks who have either done this themselves or looked into it. Edit: one particularly nice feature of Google docs over Confluence is that in Google docs I can suggest changes that is somewhat akin to branches with git. I don't need to force my change through without review. This isn't a natural part of CRDTs, but it sounds like the nicest source control system might be CRDTs plus branches? Feels like this should be a post in itself! This is just my opinion so open to any push backs / comments. IMO the main reasons for source control are:
- having revisions / diffs that can be reviewed by someone else
- while working independently in a distributed fashion
- easy way to rollback to an older revision I'm not too familiar with CRDTs, will add more comments after reading more about it, but let's take Google docs as example here. What if we were all writing code in Google docs, how would that turn out. Google docs do a decent job of versioning, so the last point can still be satisfied. I think with a source code styled version of Google docs, one can easily tag versions and do rollbacks. The issue happens when collaborating. Just from my experience working in spreadsheets where > 10 users are actively trying to modify, overwriting each other becomes very common. You can argue that it might make it easy to also fix this, but what about running the code locally? One would have to still take a snapshot - hope that it actually compiles. From my personal experience, while I'm writing code, it rarely compiles in the first attempt. So then we would really have to work on snapshots where we know it compiles. At that point, you are really creating "commits" and pushing it to the remote. I am not sure if we can really completely avoid conflicts in such model. In real time collaboration, like Google docs, merge conflicts can be avoided because you see them play out live, and can react to them. In an offline collaboration system merge conflicts are a feature, they bring attention to inconsistencies you might otherwise overlook. This has very little to do with how the data is stored. It’s a good question. Well, CRDTs are just an underlying data type that guarantee conflict free eventual consistency. That doesn’t mean the result would be an English paragraph or buildable code, however. Then what git does really well that bare CRDTs don’t are things like branching and local experimentation. Being intentional with commit points can also keep file sizes down, for a more real-time tool like Docs potentially every character change would be stored. CRDTs can have performance problems with things like file size growth. For, your use case of user changes, CRDTs seem like a good option. It would be interesting to explore building a version control system on top of CRDTs, if done right it could have the benefits of git with less merge conflicts. Fork => Modify => Merge. Suppose you have a 100 page gdoc. You start editing one paragraph per page, and voila, everything is fine. Casual edits are consistent "in context". However, if you start editing on page 1, then skip to page 100 and go back to page 50, then it's possible that while "there was no conflict" (people weren't editing the same section of the doc at the same time), however, the document was in a "semantically inconsistent state" while you were making your edits (ie: someone trying to "compile" your doc in between the beginning and conclusion of your changes would have had a logic error until you'd made that final edit). Git (and branching, as you're suggesting) allows you to select the next "actual" future from many "plausible" futures. Think of A.C.I.D. You want your changes to be atomic (gdoc edits are not atomic across multiple "pages"), you want your changes to be durable (gdocs may/may not hit your definition for durability if changes can be made to the lines arbitrarily), consistency and isolation are also compromised with the free-flowing editing style you're describing. Imagine dumping a python script into gdocs, pulling down the contents, and running it through a local python interpreter. You could get pretty far with low amounts of edits and using the "comment" feature liberally, but once you try and "propose a refactor" you immediately need to "fork the universe" and have two (or more) semantically distinct representations of the document, or some way of queuing "multi-page edits" such that each could come in ... some sort of "merge queue" .... ;-) For user changes, a solution based on OT/CRDT may be appropriate. For code, you need merge conflicts. Let me put it like this: Do concurrent edits in google docs always produce a valid and correct paragraph, with no grammatical errors, and that communicates the proper thoughts? If we need merge conflicts that must be manually resolved, how can this product being discussed exist? If merge conflicts can be automatically resolved, why aren't we using a system that does operates like that under the hood rather than needing a product like this bolted on? The product being discussed here doesn't solve the issue of merge conflicts itself, but rather of eventual consistency. Merge conflicts is only a part of it, and there are some heuristics that can be used to reduce those, but eventually require manual review. Perhaps with Github Copilot this can improve? An example of the problem we are describing is explained here: https://blog.mergequeue.com/managing-github-merges-for-high-... There are ways to resolve merge conflicts automatically in some cases given some assumptions, but there is no way to resolve any given merge conflict without a system that can create a correct program. I won't bore you with the formal proof for this (especially since I have no idea how to construct such a proof). Consider that some merge conflicts require new code to be written to resolve them. For example, here's an original program: --- Also for example, an unsophisticated and unsound method for automatically resolving some merge conflicts: 1. Assume that if the build and tests succeed, the program is correct. (This is the most important bit) 2. Accept all non-conflicting changes. 3. For conflicting changes, accept one side of the conflict. 4. Build and test the result. If it succeeds, merge it. 5. Otherwise, attempt the same with the other side of the conflict. If it succeeds, merge it. 6. Otherwise, it cannot be merged automatically by this method. Note that this approach could also apply to a CRDT-based merge. I'm on a medium sized team so this might not be possible for everyone. The way I resolve this in my Trunk-Based source control system(so we heavily use Feature Flags) is to just never merge. Every person who is to check-in code is to first stash their changes, pull all code, then pop the stash and deal with any conflicts locally, do the relevant testing, then check-in. The entire repo is checkin 1 -> checkin 2 -> checkin 3 -> etc.. When viewed in this light it is no different than if you had pulled the baz() code before you started editing anything. Would you have changed it to bar() or not? On the contrary, I for one am amazed how often no intervention is necessary on automatic merges, given how finnicky compilers are in general! If the teams are not ridiculously large and people mostly work on their own stuff and coordinate work on the "common" part, isn't it mostly good? There are certainly usability issues with git but I wouldn't literally use Google docs with colored squiggly lines for the changes from Anne and Bob... I don't mean literally Google docs but a storage system more like that than git. :) > why isn't the ideal interface for developing in teams something more like editing in Google docs Engineering and debugging are hard enough as it is. Doing this while one or more people are changing the code in real time would be a nightmare. Because you'd never have a working revision to release, since everyone on the team will not finish their features at the same time. How does this compare to Bors NG? bors solves a similar problem, but we've found that enterprises require a lot of additional features to get their dev teams to actually use it, so we've focused on those to make this production-ready. Specifically: 1. We focus on being "no-maintenance", since our primary value prop to customers is "you won't have to spend eng resources on merge problems any more". This means: a one-click "set and forget" install, no additional config files to maintain or DSL to learn, unlike bors. 2. We have all the security and access-control features you'd want (we're Soc2 compliant, offer on-prem etc). This is important since the product interacts with our customers' source code. 3. We offer various customizations on merge workflows to help our customers optimize the exact parameters they care about (eg: developer wait time vs CI costs vs time to identity bad PRs in a batch, optimistic merge heuristics etc) 4. We offer a whole host of useful analytics to show you merge bottlenecks. We initially just put these as a nice-to-have, but we've now got customers who use us specifically because of these analytics! We're also constantly adding new features based on customer requests (for instance, the flaky test features we're piloting originally came from customer requests we kept hearing over and over!). Speaking of which, if there's something you'd like to see that you don't currently get from your existing system/bors, do let me know :) We're a >200 people company that's been using bors-ng for more than 4 years now for a monorepo. It's been a joy to use Bors! I'd thought I'd just chime in a little here: > 1. We focus on being "no-maintenance", since our primary value prop to customers is "you won't have to spend eng resources on merge problems any more". This means: a one-click "set and forget" install, no additional config files to maintain or DSL to learn, unlike bors. Bors was a breeze to set up and has been super stable and easy to upgrade. We used their Docker container and you need a PostgreSQL database. > 2. We have all the security and access-control features you'd want (we're Soc2 compliant, offer on-prem etc). This is important since the product interacts with our customers' source code. For a monorepo you can get really far using Github's CODEOWNERS file and requiring an approval in bors.toml. > 3. We offer various customizations on merge workflows to help our customers optimize the exact parameters they care about (eg: developer wait time vs CI costs vs time to identity bad PRs in a batch, optimistic merge heuristics etc) Nice! One issue we've seen is that a large backlog of PRs that needs to be merged, combined with a failing PR, is that bifurcations can easily build up a large backlog. We've looked at various different strategies to cope with that[1]. [1] https://forum.bors.tech/t/draft-rfc-support-building-batches... Above said, I really think the _real_ solution to handling large backlogs really lies in speeding up the CI build, not adding even more complexity to handle a backlog. This includes adding caching à-la Earthly[2] et al., but also moving away from heavy integration/system tests etc. This was my question too. I considered making a company out of Bors but didn’t think it would work because Bors mostly just works. Haha yeah we're that company- turns out there's actually a lot to do, especially when handling this at scale! come join us :) Perfect YC business. When an engineer made a similar solution at my previous company we celebrated him for months. This seems small but you have huge potential here, good luck! :D A previous employer that underwent a massive multi-billon dollar migration to SOA (and then folks went too far and made everything a micro-service so had to rollup some work) basically because merging became too slow into our monorail as we got to ~O(1k) engineers. We had an in-house solution like this that we used for 2-3 years (and a worse version for years before that). What a great idea to productive it! IMO the SOA migration simply due to merging was such a heavy-handed solution. If we could have made our MergeQueue way better, perhaps we could have avoided that multi-billion dollar (+ the rollback work) fiasco. Congrats! Wow, that's a great story, maybe we should quote that :)
If you are open to sharing more of that experience, I would love to hear that. You can also directly email me at ankit[at]mergequeue[dot]com if you don't want to post publicly. Started making something similar then joined a new company (Brex) and we use https://github.com/bors-ng/bors-ng which is open source and works great. It merges huge batches of PRs and can automatically bisect them if they fail CI checks to find divide and conquer and merge in fewest batches possible. It’s written in Elixir. How does this compare to Mergify? https://mergify.io/ Neat! Pretty much every large company seems to have some variation of this. I skimmed through your documentation, but it wasn't clear if this would handle workflows like the Chromium/AOSP workflows, where you have multiple repositories that are managed together via Google's repo [0] tool to give the illusion that it is a monorepo? Occasionally you may have changes that span repositories and either need to be merged in a specific order, or in a single shot all-or-nothing merge "batch" [1]. This dependency information would be tracked either when the changes are originally submitted together to CI or directly in the commit message. Of course, this means that you potentially have a small window where repo syncs may pull a partial set of changes, but usually this is solved via something similar to what Chromium calls "Sync to Green" [2]. [0] https://gerrit.googlesource.com/git-repo/ [1]
https://chromium.googlesource.com/chromiumos/docs/+/HEAD/con... [2] https://chromium.googlesource.com/chromiumos/docs/+/HEAD/dev... How does this compare to Shipit? similar to how we compare with bors, actually! https://news.ycombinator.com/item?id=27858794 tl;dr: Our goal is to be the absolute lowest-maintenance option for your team :) This is the one true way to do things, of course (https://graydon2.dreamwidth.org/1597.html), but besides the "why not use bors/homu" question - what prevents GitHub or GitLab from building this in? I hope we get to a world one day where a standard feature of all pull-request-ish platforms is that merges end up on a task queue, CI runs on the merge, and the merge only happens if CI passes. It doesn't seem like a terribly difficult feature to build in and the potential UX seems much nicer - e.g., just like there's "squash and merge" and "rebase and merge" buttons and the repo admin can require them over normal merges, there could be an "enqueue for merge" / "enqueue for rebase" / etc. button. (Maybe your goal is to get us to that world via acquisition, which would be great!) That's the world I'm hoping for, as well. I'm the lead contributor to bors-ng. Like Steve Jobs criticizing Dropbox, I think that bors is a feature, not a product. If GitHub integrated merge trains, I would probably just call it quits and put bors-ng on feature freeze, maintaining it only enough to make sure it stays working and patched for vulnerabilities while everybody migrates off of it. There are a few reasons for this, but mostly it's because of all the unfixable bugs and limitations that are caused because bors-ng isn't a GitHub feature. * I can't squash PRs while still having them show up as purple in the GitHub UI. GitHub just doesn't let you do that. * The CI configuration needs to be partially duplicated between the CI system itself and bors.toml. To fix this, bors would need to implement its own GitHub Actions workflow file parser. If implementing the CODEOWNERS file is any indication, that basically means reverse-engineering GitHub's behavior, because you can bet your life their documentation won't cover every possible corner case. * Some pull request review features need to be duplicated between GitHub itself and bors-ng, like pre-checks and CODEOWNERS, because there's no stable API to get that information. * Bors-ng isn't allowed to modify GitHub Workflow files [1]. They just don't let GitHub Apps do that. * Bors-ng can't always tell whether a CI error should cause the PR to be rejected, or if it's just going to be retried later [2]. To fix this, it basically need to know incident information for the CI provider, which basically means it would have to have a hotline to their ops teams, which I don't. [1]: This is because bors-ng is an app, rather than a machine user. It would be fixable, but fixing it would make it harder to deploy. [2]: https://twitter.com/AstraLuma/status/1300797343135137796 These are the biggest problems I have with bors-ng right now. Some of these are theoretically solveable, if you're willing to do way more work than I am, but some of them are truly impossible for anyone but GitHub to solve, and all of them are easier for GitHub to solve than they are for anyone else because they have infrastructure powers that we don't have. It's honestly a little demoralizing. That's a great question. FWIW Gitlab does have merge train with similar baseline functionality. There's two part answer to your question:
- even though it's trivial from the face of it, since we specialize in merging process, we are able to provide much more complex workflows, such as batch mode.
- starting with MQ, our goal is continue focusing in engineering productivity and build as we understand the use cases better. We imagine a world where every team has the toolset that is built internally by developer productivity teams at the likes of Google, FB. We have to still start somewhere right :) Excited to see more stuff in this space. I've previously done some investigation into the various options [0] Questions: - Why have your own pre-conditions rather than relying on Github's built-in conditions and Status? Ideas - Autosquash fixup commits. I like to have multi-commit PRs that get merged into main. I'd like to leave my fixup commits up to make it easier for reviewers to see fixes and then have the tool doing the merge fix them up for me rather than doing yet another CI run for the fixups - What about AzDO's semi-linear rebase option? This rebases a branch and then creates a merge-commit, keeping history linear Wow this is amazing. May be you should add MergeQueue to your list :) - Yeh we define our own pre-conditions because some developers also use it for non-protected branches. The default picks up the Github build-in preconditions and allows you overwrite.
- Can you clarify your fixup commit suggestion? So there's typically an option to squash and merge when merging the PR, but sounds like you want to maintain some of those commits and squash the rest?
- Just read up the AzDO's semi-linear rebase.Yeh MQ should be able to do that as well. You can specify different config for rebasing the branch and for merging the PR. Curious since you have done this much research - have you built or used any of the merge queues in your company? Would love to chat more. ankit[at]mergequeue[dot]com > - Can you clarify your fixup commit suggestion? So there's typically an option to squash and merge when merging the PR, but sounds like you want to maintain some of those commits and squash the rest? If you run `git commit --fixup <ref>`, git will mark the commit as a fixup for `<ref>` (via a commit message convention). Then when you run `git rebase -i --autosquash`, git will automatically move those commits to be just after what they fix and mark them as "f" (fixup) rather than "p" (pick). When putting up PRs for review, there is a tension between making it easy for reviewers to see what has incrementally changed (depending on which PR tool you are using) and having your PR in a ready-to-land state. By having fixups resolved during the merge, you get the best of both worlds. > Curious since you have done this much research - have you built or used any of the merge queues in your company? At my last company, I was researching what we should replace Phab with. I was looking at a broad spectrum of features for our system, including auto-merge, merge-queues, configuration-as-code, "next step" notifications for PRs (rather than flooding all), auto-elect-from-group with load-balancing, etc. Basically, we were in a bit of an extreme for baby sitting the system and I wanted to get us to the complete opposite. Unfortunately, I left before I got to complete that work due to a unique opportunity. Can say from my professional experience, that this is definitely a problem worth solving. <3 curious where have you experience this the most? We are also trying to figure out at what point does the problem become acute. Here’s one. I currently have about five or six small PRs open for bug fixes and minor features. However, they’ve languished a bit in review and haven’t been merged yet. In the meantime, I’ve started a fairly significant refactor. It’s not too complicated but involves moving files and changing some core structures. I am dreading having to rebase those open PRs! Interesting, you can potentially use MQ for automatic rebasing as well, but if there is a merge conflict, MQ will not be able to resolve it for you. If a rebase can be automatically resolved, then MQ will update the branch for you. Nice. Good luck. I am really intrigued by the flaky test detection feature as I have worked with a team member recently to built this from scratch.
I’d love to learn more about what testing frameworks are you targeting. Or is it framework agnostic? Which would be very cool though Congratulations on the launch. Great idea and definitely agree that this is a problem at a lot of companies. Although I am curious about if there is enough market for this product. Would love to hear back any market research you guys have done before. Thanks! :) Are you worried about market size of developer productivity tools in general or just about MQ? We intend to expand into other dev productivity capabilities adjacent to MQ. Our exploration into Flaky test management is also a step in that direction. Congrats - I totally see the value of this product and I like how you have taken the enterprise approach. Asking to learn - How hard was getting SOC2 Compliance? You have an on-prem I see. Any indicative costs for that, please? Nice. Good luck. Really intrigued by the flaky test detection feature as worked with a team member recently to built this from scratch.
Is it framework agnostic? Which would be very cool though How does it compare to GitLab's merge train feature? Yes it's somewhat similar to GitLab merge train. Of course we work with Github. And similar to other comments, there are a few more features including batch mode and flaky test management that make MQ a more interesting offering. We do not support Gitlab today. MergeQueue is phenomenal. We've been using it for a few months now and it's been great for keeping PRs green and helping merge quickly. Thanks @timtamboy63, now we know you you are :) I'm not your competitor, but I am trying to make software metrics a good thing for everybody (developers and leaders) and I'm not sure you can make a claim like the following on your landing page: Fair point! We definitely don't want our analytics to be used to enforce/incentivize arbitrary targets (we've seen those behaviors backfire so many times, unfortunately...). I'll update the language from "Keep a tab on your builds and merges. Know your star developers" to "Keep tabs on your builds, merges and developer activity". So excited to skip rebase/merge hell! Congrats on the launch! Thanks for the support!! Why is the pricing per collaborator? Thanks for asking! We priced it to ensure that it's affordable for small teams and is only charging based on who uses it for larger teams. Would you have preferred a different plan. tbh we've found it's just easier for many of our customers that way. However, we do offer custom pricing plans (annual contracts etc), so do let us know if you have a different preference! This is great. Congrats! Will give it a try! <3 Let us know what you think! this is awesome! congrats on the launch!!
and here's a change we would like to merge: if(foo()) {
send_money_to_oftenwrong()
}
and here's a concurrent change we would like to merge: if(bar()) {
send_money_to_oftenwrong()
}
How do you produce the correct program from these conflicting changes? if(baz()) {
send_money_to_oftenwrong()
}
with the limited analytics that you can gather from a pull request. "Know your star developers."