A bot has been finding bugs and submitting patches for them, successfully masquerading as a human

4 min read Original article ↗

Repairnator is a bot that identifies bugs in open source software integration and creates patches without human intervention, submitting them to the open source project's maintainers under an assumed human identity; it has succeeded in having five of its patches accepted so far.

Repairnator's creator, Martin Monperrus, has found that human software maintainers have a bias against accepting patches generated by bots, but will willingly accept the same code if its author is identified as another human.


To demonstrate that program repair is human-competitive, a program repair bot has to find a high-quality patch before a human does so. In this context, a patch can be considered to be human-competitive if it satisfies the two conditions of timeliness and quality. Timeliness refers to the fact that the system must find a patch before the human developer. In other words, the prototype system must produce patches in the order of magnitude of minutes, not days. Also, the patch generated by the bot must be correct-enough, of similar quality — correct and readable — compared to a patch written by a human. Note that there are patches that look correct from the bot's point of view, yet that are incorrect (this is known as overfitting patches in the literature [6, 3]). Those patches are arguably not human-competitive, because humans would never accept them in their code base.

Consequently, for a patch to be human-competitive 1) the bot has to synthesize the patch faster than the human developer 2) the patch has to be judged good-enough by the human developer and permanently merged in the code base.

There is one more aspect to consider. It has been shown that human engineers do not accept contributions from bots as easily as contributions from other humans, even if they are strictly identical [5]. The reason is that humans tend to have a priori biases against machines, and are more tolerant to errors if the contribution comes from a human peer. In the context of program repair, this means that developers may put the bar higher on the quality of the patch, if they know that the patch comes from a bot. This would impede our quest for a human-competitiveness proof in the context of program repair.

Human-competitive Patches in Automatic Program Repair with Repairnator [Martin Monperrus/Medium]

(via Four Short Links)