Disguised as a programmer bot corrects errors in the code

Automated programmer Repairnator made patches good enough to mislead people

“In this world, nothing can be stated definitely, except the inevitability of death and taxes,” wrote Benjamin Franklin in 1789. If he lived today, he could have added errors to the list in this list.

Modern computer programs are so complex that errors inevitably appear in them during the development process. Therefore, finding them and writing patches to fix them is a common part of any software development schedule. There are even companies offering such a service to developers - for example, Travis.

But the search for errors and the writing of patches is a long matter, taking away considerable resources. Various researchers have developed bots to automate this process, but they usually work slowly or give a poor quality code that is not inspected. Therefore, developers would really like to rely on a fast and high-quality bot, scanning code for errors and writing patches to them.



Today their dreams come true thanks to the work of Martin Monperas and his friends from the Royal Institute of Technology in Stockholm, Sweden. They finally made a bot capable of competing with people in finding bugs and writing quality patches.

The guys called the Repairnator bot and successfully tested it, allowing it to compete with human programmers in search of fixes. “This is an important milestone on the path to competing with people in the research of automatic program correction,” they say.

Computer scientists have long been aware of the possibility of automating the process of writing patches. It is not clear whether bots can do this work as quickly and efficiently as humans.

Therefore, Monperas and the company decided to test their bot by disguising it as a human developer and allowing it to compete with people to develop patches for GitHub, a version control programming site. “The key idea of ​​Repairnator is to automatically create patches that fix bugs in builds, show them to people-developers, and watch if developers accept these patches as worthy inclusion in the code,” they said.

The team registered the user Luc Esape, supposedly a programmer from their lab, on GitHub. “Luke has a profile photo, he looks like a junior developer eager to contribute to the development of open source code on GitHub,” they say.

In fact, Luke is a undercover Repairnator. Cheating was needed because the moderators are inclined to evaluate the work of bots and people in different ways. “Stealth was necessary to test the scientific hypothesis about the competitive ability of people,” said Monpers with the company, who had already informed all interested parties about what was happening.

The team made two approaches to verifying Repairnator. The first test went from February to December 2017, when she ran Repairnator on a permanent list of 14,188 projects with GitHub in search of errors. “We found that our prototype is able to make about 30 patches per day,” they said.

During this time, Repairnator analyzed more than 11,500 projects with errors. He was able to reproduce errors in more than 3000 cases. In 15 cases, he was able to develop a patch.

But none of these patches were accepted for assembly, since Repairnator either made them too long or wrote patches of unacceptable quality.

The second approach was more successful. The team sent Luke to work on the Travis continuous integration service from January to June 2018. Although the team does not specify what exactly they changed to Repairnator, on January 12, he wrote the first patch, which was accepted by the moderator in the assembly. “In other words, Repairnator was able to reach the human level for the first time,” they say. Over the next six months, Repairnator issued five patches accepted by the moderators.

An impressive achievement that opens the way to a new generation of software development methods. It also raises interesting questions. Developers pay attention to the patch that Repairnator developed on May 12 for the eclipse / ditto project.

After that, the team received a letter from one of the project developers: “We accept pull-requests from users who have signed the Eclipse Foundation Contributor license agreement”.

This creates a nasty problem, because the bot can not sign a license agreement. “Who owns the intellectual property and who is responsible for the bot’s contribution: bot operator, bot author, fixer algorithm designer?” The development team asks.

This kind of problem is worth solving before people and bots can work together further. But Monperas and his team are optimistic about the future. “We believe that Repairnator serves as a prototype for future software development, in which bots and people collaborate without problems and become partners in finding flaws in software,” they say.

Franklin, the famous inventor himself, would surely be impressed with what was happening.