An application combining crowdsourcing and machine learning can track anything

The Zensors application , which uniquely combines crowdsourcing and machine learning to process information from any images and create notifications based on them, has entered the beta phase and offers everyone to take part in testing. The project was presented this week at the conference “Human-Computer Interaction” in Seoul.

Startup created by students of Carnegie Melon University offers the opportunity to turn a smartphone(or webcam) to an intelligent surveillance device. You open in the application the picture that the camera shows, trace the fragment of interest with your finger, and ask in a free form a question regarding this fragment. Is there a parking space? Is there a queue at the cash register? Is there more food in the bowl? Are ATMs stolen from the store? Have sausages brought into the dining room?

The question asked is submitted to the crowdsourcing platform. In the testing process, the developers used the Amazon's Mechanical Turk platform. People for a nominal fee check the image and in case of an event, an alert arrives in the system and comes to your smartphone.


Brief job description

An interesting feature of Zensors is machine learning, which takes place in parallel with the work of crowdsourcing. The system compares people's answers with images and at some point gains the opportunity to independently answer the question. In this case, the process is automated, and only sometimes the image is still sent for human verification to control the operation of the algorithm. During the tests, the developers found out that the cost of processing one picture is 2 cents, and the cost of training the algorithm for battery life is about $ 15. Such costs are not comparable with the time and money spent on writing a similar program that recognizes images on order.

“Natural language processing, machine learning, and computer vision are the three biggest challenges in the computer field,” says Chris Harrison , an assistant professor at the university who specializes in human-computer interactions. - Using crowdsourcing allows us to circumvent these difficulties. At the same time, we use people for tuning work, and we get all the benefits of machine learning. ”