PixelTone: Voice Protected Image Editor Prototype

    At the University of Michigan, together with Adobe Research, they created a prototype of a graphic editor controlled by voice and gestures. The prototype, called PixelTone , so far can only do the simplest operations and gestures. He understands simple commands and is guided in descriptive terms like “top”, “bottom”, “brighter”, “darker”. In addition, he can remember the names of objects and the names of people in photographs, which allows you to use commands like “Make Vasya a little more contrast”.



    Such a combined interface can be useful on tablets and phones, where there is simply no place for numerous palettes and menus, and there is no keyboard to use hot keys. The prototype will be presented at the Computer-Human Interaction conference which will be held in Paris from April 27 to May 2. In the meantime, you can download a PDF with a description of the technology.

    This document, prepared according to all the rules of a serious scientific publication, is perhaps more interesting than the prototype itself. When creating the interface, scientists analyzed the system of generally accepted abbreviations and annotations for photographs, which was developed among professional photographers and retouchers, and then using Amazon Mechanical Turk collected statistics of words and phrases that non-professional users use to describe the desired changes in the photograph.

    Usability studies conducted after the creation of the prototype showed that the voice interface allows you to perform basic photo editing operations no worse than the traditional one. The joint use of voice and touch allows you to accurately and confidently interpret the fuzzy formulations of the natural language like "here", "from here and above." As with keyboard shortcuts, it takes some time to learn and remember phrases that the program understands, which may require graphic hints. Having mastered, the user can control the editor almost exclusively by voice and direct manipulation on the touchscreen.


    Also popular now: