Controlling a computer with your eyes - practical implementation

1. The need and problem statement

The article is written primarily for people who find it difficult or impossible to work on a computer in the generally accepted way: by pressing buttons on the keyboard and moving the mouse. I will try to tell how a system was made that allows you to control a computer with just one glance. Immediately make a reservation that the above solution may not work well if a person wears astigmatic glasses / glasses with lenses of different diopters .
So, there are only eyes and a computer. Nothing else can be used - neither pedals, nor turning the head, nor converting speech into text.

2. The choice of existing solutions

A search on the Internet for the keywords “Eye Gaze Tracking” and “Assisitive Technology” gives a lot of links, after digging in which I come to the conclusions:

• Ready-made devices exist, but they cost amazing money; in Russia no one sells them, does not serve them; the software does not provide for working with the Russian language (in particular, there is no keyboard with a Cyrillic alphabet).
• There is open source software aimed at creating home-made systems. Tests with your camera do not give an acceptable quality of work.
• There is an opportunity to buy devices for developers (developer edition) at a reasonable price from Tobii and TheEyeTribe. But according to the scheme “in the morning - money, in the evening - chairs”, that is, pre-order can be issued in the fall of 2013, and to receive the device - in early 2014.

For reference, in the section “5. Other methods that I have not tried ” provide a description of the different devices about which I typed information.

Suddenly, in mid-December, Tobii changed its concept, switched to developing a new EyeX device, and sent its old REX device to developers at a price three times lower than the original one and right now, assuring that the code written for REX would work with EyeX.
He did not think for long, ordered REX, on the basis of which the system was made.

[correction from 08/22/2017] At the moment, there are more modern IT trackers that can also be used (see below)

3. Implementation Details

3.1. Hardware

3.1.1. Option One: Tobii Eye Tracker 4C

The most relevant model of an inexpensive IT tracker for the summer of 2017.

I recommend this particular device. Compared to EyeX:

• sold in Russia (about 11,500 rubles)
• does not require a USB 3.0 port, USB 2.0 is enough
• shines less in the eyes with red bulbs
• processor load less
• the USB wire is hard-wired in the case, and does not fall out of the mini-USB connector
• newer iron

3.1.2. Option Two: Tobii EyeX

Previous model from Tobii. Now you can still buy it, but only if it’s very tight with the money. Get ready to stare at the red bulbs. Nevertheless, a very worthy device for its time.

Attention! EyeX requires USB3 to work! If your computer does not have USB3 ports, the device will not work!

You can order the device on the Tobii website .
Delivered by UPS from Sweden.

3.1.3. Other supported devices

Tobii REX - no longer for sale, but there was a very good machine with two cameras.

TheEyeTribe - no longer for sale. So-so piece of iron. Although it made a breakthrough in the field of low-budget IT trackers, forcing Tobii to significantly reduce prices.

GazePoint GP3 is a very expensive device, at the same time inferior to cheaper Tobii devices.

3.2 Software

Complete with devices are only demo programs designed to show what the device is capable of, and tools for writing your own programs (SDKs). The documentation that comes with the SDK can be said to be none. But there are examples in C ++ and C #, having studied which, you can understand how and what to do.

When trying to use software from other, more expensive Tobii models, called Tobii Gaze Interaction, nothing came of it. But I liked the idea implemented there.
So, I had to write the software myself.

A program was written that allows you to simulate with your eyes the actions of the mouse: left, right, double click, drag-and-drop, scroll wheel, and also type text with your eyes on the virtual keyboard.
It is better, however, to see once. I invite you to see the demonstration of the program here .


The program is available for download (including in source codes) on github: here .
To download, you need to click the "RAW" button - as is customary on github.

Detailed instructions for installing the program are given here . Let it not bother you that only work in table mode is described there.

The program came in handy to several people from different countries with amyotrophic lateral sclerosis (ALS). Here in this clip of the Charity Foundation one of these stories is told.

4. The problem of working with glasses

The device used, like most other similar ones, uses glare on the pupils, which are obtained from infrared illumination, to determine the direction of view. Unfortunately, glasses add spurious glare.

Despite the fact that when setting up the device for a specific user, it is possible to indicate that a person is wearing glasses, this does not help much, the device constantly loses the position of the pupils.
I would be grateful for the proven recommendations for eliminating glare on the lenses of glasses in the infrared range.

5. Other ways that I have not tried

[outdated information, this was relevant in 2013-2014]

I must say right away that devices are used not only to facilitate work on a computer for people with disabilities, but also, for example, to study which parts of web pages users pay the most attention to . Therefore, sometimes for such systems monstrous monstrous money. However, amazing devices often get money for devices for the disabled.

5.1. Products of commercial companies

SMI - senso-motoric instruments

Their devices can be viewed here .
Devices can be bought in Russia (I will not give a link), but when I asked about the price of the cheapest, I was announced the price of one million two hundred thousand rubles. Goodbye, SMI.

Tobii

The company has been releasing products for people with disabilities for a long time. Even Intel invested several million dollars in them. They had an alliance with Dell, when the tablet came with a piece of iron for eye tracking, and Dell provided technical support.
You can read in Russian here .
In English - on their website .
You can watch a lot of things about these devices on youtube.
More recently, prices started at $ 4,000. Currently (January 2014), the price of PCEye has dropped to $ 2,000, but the software has not been Russified (Tobii Gaze Interaction).

Eyetech

They have also been working in this area for a long time, it seems from the end of the 90s. Their devices are more clumsy, but I really liked the software, a demonstration of which can be seen here .

At the same time, there is a clip in which a person blinks his eyes to click on the mouse button - this is not like it, you blink. In Tobii, I have not seen this.
There are three devices that differ only in size, for different monitors, from laptops to 30 inches. The price I managed to find is $ 8700. Gentlemen are not shy.
But we will notice how their software works.

Dynavox

Specialized device , you can not use standard computer programs.

EyeGaze Edge (from LC Technologies)

A monstrous device that includes a camera and a computer that processes its data. You can connect this entire unit to your computer (including wirelessly), then it will replace the mouse and keyboard, which will allow you to work with ordinary programs. Again 8-10 thousand dollars.
It seems to be a set of “EyeGaze Edge Pack”, where you use your computer, just push their camera and software into it, but you haven’t found a price for it anywhere.

These are the main players in the market. One of the small new ambitious offices, which itself could not bring its product to mind, nevertheless was not too lazy to criticize the leaders and brought here such a tablet, from which it is clear that they themselves are the coolest (although there is no product yet, and a demo video shows that the position of the pupil is constantly jumping). But a comparison of the rest is very revealing. Although the equipment there is often already outdated, it is possible to evaluate the order of prices and who has moved on.

5.2. Open source projects

Unfortunately, none of the programs listed below gave a sufficiently accurate determination of the direction of view on existing equipment (PC104E camcorder with night mode and a separate infrared LED backlight). However, perhaps someone will work better.

GazeTracker

I recommend starting with watching a clip on this page .
The project was developed at the University of IT of Copenhagen. It developed somewhere until the end of 2010, after which it fell into apathy. Nevertheless, many respect for their openness, non-commercialization and certain achievements. Initially, it was aimed at the ability to work with equipment such as "do it yourself" (DIY - do it yourself). Many homeowners used this software with their pieces of iron, often costing them $ 20-30. See, for example, this .
The current version is 2.0 beta, before that there were a number of stable versions, for example 1.6, 1.8.
The instructions for use mention that the developers themselves used a Sony HDR-HC5 camera with one or two backlight from the same Sony, model HVL-IRM.
The site has a [relatively] live forum, you can talk, if not with developers, then with apologists.

Haytham

The University of IT in Copenhagen did not give up work after the GazeTracker team collapsed. Now they have a much more beautiful system :

This is not one program, but a server that distributes the position of the pupil to clients on the network. One client is the Haytham_Monitor program, which controls the mouse cursor. If she worked well on my hardware, there would be no need to do anything. Good luck guys.