How to use “Tobii Eye Tracker 4C” to communicate with a child who cannot speak and move (SMA, Cerebral Palsy, ALS)

  • Tutorial


The methodology is based on tracking the direction of gaze that the “Tobii Eye Tracker 4C” device performs. A table with several pictures is displayed on the computer screen. If you hold your eyes on a separate picture, the corresponding phrase will be announced, for example, “drink”, “turn on its side”, “watch cartoons”.

When the child learns the letters, you can use more complex methods, like typing with your eyes .

What is required (in a nutshell)?

  1. computer
  2. Aytreker - device for tracking the direction of sight
  3. free software (from the author of the article)

Content


How it works?

Detailed instructions
1. Choosing a computer
2. Tobii Eye Tracker 4C device
2.1 Mounting an eytreker on a monitor
3. Installing software
3.1. (No longer required). Retrieve the file TobiiGazeCore32.dll version 4.1.3.938 or newer
3.2. Driver installation and calibration
3.3. (No longer required) Installing “Microsoft Visual C ++ Redistributable for Visual Studio 2012 Update 4”
3.4. Installing and configuring free software (from the author of the article)
3.4.1. Setting up the program for working in the table mode
4. Using the program
5. How to change the contents of table
6. Calling external programs(November 2018 update)
7. Controlling external devices: electrical appliances, a motorized sidecar, a model on a radio control (update November 2018).
Or can you buy a ready-made system from a reputable manufacturer instead?
Greetings and thanks

How it works?


Video illustrating the program:


Violetta learns to work with the program:


Violetta is capricious and says “No!” With the help of the program:


If you do not have an eytreker, you can still evaluate the performance of the program, since it can also work with any device that moves the mouse cursor around the screen. Drag the mouse on the screen as you would move your eyes. Stop the cursor on the pictures (no need to press anything), - this will correspond to the fixation of the look on the picture when using the eytreker.

In order for the program to work in this mode, you need to select Aeromouse as the device used (2 in the figure below), but first do not forget to put a tick in the field “Table mode (GRID) for children” (1).



detailed instructions


1. Choosing a computer


Which computer is suitable for this? I would say this: modern with average features and operating system Windows versions 7, 8.1, 10 (Windows 10 - only 64 bits). In spite of the fact that the recommendations for using the Tobii 4c eytreker (https://help.tobii.com/hc/en-us/articles/213414285-Specifications-for-the-Tobii-Eye-Tracker-4C) recommend using a computer with an Intel i5 or i7 processor, I was able to work on modern processors cheaper: i3, Pentium, Celeron, and even on Atom x5, although the work on the last two may seem to be slow.

In other words, if you already have a computer and it is not archaic, then you should try to work on it. If you are thinking of buying a computer, the main thing is not to take the old one, and do not take it with the Atom processor. Although, if there is no money at all, then you can try this, but it is better to discuss the possibility of a return.

A computer must have at least one USB port of version 2 or 3. It is better that there are at least two ports so that you can plug the mouse apart from the eytreker. (Tablets, by the way, often sin because they have only one USB port).

2. Tobii Eye Tracker 4C device


This device is manufactured by Tobii for gamers and is sold at a reasonable price. In order not to create competition for its much more expensive devices, the “Tobii Eye Tracker 4C” is made incompatible with programs for people with disabilities, produced by it.

However, the development tools released by Tobii left a loophole for me to write such programs myself. Perhaps not so perfect, but free.

Today (autumn 2018) this device can be easily purchased in several stores. You can easily find the most convenient option. I took in the spring of 2018 for 11,500 Russian rubles.

2.1 Fastening the eytreker to the monitor.


Video illustrating installation on a laptop:


A computer can be large with a stand-alone monitor or mobile (laptop). Maybe even a Windows tablet. But you need to remember that the eytreker needs to be positioned exactly under the monitor so that the mount is rigid. And if to a regular monitor, the eytreker is attached with a self-adhesive magnet that comes with it, then it has to be fastened to the tablet somehow masterly. In the photo below, the tablet and the eytreker are fixed on a piece of board, which, in turn, is connected to a bracket that rotates at the corner of the table.



You also need to remember that behind the dark glass (transparent only in the IR range) in the center of the eytreker there is a camera, and at the edges there are infrared LEDs. When fastening with clamps, it is important not to block one or the other.

The eye line should coincide with the line of the location of the eytreker. If a person is lying on his side, then the monitor should be tilted to the side.

The distance from the eyes to the monitor with the eytreker should be about 50 cm (arm's length) or a little further. Therefore, fixing the eytreker to the TV on the wall will not work - too far.

As already mentioned, there is a camera in the eytreker, and it should be aimed at the eyes. The front surface of the ytreker slightly reflects light. Try to see your eyes in it, as in a mirror, this will mean that the camera is looking at them. Tilt the monitor if necessary.

3. Software installation


Do everything exactly in this sequence!

3.1. (No longer required). Retrieve file TobiiGazeCore32.dll version 4.1.3.938 or newer.



Starting with version 'F' of the program (from 03/04/2019) this step is not necessary
Ранее этот файл можно было скачать в составе Tobii Gaze SDK. Сейчас компания Tobii прекратила его предоставлять, и приходится пользоваться трюком, описанным ниже.

Нужно скачать и установить программное обеспечение Tobii EyeX 2.1.1. После этого в одном из каталогов появится этот файл. Нужно скопировать его в надёжное место, после чего установленное программное обеспечение можно удалить.

Устанавливать это программное обеспечение нужно ДО установки драйверов от Tobii, иначе нужный нам файл TobiiGazeCore32.dll может не появиться (кому интересно, мы решали эту проблему вот здесь: github.com/MastaLomaster/bkb/issues/6 )

Итак, скачиваем Tobii EyeX 2.1.1 ( files.update.tech.tobii.com/engine/Tobii.EyeX_2.1.1.293.exe) и запускаем. Соглашаемся с условиями: ставим галочку и жмём кнопку ‘Next”.



Ждём…



Готово, жмём “Finish”.



После установки появилось два файла TobiiGazeCore32.DLL. Более новый лежит в каталоге “C:\Program Files (x86)\Tobii\Tobii EyeX”. Это можно проверить, если нажать на имени файла правую кнопку мыши и выбрать «Свойства» в контекстном меню.



И затем, на закладке «Подробно» посмотреть на номер версии:



Копируем этот файл в надёжное место, например, создадим каталог BKB на диске C и положим файл туда:



Теперь можно удалить программу Tobii EyeX – мы взяли из неё всё, что нужно.
В Windows 10 для этого жмём кнопку «Пуск», потом — появившуюся кнопку с шестерёнкой.



Затем выберем раздел «Приложения»



Находим и удаляем всё, что начинается на “Tobii”, нажимая на кнопку «Удалить»:




3.2. Driver installation and calibration


Go to the tobiigaming.com/getstarted page and click on the “Tobii Eyetracking” button:



Then click on the “Download” button that appears:



Save the file (for the future), and then launch it:



Put a tick, click the “Next” button : We’re



waiting ...





We’ll click “Continue”:



Then press the “Get Started” button:



Next you need to drag the lines on the screen with the mouse so that they match the stripes on the eytreker case:





When done, click the “Done” button.

Then two mugs will appear on the screen that illustrate if the Aytreker sees your eyes well.



If the circles are not located in the center of the screen (as in the picture above), tilt the monitor to move them.
If the circles are large and dull - you are too close moved to the monitor.
If the circles are small and dull - you are too far from the monitor.

As I wrote above, the optimal distance from the monitor to the eyes is about 50 cm.

Then the calibration of the eytreker begins.

IMPORTANT!!!

If the child uses the eytreker, then the calibration will need to be performed through the eyes of the child !!


To begin with, you can calibrate the device under your eyes, and then under the child's eyes.
Blue dots will appear on the screen, you will need to look at each of them in turn.



After you fix your eyes on one of the points, it will shatter into pieces, as the top point in the picture above.

After calibration, enter the name of the person whose calibration was performed and press the “Save” button.



Then the game will start, in which you move the sight of the spacecraft with your eyes, and then shoot the asteroids with the spacebar:



You can exit the game by pressing the “ Escape.

An icon with two circles depicting the eyes will appear on the taskbar:



If the device sees only one eye, only one circle will be shown:



If the device does not find the eye at all, two crosses will be drawn:



IMPORTANT! In order for the programs not to interfere with each other, you need to disable the standard actions that the Tobii driver produces.

To do this, click on the icon with circles (1) and in the menu that opens, go to the Interactions section (2):



In each section (Touchpad, Mouse, Windows), turn all switches to “OFF”:







If you want the device to work one person, you need to perform a calibration for his eyes too. To do this, again click on the icon with two circles, and then on the left arrow:



Then select the item “Create New Profile ...” (1) and repeat the calibration for the new person.

Switching between profiles of different people is done in the area below (2). The current profile is marked with a blue circle:



3.3. (Not needed anymore) Installing “Microsoft Visual C ++ Redistributable for Visual Studio 2012 Update 4”


Starting with version 'F' of the program (from 03/04/2019) this step is not necessary
Если вы пропустите этот шаг, то при запуске программы вы увидите сообщение о том, что не найден файл msvcrt110.dll:



Заходим по ссылке: www.microsoft.com/ru-ru/download/details.aspx?id=30679



Качаем и устанавливаем версию для процессоров x86:












3.4. Installing and configuring free software (from the author of the article)


Go to the link.

After that click the button “Download” or “View Raw”



Save the file.



(if anyone is interested, the source codes in C ++ are also nearby)

Create the “C: \ bkb” directory and copy the contents of the downloaded “bkb32c-Russian.zip” archive there.

You should have a directory with such files and the “grid” subdirectory:



Generally, you can choose another directory for the program, but it’s better that this is not the “Program Files ...” subdirectory. The point is that the program can save anything (for example, an error log) in this directory, and the “Program Files ...” subdirectories are closed for writing. (Yes, I know that I had to write in% AppData%, but historically it happened).

3.4.1. Setup of the program for work in the table mode

The program is written in such a way that if you do not click on its first window, then it starts to determine for itself which device it should work with. This is done in order to be able to put it into autoload on the computer of a person who himself cannot press anything.

Therefore, when starting the program, we do not catch the crows, but quickly put a tick in the field “Table mode (GRID) - for children” (1).



I also recommend that you click the “All Settings” button (2) and set a longer glance time on the pictures, as well as a wait time for the next glance:



Specification 03.10.2018: In the latest version of the program, the fixation time for the table is taken from the settings and multiplied by 3, so that you may not need to increase it.

We press the “Save” button and select our Aytreker, for example, Tobii (3).

If you do not have time to tick or enter the settings, and the program has started, close it and try again.

You can exit the program only by closing its window in the taskbar:



4.Use of the program


After launching it in the table mode, the program will unfold the following image on the whole screen:



Collapse this table to one cell by looking at the left arrow (minimize). For example, to watch cartoons.

If you look at the folded table, it will again unfold on the whole screen.

If you look at the green circle, the word “Yes” will be pronounced.

On the red - the word “No”.

At the bell - the sound of the bell.

On one of the pictures - the corresponding phrase:
Table cell
0. Stomach ache.
1. To drink.
2. To eat.
3. How to.
4. Swim.
5. Watch cartoons.
6. Draw.
7. Turn on its side.
8. Sleep
9. Watch the window.
a.Cool forehead.
b.Read a fairy tale.


As already mentioned, you can exit the program only by closing its window in the taskbar.

5. How to change table contents


You can change the contents of the table by putting your own files in the grid directory.

For pictures, these are files in BMP format. (It is advisable to save them in Microsoft Paint, otherwise the program may not read them).

For sounds, these are files in WAV format.

In the first cell of the table, the 0.BMP file is displayed and 0.WAV sound is played.
In the second cell, 1.BMP and 1.WAV
, there can be up to 16 cells at the same level, they are numbered as follows:
0,1,2,3,4, 5,6,7,8,9, A, B, D, E, F

That is, the F.BMP and F.WAV files will be attached to the 16th cell

. Additional levels of the table can be made.
For example, when selecting the “Eat” cell, you can open a table with a choice of food.

In the second-level table, files are referred to as two-digit values.
For example, “Eat” at the first level has the number 2, and from it we get to the table with the cells starting at 2: 20, 21, 22 ……

Try this:

Take a 2.BMP file and copy it into the 20.bmp file and another file 21.bmp. Then in the graphical editor, edit the file 20.bmp so that a plate of porridge appears there. And edit the file 21.bmp so that there is an image of mashed potatoes.

Now record two sound files: 20.wav and 21.wav. In the first record how you pronounce the word “porridge”, and in the second - the word “mashed potatoes”.

Now restart the program, and when the child looks at the picture with the word “eat,” after pronouncing this word a new table will open with your pictures 20.bmp, and 21.bmp. And when you stop looking at these pictures - the words recorded in the files 20.wav and 21.wav will be pronounced, respectively.

There can be three levels in total, as shown below:



Cells of the next level add another digit to the name of the top-level file:
2-food, 20-cereal, 200-semolina porridge.

After adding new files to the grid directory, the program must be restarted for it to see them.

Record your sounds, create your own pictures, combine them into your own tables.

Examples of screens created by other people:







6. Calling external programs (update November 2018)


Now you can call external programs by simply adding a file with the .cmd extension and the same name as the file with the .bmp extension.

For example, for the 5.bmp image you can create the 5.cmd file , and if you look at this picture in the table, not only will the 5.wav sound be played , but the 5.cmd file will be launched .

When you call the external program, the window with the table is minimized, turning into a small box with an arrow so that you can see the window of the running program.

7. Control of external devices: electrical appliances, a motorized sidecar, a model on a radio control (updated in November 2018)



A separate article is devoted to this: habr.com/post/430078
Here, I will simply provide video clips showing how this can work:







Can one buy a ready-made system from a reputable manufacturer instead?


Sure you may. But they ask for money for software, and these programs do not work with cheap gamer aytrekery.

For example, a tracker with a basic program for $ 1,700 .

Tables for communication - Grid 3 for £ 480

Tables simpler - Snap + Core First for $ 180

- I don’t know if they work with the Russian language and keyboard.

Greetings and thanks


I made the table mode for children’s communication with the suggestion of Alexander Makarchuk, the head of the “Owl's charity foundation” ( sova.by ) from Belarus. Alexander also helped me with requirements collection and testing. The Foundation, among other things, organizes distance education for people with disabilities with subsequent employment. Sasha, hello and success to you!

Also popular now: