Eye movement research: eye tracking without a video camera and other solutions
Is it possible to collect data on the eye movements of all planetarium visitors without special equipment? How to study visual attention during the active actions of several people without constructing a complex system with IT trackers? How to achieve exceptional accuracy and record eye movements with a frequency of 8000 Hz? In our article, we will try to answer these questions.
Currently, there are well-known IT trackers based on the method of video recording of eye movements: these are trackers of famous brands like EyeLink , Tobii , as well as various hand-made solutions and developments based on webcams (we wrote about this in a previous articlededicated to IT tracking). However, there are still many interesting and unusual approaches for studying the visual attention of a person.
In 2016, Wang et al. Published a new eye tracking system based on infrared thermography. Efron et al., (1989) showed that the edge of the cornea is 0.45 ° warmer than its center, and this discovery formed the basis for the development of thermal eye tracking.
Wang et al., 2016, in order to validate their system, simultaneously recorded eye movements using video oculography (Eye Link, 500 Hz) and thermography (60 Hz) methods in ten subjects. The median verification error was 1.4 ° compared to EyeLink, which shows the good potential of this method for its further application.
It is curious that thermography is also successfully used to recognize the emotional state of a person (for example, see Park et al., 2013) and respiratory rate (for example, see Al-Khalidi et al., 2011). Thus, thermal IT tracking is becoming a promising technique for obtaining multimodal data.
The study of the visual attention of a large number of people at one time is a very attractive task. However, its implementation, at first glance, seems to be a difficult and expensive process (at least already at the stage of purchasing, debugging or creating equipment with its subsequent synchronization).
Shillcock, Wase (2015) proposed an original technique for recording the visual attention of a crowd of people, for the implementation of which only a box of pens, a pack of paper and their own stimulus material are required. The experiment was successfully reproduced by Bielecki et al. (2017) in the study of the visual attention of planetarium visitors using a 16-meter hemispherical screen.
The essence of the technique is that the presentation is periodically, for a short period of time, interrupted by a slide with a grid, in each of which there is a letter or some other symbol. The task of the subjects is to write down the symbol that they just saw. Thus, it is possible to obtain heat maps of the visual attention of the entire audience as a whole, which allows us to analyze its distribution among the lecture participants not only depending on its content, but also on the location of an individual participant in the hall.
Of course, this technique is not applicable for "field" experiments when there is no stimulus material in the form of video or images. However, for the natural conditions of the experiment, too, their tricky solutions were found.
The study of oculomotor behavior in the most natural, everyday, everyday conditions is the dream of many researchers of visual attention. If the task of recording eye movements in case of active actions can be handled relatively well by eye-tracking glasses (for example, Ronaldo kicks a ballin glasses-tracker), then for an qualitative study of oculomotor behavior during social interactions, such an experimental design is not suitable, since eye contact, facial expressions of the whole face are important elements of non-verbal communication. To date, there are no software soft-trackers available that would receive the gaze trajectory approximately the same as when using IT-trackers (but our team is in the process of developing and testing the first EyeCatcher soft-tracker that can replace a full-fledged laboratory tracker) . G3E
Projectfocused on extracting the person’s visual attention data from the video, but its initiators abandoned the idea of obtaining gaze paths in the form of changing the coordinates of the pupil’s position and focused on obtaining the direction of the gaze in the context of the task of determining whether the subject is looking at the interlocutor or not. This system was developed on the basis of algorithms for predicting the position of the gaze depending on the position of the head (Ba, Odobez, 2006) and gaze appearance model (this model includes pairs of images with the position of the eye in the orbit and the corresponding direction of view) (Mora, Odobez , 2013). In addition, to improve the functional parameters of the system, we used data tagged with annotators video (Siegfried R., Odobez, 2017).
Video annotation has found its application not only as an effective tool to improve the quality of the algorithms, but also as an independent method for studying eye movements.
So in a study of visual attention and behavioral strategies during the card game Campos et al., 2015 recorded each of the two players on camera. After that, the video was marked out by two annotators in the ELAN program in three categories: look-at-game, look-at-other and look-elsewhere, the consistency of annotators was calculated using Cohen's kappa and compared the results with strategies of behavior (cooperation, competition and avoidance games). It turned out that with the motivation for cooperation, the time devoted to viewing the face of the interlocutor is significantly greater, as well as mutual eye contacts. When motivated to compete, on the contrary, there are short but frequent glances at the interlocutor’s face, a tendency to avoid eye contact is singled out.
Jarik, Kingstone, 2015, in the study of the dynamics of mutual eye contact after the game in the conditions of cooperation and competition, also turned to the method of annotating the video (in this study, as in the previous one, two annotators participated). Here, the subjects after the cooperative game, the duration of eye contact was shorter, more often interrupted compared with another group of participants in the experiment.
In terms of alternatives to the method of videooculography, contact methods should also be mentioned (all the tracking started with them!). A separate article can be written on the history of IT tracking (creepy suckers attached to the eyeball, or, for example, a description of mechanical recorders the size of a whole table, require a detailed text with explanations), so we will stop only at the present stage of development of this group of methods. Among the contact methods of registering eye movement that are actively used today, it is worth highlighting electromagnetic eye tracking and the EOG method.
A prime example of electromagnetic trackers are the Primelec Swiss IT trackers.that can detect eye movements with extremely high accuracy - 0.0002 ° and a sampling frequency of 8000 Hz. The principle of their operation is based on the induction of electric current caused by the movement of a metal circuit in a magnetic field. The movable element is a ring with two inductance circuits located inside, which is worn on the subject’s eye like a contact lens (eyewitnesses say that installing this magical “lens” is not pleasant). Using such instruments, it turns out to study various features of eye micromotion, including their changes in astronauts in zero gravity.
Electrooculogram (EOG) is a method of recording electrical activity that occurs when the eye moves. Since the cornea of the eye has a positive charge relative to the retina, this creates a constant (corneoretinal) potential. When the position of the eye changes, this potential is corrected. EOG is recorded by electrodes, which are installed near the nasal and temporal angle of the palpebral fissure (for recording horizontal saccades ), as well as above the eyebrow and under the eye, for recording vertical saccades. This method is still widely used in medical practice, sometimes in combination with the EEG method .
The Neurodata Lab IT tracking and visual sensing system working group continues its work.
Material author:
Maria Konstantinova, Researcher at the Neurodata Lab , biologist, physiologist, specialist in the visual sensory system, oculography and oculomotorics.
Currently, there are well-known IT trackers based on the method of video recording of eye movements: these are trackers of famous brands like EyeLink , Tobii , as well as various hand-made solutions and developments based on webcams (we wrote about this in a previous articlededicated to IT tracking). However, there are still many interesting and unusual approaches for studying the visual attention of a person.
Thermal tracking
In 2016, Wang et al. Published a new eye tracking system based on infrared thermography. Efron et al., (1989) showed that the edge of the cornea is 0.45 ° warmer than its center, and this discovery formed the basis for the development of thermal eye tracking.
Wang et al., 2016, in order to validate their system, simultaneously recorded eye movements using video oculography (Eye Link, 500 Hz) and thermography (60 Hz) methods in ten subjects. The median verification error was 1.4 ° compared to EyeLink, which shows the good potential of this method for its further application.
It is curious that thermography is also successfully used to recognize the emotional state of a person (for example, see Park et al., 2013) and respiratory rate (for example, see Al-Khalidi et al., 2011). Thus, thermal IT tracking is becoming a promising technique for obtaining multimodal data.
Collective IT tracking
The study of the visual attention of a large number of people at one time is a very attractive task. However, its implementation, at first glance, seems to be a difficult and expensive process (at least already at the stage of purchasing, debugging or creating equipment with its subsequent synchronization).
Shillcock, Wase (2015) proposed an original technique for recording the visual attention of a crowd of people, for the implementation of which only a box of pens, a pack of paper and their own stimulus material are required. The experiment was successfully reproduced by Bielecki et al. (2017) in the study of the visual attention of planetarium visitors using a 16-meter hemispherical screen.
The essence of the technique is that the presentation is periodically, for a short period of time, interrupted by a slide with a grid, in each of which there is a letter or some other symbol. The task of the subjects is to write down the symbol that they just saw. Thus, it is possible to obtain heat maps of the visual attention of the entire audience as a whole, which allows us to analyze its distribution among the lecture participants not only depending on its content, but also on the location of an individual participant in the hall.
Of course, this technique is not applicable for "field" experiments when there is no stimulus material in the form of video or images. However, for the natural conditions of the experiment, too, their tricky solutions were found.
Recording eye movements in vivo
The study of oculomotor behavior in the most natural, everyday, everyday conditions is the dream of many researchers of visual attention. If the task of recording eye movements in case of active actions can be handled relatively well by eye-tracking glasses (for example, Ronaldo kicks a ballin glasses-tracker), then for an qualitative study of oculomotor behavior during social interactions, such an experimental design is not suitable, since eye contact, facial expressions of the whole face are important elements of non-verbal communication. To date, there are no software soft-trackers available that would receive the gaze trajectory approximately the same as when using IT-trackers (but our team is in the process of developing and testing the first EyeCatcher soft-tracker that can replace a full-fledged laboratory tracker) . G3E
Projectfocused on extracting the person’s visual attention data from the video, but its initiators abandoned the idea of obtaining gaze paths in the form of changing the coordinates of the pupil’s position and focused on obtaining the direction of the gaze in the context of the task of determining whether the subject is looking at the interlocutor or not. This system was developed on the basis of algorithms for predicting the position of the gaze depending on the position of the head (Ba, Odobez, 2006) and gaze appearance model (this model includes pairs of images with the position of the eye in the orbit and the corresponding direction of view) (Mora, Odobez , 2013). In addition, to improve the functional parameters of the system, we used data tagged with annotators video (Siegfried R., Odobez, 2017).
Video annotation has found its application not only as an effective tool to improve the quality of the algorithms, but also as an independent method for studying eye movements.
So in a study of visual attention and behavioral strategies during the card game Campos et al., 2015 recorded each of the two players on camera. After that, the video was marked out by two annotators in the ELAN program in three categories: look-at-game, look-at-other and look-elsewhere, the consistency of annotators was calculated using Cohen's kappa and compared the results with strategies of behavior (cooperation, competition and avoidance games). It turned out that with the motivation for cooperation, the time devoted to viewing the face of the interlocutor is significantly greater, as well as mutual eye contacts. When motivated to compete, on the contrary, there are short but frequent glances at the interlocutor’s face, a tendency to avoid eye contact is singled out.
Jarik, Kingstone, 2015, in the study of the dynamics of mutual eye contact after the game in the conditions of cooperation and competition, also turned to the method of annotating the video (in this study, as in the previous one, two annotators participated). Here, the subjects after the cooperative game, the duration of eye contact was shorter, more often interrupted compared with another group of participants in the experiment.
Good old contact methods
In terms of alternatives to the method of videooculography, contact methods should also be mentioned (all the tracking started with them!). A separate article can be written on the history of IT tracking (creepy suckers attached to the eyeball, or, for example, a description of mechanical recorders the size of a whole table, require a detailed text with explanations), so we will stop only at the present stage of development of this group of methods. Among the contact methods of registering eye movement that are actively used today, it is worth highlighting electromagnetic eye tracking and the EOG method.
A prime example of electromagnetic trackers are the Primelec Swiss IT trackers.that can detect eye movements with extremely high accuracy - 0.0002 ° and a sampling frequency of 8000 Hz. The principle of their operation is based on the induction of electric current caused by the movement of a metal circuit in a magnetic field. The movable element is a ring with two inductance circuits located inside, which is worn on the subject’s eye like a contact lens (eyewitnesses say that installing this magical “lens” is not pleasant). Using such instruments, it turns out to study various features of eye micromotion, including their changes in astronauts in zero gravity.
Electrooculogram (EOG) is a method of recording electrical activity that occurs when the eye moves. Since the cornea of the eye has a positive charge relative to the retina, this creates a constant (corneoretinal) potential. When the position of the eye changes, this potential is corrected. EOG is recorded by electrodes, which are installed near the nasal and temporal angle of the palpebral fissure (for recording horizontal saccades ), as well as above the eyebrow and under the eye, for recording vertical saccades. This method is still widely used in medical practice, sometimes in combination with the EEG method .
The Neurodata Lab IT tracking and visual sensing system working group continues its work.
Literature
Al-Khalidi F., Saatchi R., Elphick H., Burke D. An evaluation of thermal imaging based respiration rate monitoring in children. 2011. American Journal of Engineering and Applied Sciences 4 (4). P. 586 - 597.
Ba.SO, Odobez J.-M. A Study on visual focus of attention recognition from head pose in a meeting room. 2006. 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI).
Bielecki M., Potega vel Zabik K., Gochna M., Mikulski J. Mass measurement of eye-movements under the dome - proof of concept studies. 2017. ECEM 2017 Program and Abstracts Eds. Radach R., Deubel H., Vorstius C., Hofmann MJP121.
Campos J., Alves-Oliveira P., Paiva A. Looking for conflict: gaze dynamics in a dyadic Mixed-Motive Game. 2015. Auton. Agent Multi-Agent. Syst. 30 (1). P.112-135.
Efron N., Graeme Y., Brennan NA Ocular surface temperature. 1989. Current eye research 8 (9). P.901-906.
Funes Mora, KA, Odobez, J.-M. Person Independent 3D gaze estimation from remote RGBD cameras. 2013. In International conference on image processing. IEEE
Park KK, Suk HW, Hwang H., Lee J. A functional analysis of deception detection of a mock crime using infrared thermal imaging and the concealed information test. 2013. Frontiers in Human Neuroscience 7 (70). P.1 - 17.
Shillcock R., Wase C. Mass measurement of fixation behaviors and audience nature in a realistic educational setting. 2015. ECEM 2015 Program and Abstracts Eds. Ansorge U., Ditye T., Florack A., Leder HP66.
Siegfried R., Odobez J.-M. Supervised gaze bias correction for gaze coding in interactions. COGAIN Symposium Wuppertal, August 21st 2017.
Wang Q., Boccanfuso L., Li B., Ahn AY, Foster CE, Orr MP, Scassellati B., Shic F. Thermographic Eye Tracking. 2016. Conference paper: Proceedings of the Symposium on Eye Tracking Research and Applications.
Okutin O.L., Okutina G.Yu. The main characteristics and possibilities of using Primelec hardware to study micromotion of the eyes (Report on visits to the Department and Laboratory of Neurology of the Clinic of the University of Zurich). 2011. Experimental Psychology 4 (3).
Ba.SO, Odobez J.-M. A Study on visual focus of attention recognition from head pose in a meeting room. 2006. 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI).
Bielecki M., Potega vel Zabik K., Gochna M., Mikulski J. Mass measurement of eye-movements under the dome - proof of concept studies. 2017. ECEM 2017 Program and Abstracts Eds. Radach R., Deubel H., Vorstius C., Hofmann MJP121.
Campos J., Alves-Oliveira P., Paiva A. Looking for conflict: gaze dynamics in a dyadic Mixed-Motive Game. 2015. Auton. Agent Multi-Agent. Syst. 30 (1). P.112-135.
Efron N., Graeme Y., Brennan NA Ocular surface temperature. 1989. Current eye research 8 (9). P.901-906.
Funes Mora, KA, Odobez, J.-M. Person Independent 3D gaze estimation from remote RGBD cameras. 2013. In International conference on image processing. IEEE
Park KK, Suk HW, Hwang H., Lee J. A functional analysis of deception detection of a mock crime using infrared thermal imaging and the concealed information test. 2013. Frontiers in Human Neuroscience 7 (70). P.1 - 17.
Shillcock R., Wase C. Mass measurement of fixation behaviors and audience nature in a realistic educational setting. 2015. ECEM 2015 Program and Abstracts Eds. Ansorge U., Ditye T., Florack A., Leder HP66.
Siegfried R., Odobez J.-M. Supervised gaze bias correction for gaze coding in interactions. COGAIN Symposium Wuppertal, August 21st 2017.
Wang Q., Boccanfuso L., Li B., Ahn AY, Foster CE, Orr MP, Scassellati B., Shic F. Thermographic Eye Tracking. 2016. Conference paper: Proceedings of the Symposium on Eye Tracking Research and Applications.
Okutin O.L., Okutina G.Yu. The main characteristics and possibilities of using Primelec hardware to study micromotion of the eyes (Report on visits to the Department and Laboratory of Neurology of the Clinic of the University of Zurich). 2011. Experimental Psychology 4 (3).
Material author:
Maria Konstantinova, Researcher at the Neurodata Lab , biologist, physiologist, specialist in the visual sensory system, oculography and oculomotorics.
