Multitouch technology

Foreword

A small article about multitouch technologies has already been published on Habré in distant October 2008. In April 2011, the DMikhail habrayuser described his experience in creating an interactive table . It's October 2012 and it's time to tell the next batch of news from the world of multi-touch.

Introduction

35 years have passed since the birth of the first multi-touch display, during which time a variety of technologies have appeared that support from one to infinity touches. In this article, I will talk about several common methods for converting a conventional display tool to an interactive device.

Projection Capacitive Technology (Projected Capacitive Touch)

The principle of operation of this technology is described in the above article by the meako habrayuzer, as well as on Wikipedia .
At the time of writing, there are films up to 100 ”on the diagonal (for 4: 3 and 16: 9 formats) with support for up to 12 independent touches. Also, the company Perceptive Pixel ( recently acquired by Microsoft corp. ) Presented a 82 ”multi-touch LCD display with support for an unlimited number of touches.
The beauty of this technology is that the sensor film is glued on the back side of the protective glass, i.e. located between the display means (LCD panel, projection screen, etc.) and the glass, thus being protected from weather conditions and mechanical damage. Some manufacturers stated that the glass thickness can reach up to 20mm. It is on this technology that most of the touch displays / terminals / kiosks installed in business centers, airports, train stations, metro and other public places are based.
The disadvantages of this technology include only the lack of tag recognition (fiducial marker, see below) and the limited area of ​​the sensor film. The last drawback can be partially offset by the possibility of stick-on-to-joint film stickers.

Optical technologies

These technologies were widely used by DIY lovers after the release of Microsoft Surface 1.0 in May 2007, and have survived to this day with virtually no change.
Of the variety of optical methods, the following are most widely used:

• Rear Diffused Illumination (DI);
• Frustrated Total Internal Reflection (FTIR);
• Diffused Surface Illumination (DSI).

Before proceeding to a detailed description of each of the technologies, I will describe the general advantages and disadvantages inherent in these methods.

Advantages:

• support for an almost unlimited number of simultaneous independent touches (in fact, the number is limited by the resolution of the cameras (s) and the power of the computer);
• the ability to make a screen round, triangular, hexagonal, etc. forms.

Disadvantages:

• since the technology is based on infrared radiation of a certain wavelength (see below), the presence of direct sunlight in the installation zone is unacceptable;
• installation of the projector and cameras (s) requires quite a lot of space behind the screen, which is not always acceptable;
• it is necessary to develop special software that makes this system work, i.e. actually write a driver.

DI Technology

The most common method (from optical), widely used in commercial installations due to its simplicity and clarity.

Diffuser - a material that disperses the flow of the projector, i.e. rear projection screen.
The substrate is an optically transparent material that gives strength to the screen.
Projector - a projector.
IR spotlight - a monochrome emitter in the infrared range invisible to the human eye.
IR camera - a camera with a narrow-band filter installed, passing only the wavelength corresponding to the wavelength of the IR illuminator.

The principle of operation is as follows:

• IR illuminators create a uniform IR background that passes through the substrate and diffuser, and only a small fraction of the radiation is reflected back.
• when you touch the diffuser with any object that reflects infrared radiation (hand, stylus, mug, etc.), in this place the camera catches a fairly bright spot on a gray background.
• the signal from the camera is processed by special software and at the output it gives out the coordinates (and sometimes the shape / size / pattern) of the bright spots present in a predetermined area of ​​the screen.

Advantages of technology:

• the ability to recognize tags (see below);
• extreme ease of implementation;
• the ability to make a screen of any most complex shape, including non-flat;
• the ability to make a screen of any size.

Disadvantages of technology:

• the lowest contrast of the image from the camera (among these three technologies);
• the possibility of false positives (when you bring your finger to the screen, but without touching it, poorly implemented and tuned software can recognize this as a full touch).

FTIR Technology

A fairly common method, however, it is extremely rare in commercial products.

Diffuser - a material that disperses the flow of the projector, i.e. rear projection screen.
The substrate is an optically transparent material, a medium for the propagation of infrared waves.
Projector - a projector.
IR LED - LED with a wavelength lying in the infrared range.
IR camera - a camera with a narrow-band filter installed, passing only the wavelength corresponding to the wavelength of the IR LED.

Along the perimeter of the substrate, lines of IR LEDs are installed, the radiation of which penetrates into the substrate and propagates there like a wave in an optical fiber (a phenomenon called total internal reflection). If the finger touches the screen, then due to a change in the refractive index, the wave passes beyond the border of the substrate and is reflected (scattered) from the finger, and this scattered radiation is detected by the camera. To enhance the contrast of the camera image between the diffuser and the substrate, a layer of the so-called “compliant layer” is applied, most often made of silicone.
Scattered infrared radiation is detected by the camera and processed by special software.

Advantages of technology:

• the highest contrast of the image from the camera (among the indicated technologies);

Disadvantages of technology:

• the need to install end IR illumination;
• does not recognize tags (see below).

DSI Technology

The technology is identical to FTIR except that the substrate is made of special plexiglass.

Diffuser - a material that disperses the flow of the projector, i.e. rear projection screen.
The substrate is a special Plexiglas Endlighten , a medium for the propagation of infrared waves.
Projector - a projector.
IR LED - LED with a wavelength lying in the infrared range.
IR camera - a camera with a narrow-band filter installed, passing only the wavelength corresponding to the wavelength of the IR LED.

The Endlighten feature is as follows: metal “dust” is placed in the material in such a way that when the plexiglass is illuminated from the end, the radiation is scattered on the “dust” particles and the self-luminescence effect of the substrate appears, while the material still remains transparent to the light.



Thus, in addition to recognition of clicks, the technology allows recognition of marks.

Advantages of technology:

• the ability to recognize tags (see below);

Disadvantages of technology:

• the need to install end IR illumination;
• the technology is tied to a specific plexiglass manufacturer.

Tags (fiducials)

The mark is a flat figure, on which a drawing is applied, contrasting in the infrared range. Most often, this is a sticker with a black and white pattern applied to a real object.


An example of labels.

Infrared radiation from the white region is reflected, and the black one is absorbed. Thus, in DI and DSI technologies, an IR camera captures a pattern that is easily recognized by software. In FTIR technology, such a focus will not succeed: the camera will see only the outline of the mark, but not its content.
A striking example of the use of labels is Reactable:

Technical details

Diffuser

During the existence of optical multitouch technologies, a large number of various films and plastics have been tried. However, Evonik 7D006 is not without reason considered the best material . It is not unreasonable because precisely such material was used in Microsoft Surface 1.0.

IR spotlight

It is selected from the range of equipment of security systems. Spotlights with two wavelengths are available on the market: 850 nm and 940 nm.

IR camera

As far as I know, any CCD is sensitive to near IR, so you can use any camera, the main thing is that the lens is without a filter that blocks infrared radiation. DIY lovers use conventional webcams for these purposes, from which IR filters are independently removed. I tried using 3 cameras: PlayStation 3 Eye (classic of the genre), Microsoft LifeCam Cinema, Logitech C910. Most of all I liked the Logitech camera, which has the largest CCD sensor and an easily removable filter.
In order for the camera to see only the “necessary” radiation, a narrow-band IR filter must be applied to the lens, passing only the wavelength corresponding to the wavelength of the IR projector. You can buy IR filters, for example, on aliexpress.com .
Of professional cameras, Point Gray cameras are most commonly used .

Projector

There are no restrictions on the projector, the main thing is to choose the right resolution, stream and lens. Quite a lot of installations use ultra-short throw projectors, which are located very close to the screen, saving space. Also, do not ignore laser-LED projectors, which at the moment are just beginning to gain popularity, the main advantage of which is the high life of the light source (more than 20 thousand hours against 3-4 thousand for lamp projectors).

Examples of commercial products

Mediascreen MonkeyBook (DI):


Microsoft Surface 1.0 (on DSI technology):

UPD MS PixelSence Technology

In January 2011, Microsoft and Samsung introduced the new Microsoft Surface 2.0.



In 2012, due to the release of the MS Surface tablet, the Surface 2.0 table was renamed SUR40.
The SUR40 table uses proprietary MS PixelSence technology.



The technology is similar to DI technology, except that instead of an IR camera, an IR sensor is used, which is “built-in” in each pixel of the LCD matrix. Thus, PixelSence allows you to recognize not only fingers, but also marks, while the thickness of the structure is minimized.