Long Polling A to Z DIY

How to implement long polling using Nginx and Javascript on the network is a lot of material. But I have not yet met a complete guide. There are problems with compiling the module under Nginx, then the download icon for the long poll requests turns in the browser. Under the cut, full material, how can you do it right?

Compiling Nginx module under linux

To support long polling connections in the Nginx server, the wonderful nginx-push-stream-module module is implemented . Since it is not included in the official delivery, it needs to be downloaded, configured and compiled with Nginx.

Before this, you must have all the necessary packages installed.

apt-get install git
apt-get install make
apt-get install g++
apt-get install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev

Next, you need to download the nginx-push-stream-module, nginx module itself and compile them together.

Clone a project from GIT

git clone http://github.com/wandenberg/nginx-push-stream-module.git

Download and unpack the latest nginx

NGINX_PUSH_STREAM_MODULE_PATH=$PWD/nginx-push-stream-module
wget http://nginx.org/download/nginx-1.2.6.tar.gz
tar xzvf nginx-1.2.6.tar.gz

We configure and compile nginx with nginx-push-stream-module

cd nginx-1.2.6
./configure --add-module=../nginx-push-stream-module
make
make install

If there are no compilation errors, you're done. We’ll verify that we installed exactly that nginx and that now it really has a nginx-push-stream-module

check: /usr/local/nginx/sbin/nginx -v
test configuration: /usr/local/nginx/sbin/nginx -c $NGINX_PUSH_STREAM_MODULE_PATH/misc/nginx.conf -t

After executing these commands, you should see this:

nginx version: nginx/1.2.6
the configuration file $NGINX_PUSH_STREAM_MODULE_PATH/misc/nginx.conf syntax is ok
configuration file $NGINX_PUSH_STREAM_MODULE_PATH/misc/nginx.conf test is successful

Configuring Nginx for Long Polling Connections

To configure long polling support, in the Nginx configuration, you need to register at least two controllers. The first for subscribers (those who will receive messages), the second for the publication of messages (those who will send messages).

Omitting the configuration of the remaining server parameters, the configuration file /usr/local/nginx/nginx.conf should look like this:

...
http {
    ...
    server {
        listen 80;
        server_name stream.example.com;
        charset utf-8;
        location /pub {
            push_stream_publisher admin;
            set $push_stream_channel_id             $arg_id;
            allow  1.1.1.1  # ip адрес сервера посылающего событие
        }
        location ~ /sub/(.*) {
            push_stream_subscriber                  long-polling;
            set $push_stream_channels_path          $1;
            push_stream_last_received_message_tag   $arg_tag;
            push_stream_last_received_message_time  $arg_time;
            push_stream_longpolling_connection_ttl  25s;
        }
    }
}

In this example, / pub is the address for posting messages, only your server (1.1.1.1) from which events are arriving should see it, / sub is the address for subscribers who will be sent messages. The identifier that will identify the subscribers is passed after / sub, and is accepted as the id parameter in / pub.

The very important parameters push_stream_last_received_message_tag and push_stream_last_received_message_time will be discussed below when we touch on javascript.

Example for understanding the work:
You can create several subscribers by calling: stream.example.com/sub/1 , stream.example.com/sub/2 , stream.example.com/sub/3. Each of them will “hang” on the Nginx server for 25 seconds (push_stream_longpolling_connection_ttl). If we call the POST request on stream.example.com/pub?id=2 and send the message “Hello” in the body, the subscriber “hanging” on / sub / 2 will receive a response “Hello”. It is convenient to check this in the Poster plugin for FireFox.

Creating Subscribers in Javascript

Most likely, you need to use long polling to update any data in the browser, and for this you need to write a Javascript client.

I tried different methods, but chose XMLHttpRequest as the standard . Compared with other methods, it has the following advantages:

• Works great in all Chrome, Firefox, Opera, IE 8, 9, 10 browsers
• In browsers, the page load icon does not hang
• Works on different domains (cross-domain, if the server has CORS support )

Let the variable subID - a unique value is stored for the subscriber

var LongPolling = {
  etag: 0,
  time: null,
  init: function () {
    var $this = this, xhr;
    if ($this.time === null) {
      $this.time = $this.dateToUTCString(new Date());
    }
    if (window.XDomainRequest) {
      // Если IE, запускаем работу чуть позже (из-за бага IE8)
      setTimeout(function () {
        $this.poll_IE($this);
      }, 2000);
    } else {
      // Создает XMLHttpRequest объект
      mcXHR = xhr = new XMLHttpRequest(); 
      xhr.onreadystatechange = xhr.onload = function () {
        if (4 === xhr.readyState) {
          // Если пришло сообщение
          if (200 === xhr.status && xhr.responseText.length > 0) {
            // Берем Etag и Last-Modified из Header ответа
            $this.etag = xhr.getResponseHeader('Etag');
            $this.time = xhr.getResponseHeader('Last-Modified');
            // Вызываем обработчик сообщения
            $this.action(xhr.responseText);
          }
          if (xhr.status > 0) {
            // Если ничего не пришло повторяем операцию
            $this.poll($this, xhr);
          }
        }
      };
      // Начинаем long polling
      $this.poll($this, xhr);
    }
  },
  poll: function ($this, xhr) {
    var timestamp = (new Date()).getTime(),
      url = 'http://stream.example.com/sub/' + subID + '?callback=?&v=' + timestamp;
      // timestamp помогает защитить от кеширования в браузерах
    xhr.open('GET', url, true);
    xhr.setRequestHeader("If-None-Match", $this.etag);
    xhr.setRequestHeader("If-Modified-Since", $this.time);
    xhr.send();
  },
  // То же самое что и poll(), только для IE
  poll_IE: function ($this) {
    var xhr = new window.XDomainRequest();
    var timestamp = (new Date()).getTime(),
      url = 'http://stream.example.com/sub/' + subID + '?callback=?&v=' + timestamp;
    xhr.onprogress = function () {};
    xhr.onload = function () {
      $this.action(xhr.responseText);
      $this.poll_IE($this);
    };
    xhr.onerror = function () {
      $this.poll_IE($this);
    };
    xhr.open('GET', url, true);
    xhr.send();
  },
  action: function (event) {
    // получили сообщение, и теперь можем что-то обновить
    ...
  },
  valueToTwoDigits: function (value) {
    return ((value < 10) ? '0' : '') + value;
  },
  // представление даты в виде UTC
  dateToUTCString: function () {
    var time = this.valueToTwoDigits(date.getUTCHours())
        + ':' + this.valueToTwoDigits(date.getUTCMinutes())
        + ':' + this.valueToTwoDigits(date.getUTCSeconds());
    return this.days[date.getUTCDay()] + ', '
           + this.valueToTwoDigits(date.getUTCDate()) + ' '
           + this.months[date.getUTCMonth()] + ' '
           + date.getUTCFullYear() + ' ' + time + ' GMT';
  }
}

It is important to mention the two parameters etag and time.

xhr.setRequestHeader("If-None-Match", $this.etag);
xhr.setRequestHeader("If-Modified-Since", $this.time);

Without them, long polling did not always work, and messages arrived every other time. These two parameters are needed by the nginx-push-stream-module to identify messages that the subscriber has not yet received. So for stable operation it is simply necessary.

In custody

The method described in this topic is used and successfully works in our Cackle commenting system . Every day we have about 20,000 - 30,000 concurrent subscribers, and we have never seen any errors in message delivery. For production solutions, this is exactly what you need.