Statistics collection MTProto Proxy

Published on July 03, 2018

Statistics collection MTProto Proxy

    Content
    • Prehistory
    • Сбор статистики
    • Отображение статистики
    • Визуализация и ведение статистики
    • Развертка
    • Заключение


    Prehistory


    Hi habr, telegrams are now at the peak of popularity, all the scandals, intrigues, locks revolve around him, in connection with which telegrams rolled out his proxy version called MTProto Proxy which is designed to help with the bypass of blocking. However, the telegram-provided services for monitoring MTProto Proxy do not provide an opportunity to observe the statistics in real time and collect it to monitor its changes, so we will solve the problem on our own.

    Statistics collection


    The official MTProto Proxy page on the Docker Hub indicates that we can use the command docker exec mtproto-proxy curl http://localhost:2398/statsto get statistics directly from the MTProto Proxy which is in the container, so our code will look like this.

    package main
    import (
    	"io/ioutil"
    	"net/http"
    	"strings"
    	"time"
    )
    type User struct {
    	Num string
    }
    var Users User
    func CurrenUsers() (err error) {
    	// Тянем статистику
    	response, err := http.Get(`http://localhost:2398/stats`)
    	if err != nil {
    		return
    	}
    	body, err := ioutil.ReadAll(response.Body)
    	if err != nil {
    		return
    	}
    	defer response.Body.Close()
    	stat := strings.Split(string(body), "\n")
    	for _, item := range stat {
    		// Проверяем что у нас есть нужное поле
    		// которое содержит количество пользователей
    		if strings.HasPrefix(item, `total_special_connections`) {
    			Users.Num = strings.Split(item, "\t")[1]
    		}
    	}
    	return nil
    }
    func main() {
    	for t := time.Tick(10 * time.Second); ; <-t {
    		if err := CurrenUsers(); err != nil {
    			continue
    		}
    	}
    }
    

    total_special_connectionsis indicated on the same Docker Hub as the number of incoming client connections

    Statistics display


    Next, we need to display the current number of users in a simple and convenient form, we will display it in a browser.

    package main
    import (
    	"html/template"
    	"io/ioutil"
    	"net/http"
    	"strings"
    	"time"
    )
    type User struct {
    	Num string
    }
    type HTML struct {
    	IndexPage string
    }
    var Users User
    var IndexTemplate = HTML{
    	IndexPage: `<!DOCTYPE html>
        <html>
        <head>
            <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css" integrity="sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4"
                crossorigin="anonymous">
            <title>Stats</title>
            <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
        </head>
        <body>
            <div class="container-fluid">
                <div class="row justify-content-center text-center" style="margin-top: 20%">
                    <h1>Count of current users of MTProto Proxy: {{.Num}}</h1>
                </div>
            </div>
        </body>
        </html>`,
    }
    func CurrenUsers() (err error) {
    	// Тянем статистику
    	response, err := http.Get(`http://localhost:2398/stats`)
    	if err != nil {
    		return
    	}
    	body, err := ioutil.ReadAll(response.Body)
    	if err != nil {
    		return
    	}
    	defer response.Body.Close()
    	stat := strings.Split(string(body), "\n")
    	for _, item := range stat {
    		// Проверяем что у нас есть нужное поле
    		// которое содержит количество пользователей
    		if strings.HasPrefix(item, `total_special_connections`) {
    			Users.Num = strings.Split(item, "\t")[1]
    		}
    	}
    	return nil
    }
    func sendStat(w http.ResponseWriter, r *http.Request) {
    	if r.Method == "GET" {
    		t := template.Must(template.New("indexpage").Parse(IndexTemplate.IndexPage))
    		t.Execute(w, Users)
    	}
    }
    func init() {
    	go func() {
    		for t := time.Tick(10 * time.Second); ; <-t {
    			if err := CurrenUsers(); err != nil {
    				continue
    			}
    		}
    	}()
    }
    func main() {
    	http.HandleFunc("/", sendStat)
    	http.ListenAndServe(":80", nil)
    }
    

    what is init
    init в любом случае будет вызван перед вызовом main

    Now, by going to the IP address of our MTProto Proxy, we can see the current number of clients.

    image

    Visualization and statistics


    There are many options for visualizing and maintaining statistics Datadog , Zabbix , Grafana , Graphite . I will use Datadog. Using the command, we go get -u github.com/DataDog/datadog-go/statsdimport the library statsd and use it in the code.

    package main
    import (
    	"html/template"
    	"io/ioutil"
    	"net/http"
    	"os"
    	"strconv"
    	"strings"
    	"time"
    	"github.com/DataDog/datadog-go/statsd"
    )
    var (
    	datadogIP = os.Getenv("DDGIP")
    	tagName   = os.Getenv("TGN")
    	t, _      = strconv.Atoi(os.Getenv("TIMEOUT"))
    	timeout   = time.Duration(t) * time.Second
    )
    type User struct {
    	Num string
    }
    type HTML struct {
    	IndexPage string
    }
    var Users User
    var IndexTemplate = HTML{
    	IndexPage: `<!DOCTYPE html>
        <html>
        <head>
            <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css" integrity="sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4"
                crossorigin="anonymous">
            <title>Stats</title>
            <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
        </head>
        <body>
            <div class="container-fluid">
                <div class="row justify-content-center text-center" style="margin-top: 20%">
                    <h1>Count of current users of MTProto Proxy: {{.Num}}</h1>
                </div>
            </div>
        </body>
        </html>`,
    }
    func (u User) convert() int64 {
    	num, _ := strconv.Atoi(u.Num)
    	return int64(num)
    }
    func CurrenUsers() (err error) {
    	// Тянем статистику
    	response, err := http.Get(`http://localhost:2398/stats`)
    	if err != nil {
    		return
    	}
    	body, err := ioutil.ReadAll(response.Body)
    	if err != nil {
    		return
    	}
    	defer response.Body.Close()
    	stat := strings.Split(string(body), "\n")
    	for _, item := range stat {
    		// Проверяем что у нас есть нужное поле
    		// которое содержит количество пользователей
    		if strings.HasPrefix(item, `total_special_connections`) {
    			Users.Num = strings.Split(item, "\t")[1]
    		}
    	}
    	return nil
    }
    func sendStat(w http.ResponseWriter, r *http.Request) {
    	if r.Method == "GET" {
    		t := template.Must(template.New("indexpage").Parse(IndexTemplate.IndexPage))
    		t.Execute(w, Users)
    	}
    }
    func init() {
    	if t == 0 {
    		timeout = 10 * time.Second
    	}
    	go func() {
    		for t := time.Tick(timeout); ; <-t {
    			if err := CurrenUsers(); err != nil {
    				continue
    			}
    		}
    	}()
    	// Отправляем статистику агенту Datadog
    	go func() error {
    		c, err := statsd.New(datadogIP + ":8125")
    		if err != nil || len(datadogIP) == 0 {
    			return err
    		}
    		c.Namespace = "mtproto."
    		c.Tags = append(c.Tags, tagName)
    		for t := time.Tick(timeout); ; <-t {
    			c.Count("users.count", Users.convert(), nil, 1)
    		}
    	}()
    }
    func main() {
    	http.HandleFunc("/", sendStat)
    	http.ListenAndServe(":80", nil)
    }
    


    It remains to collect all the docker image

    FROM telegrammessenger/proxy
    COPY mtproto_proxy_stat .
    RUN echo "$(tail -n +2 run.sh)" > run.sh && echo '#!/bin/bash\n./mtproto_proxy_stat & disown' | cat - run.sh > temp && mv temp run.sh
    CMD [ "/bin/sh", "-c", "/bin/bash /run.sh"] 

    Scan


    First we need to run the container with the Datadog agent.

    docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true -e DD_API_KEY=ВАШ_КЛЮЧ datadog/agent:latest

    IMPORTANT in order for us to send our data to the agent, we need to set the value true for the environment variable. DD_DOGSTATSD_NON_LOCAL_TRAFFIC

    Next, with the help of the command, docker inspect dd-agentwe need to look at the IP container to send it data

    image

    and run our MTProto Proxy by connecting it with the agent container

    docker run -d -p 443:443 -p 80:80 -e WORKERS=16 -e DDGIP=172.17.0.2 -e TGN=mtproto:main --link=dd-agent --name=mtproto --restart=always -v proxy-config:/data trigun117/mtproto_proxy_stat

    And after a couple of minutes, we can already build a graph by selecting the desired metric and source (the tag that is specified when launching the container with MTProto Proxy)

    image

    and display on it our Live statistics

    image

    .

    Conclusion


    For myself, I discovered new tools for convenient work with data, I became acquainted with their great diversity and chose something suitable for your taste.

    Thank you for your attention, I invite everyone to share their opinions, comments and suggestions in the comments.

    Github repository