Statistics collection MTProto Proxy

Content

Prehistory
Сбор статистики
Отображение статистики
Визуализация и ведение статистики
Развертка
Заключение

Prehistory

Hi habr, telegrams are now at the peak of popularity, all the scandals, intrigues, locks revolve around him, in connection with which telegrams rolled out his proxy version called MTProto Proxy which is designed to help with the bypass of blocking. However, the telegram-provided services for monitoring MTProto Proxy do not provide an opportunity to observe the statistics in real time and collect it to monitor its changes, so we will solve the problem on our own.

Statistics collection

The official MTProto Proxy page on the Docker Hub indicates that we can use the command docker exec mtproto-proxy curl http://localhost:2398/statsto get statistics directly from the MTProto Proxy which is in the container, so our code will look like this.

package main
import (
	"io/ioutil""net/http""strings""time"
)
type User struct {
	Num string
}
var Users User
funcCurrenUsers()(err error) {
	// Тянем статистику
	response, err := http.Get(`http://localhost:2398/stats`)
	if err != nil {
		return
	}
	body, err := ioutil.ReadAll(response.Body)
	if err != nil {
		return
	}
	defer response.Body.Close()
	stat := strings.Split(string(body), "\n")
	for _, item := range stat {
		// Проверяем что у нас есть нужное поле// которое содержит количество пользователейif strings.HasPrefix(item, `total_special_connections`) {
			Users.Num = strings.Split(item, "\t")[1]
		}
	}
	returnnil
}
funcmain() {
	for t := time.Tick(10 * time.Second); ; <-t {
		if err := CurrenUsers(); err != nil {
			continue
		}
	}
}

total_special_connectionsis indicated on the same Docker Hub as the number of incoming client connections

Statistics display

Next, we need to display the current number of users in a simple and convenient form, we will display it in a browser.

package main
import (
	"html/template""io/ioutil""net/http""strings""time"
)
type User struct {
	Num string
}
type HTML struct {
	IndexPage string
}
var Users User
var IndexTemplate = HTML{
	IndexPage: `<!DOCTYPE html>
    <html>
    <head>
        <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css" integrity="sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4"
            crossorigin="anonymous">
        <title>Stats</title>
        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    </head>
    <body>
        <div class="container-fluid">
            <div class="row justify-content-center text-center" style="margin-top: 20%">
                <h1>Count of current users of MTProto Proxy: {{.Num}}</h1>
            </div>
        </div>
    </body>
    </html>`,
}
funcCurrenUsers()(err error) {
	// Тянем статистику
	response, err := http.Get(`http://localhost:2398/stats`)
	if err != nil {
		return
	}
	body, err := ioutil.ReadAll(response.Body)
	if err != nil {
		return
	}
	defer response.Body.Close()
	stat := strings.Split(string(body), "\n")
	for _, item := range stat {
		// Проверяем что у нас есть нужное поле// которое содержит количество пользователейif strings.HasPrefix(item, `total_special_connections`) {
			Users.Num = strings.Split(item, "\t")[1]
		}
	}
	returnnil
}
funcsendStat(w http.ResponseWriter, r *http.Request) {
	if r.Method == "GET" {
		t := template.Must(template.New("indexpage").Parse(IndexTemplate.IndexPage))
		t.Execute(w, Users)
	}
}
funcinit() {
	gofunc() {
		for t := time.Tick(10 * time.Second); ; <-t {
			if err := CurrenUsers(); err != nil {
				continue
			}
		}
	}()
}
funcmain() {
	http.HandleFunc("/", sendStat)
	http.ListenAndServe(":80", nil)
}

what is init

init в любом случае будет вызван перед вызовом main

Now, by going to the IP address of our MTProto Proxy, we can see the current number of clients.

Visualization and statistics

There are many options for visualizing and maintaining statistics Datadog , Zabbix , Grafana , Graphite . I will use Datadog. Using the command, we go get -u github.com/DataDog/datadog-go/statsdimport the library statsd and use it in the code.

package main
import (
	"html/template""io/ioutil""net/http""os""strconv""strings""time""github.com/DataDog/datadog-go/statsd"
)
var (
	datadogIP = os.Getenv("DDGIP")
	tagName   = os.Getenv("TGN")
	t, _      = strconv.Atoi(os.Getenv("TIMEOUT"))
	timeout   = time.Duration(t) * time.Second
)
type User struct {
	Num string
}
type HTML struct {
	IndexPage string
}
var Users User
var IndexTemplate = HTML{
	IndexPage: `<!DOCTYPE html>
    <html>
    <head>
        <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css" integrity="sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4"
            crossorigin="anonymous">
        <title>Stats</title>
        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    </head>
    <body>
        <div class="container-fluid">
            <div class="row justify-content-center text-center" style="margin-top: 20%">
                <h1>Count of current users of MTProto Proxy: {{.Num}}</h1>
            </div>
        </div>
    </body>
    </html>`,
}
func(u User)convert()int64 {
	num, _ := strconv.Atoi(u.Num)
	returnint64(num)
}
funcCurrenUsers()(err error) {
	// Тянем статистику
	response, err := http.Get(`http://localhost:2398/stats`)
	if err != nil {
		return
	}
	body, err := ioutil.ReadAll(response.Body)
	if err != nil {
		return
	}
	defer response.Body.Close()
	stat := strings.Split(string(body), "\n")
	for _, item := range stat {
		// Проверяем что у нас есть нужное поле// которое содержит количество пользователейif strings.HasPrefix(item, `total_special_connections`) {
			Users.Num = strings.Split(item, "\t")[1]
		}
	}
	returnnil
}
funcsendStat(w http.ResponseWriter, r *http.Request) {
	if r.Method == "GET" {
		t := template.Must(template.New("indexpage").Parse(IndexTemplate.IndexPage))
		t.Execute(w, Users)
	}
}
funcinit() {
	if t == 0 {
		timeout = 10 * time.Second
	}
	gofunc() {
		for t := time.Tick(timeout); ; <-t {
			if err := CurrenUsers(); err != nil {
				continue
			}
		}
	}()
	// Отправляем статистику агенту Datadoggofunc()error {
		c, err := statsd.New(datadogIP + ":8125")
		if err != nil || len(datadogIP) == 0 {
			return err
		}
		c.Namespace = "mtproto."
		c.Tags = append(c.Tags, tagName)
		for t := time.Tick(timeout); ; <-t {
			c.Count("users.count", Users.convert(), nil, 1)
		}
	}()
}
funcmain() {
	http.HandleFunc("/", sendStat)
	http.ListenAndServe(":80", nil)
}

It remains to collect all the docker image

FROM telegrammessenger/proxy
COPY mtproto_proxy_stat .
RUN echo "$(tail -n +2 run.sh)" > run.sh && echo '#!/bin/bash\n./mtproto_proxy_stat & disown' | cat - run.sh > temp && mv temp run.sh
CMD [ "/bin/sh", "-c", "/bin/bash /run.sh"]

Scan

First we need to run the container with the Datadog agent.

docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true -e DD_API_KEY=ВАШ_КЛЮЧ datadog/agent:latest

IMPORTANT in order for us to send our data to the agent, we need to set the value true for the environment variable. DD_DOGSTATSD_NON_LOCAL_TRAFFIC

Next, with the help of the command, docker inspect dd-agentwe need to look at the IP container to send it data

and run our MTProto Proxy by connecting it with the agent container

docker run -d -p 443:443 -p 80:80 -e WORKERS=16 -e DDGIP=172.17.0.2 -e TGN=mtproto:main --link=dd-agent --name=mtproto --restart=always -v proxy-config:/data trigun117/mtproto_proxy_stat

And after a couple of minutes, we can already build a graph by selecting the desired metric and source (the tag that is specified when launching the container with MTProto Proxy)

and display on it our Live statistics

Conclusion

For myself, I discovered new tools for convenient work with data, I became acquainted with their great diversity and chose something suitable for your taste.

Thank you for your attention, I invite everyone to share their opinions, comments and suggestions in the comments.

Github repository

Tags: