Load test with Go

Good afternoon, Habrahabr.
You are probably familiar with JMeter . If in short - a very convenient tool for conducting load testing, it has huge functionality and many, many useful features. But the article is not about him.

Where did it start

There is a rather loaded node in our project, JMeter helped for a long time. Profiling and optimization gave their profit, but everything ran into a small problem. JMeter could not create very large traffic, and more precisely, after 10 seconds of the required mode, OutOfMemory took place and testing stopped, in some cases there was no problem, but the speed of sending requests decreased noticeably, while the CPU load was 400%, it was solved restarting the program. It was extremely uncomfortable to use.
So, we have a problem, and it needs to be solved, the first thing that came to mind was to make your own mini-test that meets the minimum requirements. It has long been interesting to try Go's taste. So the go-meter application was born. When writing, there were a lot of questions, the answers to which either were not, or they did not explain the problem, so I decided to share my experience and an example of working code, if you are interested, I ask a tackle.

Foreword

I think writing about what kind of language does not make sense, you can always see a tour of the language that reveals the basic elements. How to install and configure the environment is also not worth it; the documentation is written in a completely understandable language.
Why did you choose Go? There are several criteria that are very important for me: it works fast, cross-platform, there are flows that are easy to manage, unusual. Of course, you say that you can write this in any other language. I agree with you, but the task was not only to write, but also to learn something new.

Let's get started

Without hesitation, it was decided to store the test profile in JSON format, after launching the application, the profile is read and testing is started. During testing, a summary table is displayed in the console (response time, number of requests per second and percentage of errors, warnings and successful requests). With JSON, everything is simple, for this you need to make structures for each element, open and read the file:

func (this *Settings) Load(fileName string) error {
	file, e := ioutil.ReadFile(fileName); if e != nil {
		return e
	}
	e = json.Unmarshal(file, this); if e != nil {
		return e
	}
	return nil
}

Let's go further. After starting, we need to start N-threads, and after working out each of them, aggregate the data, then output it beautifully to the console. For this, in this interesting language there are Channels. A kind of "pipe" between different streams. No synchronization, locks are needed, everything is done for us. The idea is this: a thread sends a request, determines the result, and reports this to the main thread, which in turn waits until all the threads have completed and output all the data received. Streams will communicate with us by means of transmission of the structure:

type Status struct {
	IsError bool
	IsWarning bool
	IsSuccess bool
	Duration *time.Duration
	Size int64
	IsFinished bool
	Error *error
	FinishedAt *time.Time
	StartedAt *time.Time
}

Each thread will execute an M-time HTTP request to the specified resource. If we have a POST request, then still sending certain data that the user wants:

func StartThread(setts *settings.Settings, source *Source, c chan *Status){
	iteration := setts.Threads.Iteration
	//Формируем объект key, value для заголовков запроса
	header := map[string]string{}
	for _, s := range setts.Request.Headers {
		keyValue := regexp.MustCompile("=").Split(s, -1)
		header[keyValue[0]] = keyValue[1]
	}
	sourceLen := len(*source)
	//необходимый URL
	url := setts.Remote.Protocol + "://" + setts.Remote.Host + ":" + strconv.Itoa(setts.Remote.Port) + setts.Request.Uri
	if iteration < 0 {
		iteration = sourceLen
	}
	index := -1
	for ;iteration > 0; iteration-- {
		status := &Status{false, false, false, nil, 0, false, nil, nil, nil}
		index++
		if index >= sourceLen {
			if setts.Request.Source.RestartOnEOF {
				index = 0
			} else {
				index--
			}
		}
		//Получаем данные для отправки запроса
		var s *bytes.Buffer
		if strings.ToLower(setts.Request.Method) != "get" {
			s = bytes.NewBuffer((*source)[index])
		}
		//Создаем HTTP запрос
		req, err := http.NewRequest(setts.Request.Method, url, s); if err != nil {
			status.Error = &err
			status.IsError = true
			c <- status
			break
		}
		//Выставляем заголовки
		for k,v := range header {
			req.Header.Set(k,v)
		}
		//Засекаем время
		startTime := time.Now()
		//Отправляем запрос
		res, err := http.DefaultClient.Do(req); if err != nil {
			status.Error = &err
			status.IsError = true
			c <- status
			break
		}
		endTime := time.Now()
		//Записываем служебную информацию
		status.FinishedAt = &endTime
		status.StartedAt = &startTime
		diff := endTime.Sub(startTime)
		//Проверяем статус ответа и причисляем в одной из 3 групп (Error, Warning, Success)
		checkStatus(setts.Levels, res, diff, status)
		//Закрываем соединение
		ioutil.ReadAll(res.Body)
		res.Body.Close()
		//Оповещаем главный поток
		c <- status
		//Если установлена в настройках задержка, выполняем ее
		if setts.Threads.Delay > 0 {
			sleep := time.Duration(setts.Threads.Delay)
			time.Sleep(time.Millisecond * sleep)
		}
	}
	//Оповещаем главный поток о завершении работы
	status := &Status{false, false, false, nil, 0, true, nil, nil, nil}
	c <- status
}

It remains only to start our threads at program startup and listen to data from them

c := make(chan *Status, iteration * setts.Threads.Count)
for i := 0; i < setts.Threads.Count; i++{
	go StartThread(&setts, source, c)
}
for i := iteration * setts.Threads.Count; i>0 ; i-- {
	counter(<-c)
}
fmt.Println("Completed")

Instead of a conclusion

These are the most interesting moments, in my opinion. All sources are available on GitHub , there you can see the entire cycle of work with an example of use. In fact, this miracle language coped with this task with interest, when generating traffic 3 times more than it was in the case of JMeter, the processor load rarely exceeds 15%.
If it’s interesting, I’ll talk about the process of writing an HTTP Restfull Web service with storage in MongoDB and Redis.

Thanks for attention!