We write an educational application on Go and Javascript to assess the real return on equity. Part 2 - Backend Testing
- Tutorial
In the first part of the article, we wrote a small web server, which is a backend for our information system. That part was not particularly interesting, although it demonstrated the use of the interface and one of the techniques for working with gorutines. Both can be interesting for novice developers.
The second part is much more interesting and useful, because in it we will write unit tests both for the server itself and for the library package that implements the data storage. Let's get started
picture from here
So, let me remind you that our application consists of an executable module (web server, API), a storage module (entity data structures, an interface-contract for storage providers) and storage providers modules (in our example, only one module that performs the interface for storing data in memory).
We will test the executable module and the storage implementation. A module with a contract does not contain any code that can be tested. There are only type declarations.
For testing, we will use only the capabilities of the standard library - the testing and httptest packages. In my opinion, they are quite enough, although there are many different test frameworks. Look at them, maybe you will like them. From my point of view, programs on Go do not really need those frameworks (of various kinds) that now exist. This is not Javascript, which will be discussed in the third part of the article.
First a few words about the testing methodology that I use for Go programs.
First of all, I must say that I really like Go, just because it does not drive the programmer into some kind of rigid framework. Although some developers, in fairness, love to drive themselves into the framework, brought from the previous PL. For example, the same Rob Pike, said that he does not see a problem in copying code, if that is easier. Such copy-paste is even in the standard library. Instead of importing the package, one of the authors of the language simply copied the text of one function (unicode check). In the test, the unicode package is imported, so everything is OK.
By the way, in this sense (in the sense of language flexibility), an interesting technique can be used when writing tests. The point is this: we know that the interface contracts in Go are executed implicitly. That is, we can declare a type, write methods for it, and execute some kind of contract. Perhaps even without knowing it. This is well known and understandable. However, it works in the opposite direction. If the author of some module did not write an interface that would help us to make a stub for testing our package, then we can declare the interface ourselves in our test, which will be executed in a third-party package. Fruitful idea, although in our educational application and not useful.
Secondly , a few words about the time of writing tests. As everyone knows, there are different opinions about when to write unit tests. The main ideas are as follows:
I do not think that there is a single correct opinion on this matter. I will share my own and ask readers to comment in the comments. My opinion is:
Based on this, in our training application, the in-memory storage provider was written through tests. The server's executable was created through a prototype.
Let's start with the tests for the implementation of the repository.
In the repository, we have a New () factory method that returns a pointer to an instance of the repository type. There are also methods for obtaining quotes of Securities (), adding paper to the Add () list and initializing the data storage from the Mosbirzh server InitData ().
We test the constructor (OOP terms are used voluntarily, informally. In full accordance with the OOP position in Go).
This test, without much need, demonstrated the only way in Go to check the type of a variable - reflection (reflect.TypeOf (memoryStorage)). Abuse of this module is not recommended. Challenges are heavy, and generally not worth it. On the other hand, what else to check in this test except the absence of an error?
Next, we will test getting quotes and adding paper. These tests partially overlap, but this is not critical (in the paper addition test, the method for obtaining quotes for verification is called). In general, I sometimes write one test for all CRUD operations for a particular entity. That is, in the test, I create an entity, read it, change it, read it again, delete it, read it again. Not very elegant, but obvious flaws are not visible.
Test getting quotes.
It's all pretty obvious.
Now test to add paper. In this test, for educational purposes (without real need), we will use a very convenient method of tabular testing (table tests). Its essence is as follows: we create an array of unnamed structures, each of which contains the input data for the test and the expected result. In our case, at the entrance, we submit a security paper to add, the result is the number of papers in the vault (the length of the array). Next, for each element of the array of structures we perform a test (we call the test method with the input data of the element) and compare the result with the result field of the current element. It turns out like this.
Well, the test for the data initialization function.
As a result of successful execution of tests, we get: 17.595s coverage: 86.0% of statements.
You can say that it would be nice for a separate library to get 100% coverage, but specifically here unsuccessful execution paths (errors in functions) are impossible at all, because of the implementation features - everything is in memory, not connected anywhere, or dependent on anything. There is a formal error handling, since returning an error causes the interface contract and requires a linter.
Let us turn to testing the executable package - web server. Here I must say that since the web server is a super-standard construction in Go programs, then the net / http / httptest package was specially developed for testing the handlers for http requests. It allows you to simulate a web server, run the request handler, and record the response of the web server in a special structure. That is what we will use, it is very simple, surely you will like it.
At the same time there is an opinion (and not only mine) that such a test may not be very relevant to a real working system. You can, in principle, start a real server and call real connection handlers in tests.
The truth is, there is another opinion (and also not only mine) that isolating business logic from real data manipulation systems is good.
In this sense, we can say that we are writing exactly unit tests, and not integration tests involving other packages and services. Although here I am also of the opinion that a certain flexibility of Go allows you not to lock yourself on terms and write the tests that best suit your tasks. I will give my example: for tests of API request handlers, I made a simplified copy of the database on a real server on the network, initialized it with a small set of data and ran tests on real data. But this approach is highly situational.
Let's return to the tests of our web server. In order to write tests that are independent of the actual storage, we need to develop a stub storage. This is not difficult as we work with the storage through the interface (see the first part). All we need is to declare some kind of data type and implement for it the methods of the storage interface contract, even with empty data. Like this:
Now we can initialize our repository with a stub. How to do it? For the purpose of initializing the test environment, a function was added to Go of some not very old version:
This feature allows you to perform initialization and run all tests. It looks something like this:
Now we can write tests for API request handlers. We have two API endpoints, two handlers, and therefore two tests. They are very similar, so we give here the first one.
The essence of the test is this: create an http request, define the structure for recording the server response, start the request handler, decode the response body (json into the structure). Well, for clarity, we print the answer.
It turns out something like:
In the next, final part of the article, we will develop a web application for displaying graphs of the real return on shares of ETF Mosbirzhi.
The second part is much more interesting and useful, because in it we will write unit tests both for the server itself and for the library package that implements the data storage. Let's get started
picture from here
So, let me remind you that our application consists of an executable module (web server, API), a storage module (entity data structures, an interface-contract for storage providers) and storage providers modules (in our example, only one module that performs the interface for storing data in memory).
We will test the executable module and the storage implementation. A module with a contract does not contain any code that can be tested. There are only type declarations.
For testing, we will use only the capabilities of the standard library - the testing and httptest packages. In my opinion, they are quite enough, although there are many different test frameworks. Look at them, maybe you will like them. From my point of view, programs on Go do not really need those frameworks (of various kinds) that now exist. This is not Javascript, which will be discussed in the third part of the article.
First a few words about the testing methodology that I use for Go programs.
First of all, I must say that I really like Go, just because it does not drive the programmer into some kind of rigid framework. Although some developers, in fairness, love to drive themselves into the framework, brought from the previous PL. For example, the same Rob Pike, said that he does not see a problem in copying code, if that is easier. Such copy-paste is even in the standard library. Instead of importing the package, one of the authors of the language simply copied the text of one function (unicode check). In the test, the unicode package is imported, so everything is OK.
By the way, in this sense (in the sense of language flexibility), an interesting technique can be used when writing tests. The point is this: we know that the interface contracts in Go are executed implicitly. That is, we can declare a type, write methods for it, and execute some kind of contract. Perhaps even without knowing it. This is well known and understandable. However, it works in the opposite direction. If the author of some module did not write an interface that would help us to make a stub for testing our package, then we can declare the interface ourselves in our test, which will be executed in a third-party package. Fruitful idea, although in our educational application and not useful.
Secondly , a few words about the time of writing tests. As everyone knows, there are different opinions about when to write unit tests. The main ideas are as follows:
- We write tests before writing code (TDD). Thus, we better understand the task and set the quality criteria.
- We write tests while writing code or even a little later (we will consider this as incremental prototyping).
- We write tests sometime later, if there is time. And it's not a joke. Sometimes the conditions are such that there is no physical time.
I do not think that there is a single correct opinion on this matter. I will share my own and ask readers to comment in the comments. My opinion is:
- Stand-alone packages are developed on TDD, this really makes things easier, especially when running an application for verification is a resource-intensive process. For example, I recently developed a GPS / GLONASS vehicle monitoring system. Packages for protocol drivers can only be developed through tests, since launching and manually testing an application requires waiting for data from trackers, which is extremely inconvenient. For tests, I took samples of data packets, wrote them down in table tests, and the server did not start until the drivers were ready.
- If the structure of the code is not clear, then at first I try to make a minimal working prototype. Then I write tests, or even first polish the code a bit and then just tests.
- For executables, I first write a prototype. Tests later. Obviously, I don’t test the executable code at all (of course, you can take the launch of the http server from main into a separate function and call it in the test, but why test the standard library?)
Based on this, in our training application, the in-memory storage provider was written through tests. The server's executable was created through a prototype.
Let's start with the tests for the implementation of the repository.
In the repository, we have a New () factory method that returns a pointer to an instance of the repository type. There are also methods for obtaining quotes of Securities (), adding paper to the Add () list and initializing the data storage from the Mosbirzh server InitData ().
We test the constructor (OOP terms are used voluntarily, informally. In full accordance with the OOP position in Go).
// тестируем конструктор хранилищаfuncTestNew(t *testing.T) {
// вызываем тестируемый метод-фабрику
memoryStorage := New()
// создаём переменную для сравненияvar s *Storage
// сравниваем тип результата вызова функции с типом в модуле. просто такif reflect.TypeOf(memoryStorage) != reflect.TypeOf(s) {
t.Errorf("тип неверен: получили %v, а хотели %v", reflect.TypeOf(memoryStorage), reflect.TypeOf(s))
}
// для наглядности выводим результат
t.Logf("\n%+v\n\n", memoryStorage)
}This test, without much need, demonstrated the only way in Go to check the type of a variable - reflection (reflect.TypeOf (memoryStorage)). Abuse of this module is not recommended. Challenges are heavy, and generally not worth it. On the other hand, what else to check in this test except the absence of an error?
Next, we will test getting quotes and adding paper. These tests partially overlap, but this is not critical (in the paper addition test, the method for obtaining quotes for verification is called). In general, I sometimes write one test for all CRUD operations for a particular entity. That is, in the test, I create an entity, read it, change it, read it again, delete it, read it again. Not very elegant, but obvious flaws are not visible.
Test getting quotes.
// проверяем отдачу котировокfuncTestSecurities(t *testing.T) {
// экземпляр хранилища в памятиvar s *Storage
// вызываем тестируемый метод
ss, err := s.Securities()
if err != nil {
t.Error(err)
}
// для наглядности выводим результат
t.Logf("\n%+v\n\n", ss)
}
}It's all pretty obvious.
Now test to add paper. In this test, for educational purposes (without real need), we will use a very convenient method of tabular testing (table tests). Its essence is as follows: we create an array of unnamed structures, each of which contains the input data for the test and the expected result. In our case, at the entrance, we submit a security paper to add, the result is the number of papers in the vault (the length of the array). Next, for each element of the array of structures we perform a test (we call the test method with the input data of the element) and compare the result with the result field of the current element. It turns out like this.
// проверяем добавление котировкиfuncTestAdd(t *testing.T) {
// экземпляр хранилища в памятиvar s *Storage
var security = storage.Security{
ID: "MSFT",
}
// Табличный тестvar tt = []struct {
s storage.Security // добавляемая бумага
length int// длина массива (среза)
}{
{
s: security,
length: 1,
},
{
s: security,
length: 2,
},
}
var ss []storage.Security
// tc - test case, tt - table testsfor _, tc := range tt {
// вызываем тестируемый метод
err := s.Add(security)
if err != nil {
t.Error(err)
}
ss, err = s.Securities()
if err != nil {
t.Error(err)
}
iflen(ss) != tc.length {
t.Errorf("невереная длина среза: получили %d, а хотели %d", len(ss), tc.length)
}
}
// для наглядности выводим результат
t.Logf("\n%+v\n\n", ss)
}Well, the test for the data initialization function.
// проверяем инициализацию данныхfuncTestInitData(t *testing.T) {
// экземпляр хранилища в памятиvar s *Storage
// вызываем тестируемый метод
err := s.InitData()
if err != nil {
t.Error(err)
}
ss, err := s.Securities()
if err != nil {
t.Error(err)
}
iflen(ss) < 1 {
t.Errorf("невереный результат: получили %d, а хотели '> 1'", len(ss))
}
// для наглядности выводим результат
t.Logf("\n%+v\n\n", ss[0])
}As a result of successful execution of tests, we get: 17.595s coverage: 86.0% of statements.
You can say that it would be nice for a separate library to get 100% coverage, but specifically here unsuccessful execution paths (errors in functions) are impossible at all, because of the implementation features - everything is in memory, not connected anywhere, or dependent on anything. There is a formal error handling, since returning an error causes the interface contract and requires a linter.
Let us turn to testing the executable package - web server. Here I must say that since the web server is a super-standard construction in Go programs, then the net / http / httptest package was specially developed for testing the handlers for http requests. It allows you to simulate a web server, run the request handler, and record the response of the web server in a special structure. That is what we will use, it is very simple, surely you will like it.
At the same time there is an opinion (and not only mine) that such a test may not be very relevant to a real working system. You can, in principle, start a real server and call real connection handlers in tests.
The truth is, there is another opinion (and also not only mine) that isolating business logic from real data manipulation systems is good.
In this sense, we can say that we are writing exactly unit tests, and not integration tests involving other packages and services. Although here I am also of the opinion that a certain flexibility of Go allows you not to lock yourself on terms and write the tests that best suit your tasks. I will give my example: for tests of API request handlers, I made a simplified copy of the database on a real server on the network, initialized it with a small set of data and ran tests on real data. But this approach is highly situational.
Let's return to the tests of our web server. In order to write tests that are independent of the actual storage, we need to develop a stub storage. This is not difficult as we work with the storage through the interface (see the first part). All we need is to declare some kind of data type and implement for it the methods of the storage interface contract, even with empty data. Like this:
// для целей тестирования бизнес-логики создаём заглушку хранилищаtype stub int// тип данных не имеет значенияvar securities []storage.Security // имитация хранилища данных// *******************************// Выполняем контракт на хранилище// InitData инициализирует фейковое хранилище фейковыми даннымиfunc(s *stub)InitData()(err error) {
// добавив в хранилище-заглушку одну записьvar security = storage.Security{
ID: "MSFT",
Name: "Microsoft",
IssueDate: 1514764800, // 01/01/2018
}
var quote = storage.Quote{
SecurityID: "MSFT",
Num: 0,
TimeStamp: 1514764800,
Price: 100,
}
security.Quotes = append(security.Quotes, quote)
securities = append(securities, security)
return err
}
// Securities возвращает список бумаг с котировкамиfunc(s *stub)Securities()(data []storage.Security, err error) {
return securities, err
}
// контракт выполнен// *****************Now we can initialize our repository with a stub. How to do it? For the purpose of initializing the test environment, a function was added to Go of some not very old version:
funcTestMain(m *testing.M)This feature allows you to perform initialization and run all tests. It looks something like this:
// подготавливаем тестовую среду - инициализируем данныеfuncTestMain(m *testing.M) {
// присваиваем указатель на экземпляр хранилища-заглушки глобальной переменной хранилища
db = new(stub)
// инициализируем данные (ничем)
db.InitData()
// выполняем все тесты пакета
os.Exit(m.Run())
}Now we can write tests for API request handlers. We have two API endpoints, two handlers, and therefore two tests. They are very similar, so we give here the first one.
// тестируем отдачу котировокfuncTestSecuritiesHandler(t *testing.T) {
// проверяем обработчик запроса котировок
req, err := http.NewRequest(http.MethodGet, "/api/v1/securities", nil)
if err != nil {
t.Fatal(err)
}
// ResponseRecorder записывает ответ сервера
rr := httptest.NewRecorder()
handler := http.HandlerFunc(securitiesHandler)
// вызываем обработчик и передаём ему запрос
handler.ServeHTTP(rr, req)
// проверяем HTTP-код ответаif rr.Code != http.StatusOK {
t.Errorf("код неверен: получили %v, а хотели %v", rr.Code, http.StatusOK)
}
// десериализуем (раскодируем) ответ сервера из формата json в структуру данныхvar ss []storage.Security
err = json.NewDecoder(rr.Body).Decode(&ss)
if err != nil {
t.Fatal(err)
}
// выведем данные на экран для наглядности
t.Logf("\n%+v\n\n", ss)
}The essence of the test is this: create an http request, define the structure for recording the server response, start the request handler, decode the response body (json into the structure). Well, for clarity, we print the answer.
It turns out something like:
=== RUN TestSecuritiesHandlerCode on Gitkhab .
0xc00005e3e0
- PASS: TestSecuritiesHandler (0.00s)
c: \ Users \ dtsp \ YandexDisk \ go \ src \ moex_etf \ server \ server_test.go: 96:
[{ID: MSFT Name: Microsoft IssueDate: 1514764800 Quotes: [{SecurityID: MSFT Num: 0 TimeStamp: 1514764800 Price: 100}]}]
PASS
ok moex_etf / server 0.307s
Success: Tests passed.
In the next, final part of the article, we will develop a web application for displaying graphs of the real return on shares of ETF Mosbirzhi.