February 20, 2015 at 01:16We write a simulator of slow connections on Go
- Tutorial
In this article I want to show how easy it is to do quite complex things in Go, and how powerful the interfaces are. It will be about simulating a slow connection - but, unlike the popular solutions in the form of rules for iptables, we implement this on the code side - so that it can be easily used, for example, in tests.
There will be nothing complicated here, and for the sake of clarity, I recorded ascii-animations (using the asciinema service ), but I hope it will be informative.
Interfaces are a special type in the Go type system that allows you to describe the behavior of an object. Any static type for which methods (behavior) are defined implicitly implements the interface that describes these methods. The most famous example is the interface from the standard io.Reader library:
Any structure for which you define the Read ([] byte) (int, error) method can be used as io.Reader.
A simple idea, which at first does not seem too valuable and powerful, takes on a completely different look when the interfaces are used by other libraries. To demonstrate this, the standard library and io.Reader are ideal candidates.
So, let's start with the simplest application of Reader - we will output the line to stdout. Of course, for this task it is better to use the functions from the fmt package, but we want to demonstrate the work of Reader. Therefore, we will create a variable of the type strings.Reader (which implements io.Reader) and, using the io.Copy () function - which also works with io.Reader, copy this to os.Stdout (which, in turn, implements io.Writer).
And now, using composition, we will create our own type SlowReader, which will read one character from the original Reader with a delay of, say, 100 milliseconds - thus, providing a speed of 10 bytes per second.
What is p [: 1], I hope, there is no need to explain it - just a new slice, consisting of 1 first character from the original slice.
All that remains for us is to use our strings.Reader as the original io.Reader, and pass the slow SlowReader to io.Copy ()! See how simple and cool at the same time.
(ascii-cast opens in a new window, js-scripts on the hub are forbidden to be embedded) You should already begin to suspect that this simple SlowReader can be used not only for display on the screen. You can also add a parameter like delay. Better yet, put SlowReader in a separate package so that it is easy to use in further examples. We combed the code a bit.
Create the test / habr / slow directory and transfer the code there:
Or, who is interested in watching ascii-castes, like this - we put it in a separate package: And add a delay parameter of type time.Duration: (It would be more correct, after putting the code into a separate package, to name the type Reader - so that it is slow.Reader, and slow.SlowReader, but the screencast is already written like that).
And now, with almost no effort, check out our SlowReader for slow reading from files. Having received a variable of the type * os.File, which stores the handle of the open file, but at the same time implements the io.Reader interface, we can work with the file in the same way as before with strings.Reader.
Or so:
But reading from a file is too easy. Let's look at an example a little more interesting - the JSON decoder from the standard library. Although the encoding / json package provides the json.Unmarshal () function for convenience , it also allows you to work with io.Reader using json.Decoder - you can deserialize stream data in json format with it.
We will take a simple json-encoded string and “read it slowly” with the help of our SlowReader - and json.Decoder will return the finished object only after all the bytes have arrived. To make this obvious, we will add the output of each character read to the screen to the function slow.SlowReader.Read ():
This is the same in the ascii-caste: If you have not yet come across the awareness of the possibilities that such a simple concept of interfaces gives us, then we go further - in fact, we come to the topic of the post - we use our SlowReader to slowly download the page from the Internet.
You should no longer be surprised that io.Reader is used everywhere in the standard library - for everything that can read anything from somewhere. Reading from the network is no exception - io.Reader is used at several levels, and is hidden under the hood of such a simple, single-line http.Get (url string) call.
First, write the standard code for the HTTP GET request and print the response to the console:
For those who have not had time to get acquainted with the net / http-library - a few explanations. http.Get () is a wrapper for the Get () method implemented for the http.Client type - but this wrapper uses a “suitable for most cases” already initialized variable called DefaultClient. Actually, Client further does all the dusty work, including reading from the network using an object of type Transport, which in turn uses a lower-level object of type net.Conn. At first, this may seem confusing, but, in fact, it is quite easy to learn by simply reading the library sources - that’s what, and the standard library in Go, unlike most other languages, is an exemplary code in which you can (and should) learn Go and take an example from it.
Earlier, I mentioned “io.Reader is used on several levels” and this is true - for example resp.Body is also io.Reader, but we are not interested in it because we are interested in simulating not a slowed down browser, but a slow connection - then you need to find io.Reader, which reads from the network. And this, looking ahead, is a variable of type net.Conn - which means that we need to redefine it for our custom http client. We can do this with embedding:
The most difficult thing here is to still understand a little deeper into the net and net / http packages from the standard library, and correctly create our http.Client using the slow io.Reader. But, as a result of nothing complicated - I hope the logic is visible on the screencast as I look into the code of the standard library.
As a result, the following client is obtained (for real code, it is better to put it in a separate function and comb a little, but for the proof-of-concept example it will do):
Well, now we glue it all together and look at the result: In the end, you can see that the HTTP headers are output to the console normally, and the text, in fact, the page is displayed with doubling each character - this is normal, since we output resp.Body using io.Copy () and at the same time, our slightly modified implementation of SlowReader.Read () displays each character too.
As stated at the beginning of the article, interfaces are an extremely powerful toolkit, and the very idea of separating types for properties and behavior is very correct. But really this power is manifested when the interfaces are really used for other purposes in different libraries. This allows you to combine very different functionality, and use someone else's code for things that the original author might not even have suspected. And it's not just about standard interfaces - within large projects, interfaces provide tremendous flexibility and modularity.
Since the idea of this post was impudently pulled from the Francesc Campoy twitter, there is only one link :)
twitter.com/francesc/status/563310996845244416
There will be nothing complicated here, and for the sake of clarity, I recorded ascii-animations (using the asciinema service ), but I hope it will be informative.
Interfaces
Interfaces are a special type in the Go type system that allows you to describe the behavior of an object. Any static type for which methods (behavior) are defined implicitly implements the interface that describes these methods. The most famous example is the interface from the standard io.Reader library:
// Reader is the interface that wraps the basic Read method.
// ...
type Reader interface {
Read(p []byte) (n int, err error)
}
Any structure for which you define the Read ([] byte) (int, error) method can be used as io.Reader.
A simple idea, which at first does not seem too valuable and powerful, takes on a completely different look when the interfaces are used by other libraries. To demonstrate this, the standard library and io.Reader are ideal candidates.
Console Output
So, let's start with the simplest application of Reader - we will output the line to stdout. Of course, for this task it is better to use the functions from the fmt package, but we want to demonstrate the work of Reader. Therefore, we will create a variable of the type strings.Reader (which implements io.Reader) and, using the io.Copy () function - which also works with io.Reader, copy this to os.Stdout (which, in turn, implements io.Writer).
package main
import (
"io"
"os"
"strings"
)
func main() {
r := strings.NewReader("Not very long line...")
io.Copy(os.Stdout, r)
}
And now, using composition, we will create our own type SlowReader, which will read one character from the original Reader with a delay of, say, 100 milliseconds - thus, providing a speed of 10 bytes per second.
// SlowReader reads 10 chars per second
type SlowReader struct {
r io.Reader
}
func (sr SlowReader) Read(p []byte) (int, error) {
time.Sleep(100 * time.Millisecond)
return sr.r.Read(p[:1])
}
What is p [: 1], I hope, there is no need to explain it - just a new slice, consisting of 1 first character from the original slice.
All that remains for us is to use our strings.Reader as the original io.Reader, and pass the slow SlowReader to io.Copy ()! See how simple and cool at the same time.
(ascii-cast opens in a new window, js-scripts on the hub are forbidden to be embedded) You should already begin to suspect that this simple SlowReader can be used not only for display on the screen. You can also add a parameter like delay. Better yet, put SlowReader in a separate package so that it is easy to use in further examples. We combed the code a bit.
Comb the code
Create the test / habr / slow directory and transfer the code there:
package slow
import (
"io"
"time"
)
type SlowReader struct {
delay time.Duration
r io.Reader
}
func (sr SlowReader) Read(p []byte) (int, error) {
time.Sleep(sr.delay)
return sr.r.Read(p[:1])
}
func NewReader(r io.Reader, bps int) io.Reader {
delay := time.Second / time.Duration(bps)
return SlowReader{
r: r,
delay: delay,
}
}
Or, who is interested in watching ascii-castes, like this - we put it in a separate package: And add a delay parameter of type time.Duration: (It would be more correct, after putting the code into a separate package, to name the type Reader - so that it is slow.Reader, and slow.SlowReader, but the screencast is already written like that).
Read from file
And now, with almost no effort, check out our SlowReader for slow reading from files. Having received a variable of the type * os.File, which stores the handle of the open file, but at the same time implements the io.Reader interface, we can work with the file in the same way as before with strings.Reader.
package main
import (
"io"
"os"
"test/habr/slow"
)
func main() {
file, err := os.Open("test.txt")
if err != nil {
panic(err)
}
defer file.Close() // close file on exit
r := slow.NewReader(file, 5) // 5 bps
io.Copy(os.Stdout, r)
}
Or so:
Decode JSON
But reading from a file is too easy. Let's look at an example a little more interesting - the JSON decoder from the standard library. Although the encoding / json package provides the json.Unmarshal () function for convenience , it also allows you to work with io.Reader using json.Decoder - you can deserialize stream data in json format with it.
We will take a simple json-encoded string and “read it slowly” with the help of our SlowReader - and json.Decoder will return the finished object only after all the bytes have arrived. To make this obvious, we will add the output of each character read to the screen to the function slow.SlowReader.Read ():
package main
import (
"encoding/json"
"fmt"
"strings"
"test/habr/slow"
)
func main() {
sr := strings.NewReader(`{"value": "some text", "id": 42}`) // encoded json
r := slow.NewReader(sr, 5)
dec := json.NewDecoder(r)
type Sample struct {
Value string `json:"value"`
ID int64 `json:"id"`
}
var sample Sample
err := dec.Decode(&sample)
if err != nil {
panic(err)
}
fmt.Println("Decoded JSON value:", sample)
}
This is the same in the ascii-caste: If you have not yet come across the awareness of the possibilities that such a simple concept of interfaces gives us, then we go further - in fact, we come to the topic of the post - we use our SlowReader to slowly download the page from the Internet.
Slow HTTP Client
You should no longer be surprised that io.Reader is used everywhere in the standard library - for everything that can read anything from somewhere. Reading from the network is no exception - io.Reader is used at several levels, and is hidden under the hood of such a simple, single-line http.Get (url string) call.
First, write the standard code for the HTTP GET request and print the response to the console:
package main
import (
"io"
"net/http"
"os"
)
func main() {
resp, err := http.Get("http://golang.org")
if err != nil {
panic(err)
}
defer resp.Body.Close()
io.Copy(os.Stdout, resp.Body)
}
For those who have not had time to get acquainted with the net / http-library - a few explanations. http.Get () is a wrapper for the Get () method implemented for the http.Client type - but this wrapper uses a “suitable for most cases” already initialized variable called DefaultClient. Actually, Client further does all the dusty work, including reading from the network using an object of type Transport, which in turn uses a lower-level object of type net.Conn. At first, this may seem confusing, but, in fact, it is quite easy to learn by simply reading the library sources - that’s what, and the standard library in Go, unlike most other languages, is an exemplary code in which you can (and should) learn Go and take an example from it.
Earlier, I mentioned “io.Reader is used on several levels” and this is true - for example resp.Body is also io.Reader, but we are not interested in it because we are interested in simulating not a slowed down browser, but a slow connection - then you need to find io.Reader, which reads from the network. And this, looking ahead, is a variable of type net.Conn - which means that we need to redefine it for our custom http client. We can do this with embedding:
type SlowConn struct {
net.Conn // embedding
r slow.SlowReader // in ascii-cast I use io.Reader here, but this one a bit better
}
// SlowConn is also io.Reader!
func (sc SlowConn) Read(p []byte) (int, error) {
return sc.r.Read(p)
}
The most difficult thing here is to still understand a little deeper into the net and net / http packages from the standard library, and correctly create our http.Client using the slow io.Reader. But, as a result of nothing complicated - I hope the logic is visible on the screencast as I look into the code of the standard library.
As a result, the following client is obtained (for real code, it is better to put it in a separate function and comb a little, but for the proof-of-concept example it will do):
client := http.Client{
Transport: &http.Transport{
Dial: func(network, address string) (net.Conn, error) {
conn, err := net.Dial(network, address)
if err != nil {
return nil, err
}
return SlowConn{conn, slow.NewReader(conn, 100)}, nil
},
},
}
Well, now we glue it all together and look at the result: In the end, you can see that the HTTP headers are output to the console normally, and the text, in fact, the page is displayed with doubling each character - this is normal, since we output resp.Body using io.Copy () and at the same time, our slightly modified implementation of SlowReader.Read () displays each character too.
Conclusion
As stated at the beginning of the article, interfaces are an extremely powerful toolkit, and the very idea of separating types for properties and behavior is very correct. But really this power is manifested when the interfaces are really used for other purposes in different libraries. This allows you to combine very different functionality, and use someone else's code for things that the original author might not even have suspected. And it's not just about standard interfaces - within large projects, interfaces provide tremendous flexibility and modularity.
References
Since the idea of this post was impudently pulled from the Francesc Campoy twitter, there is only one link :)
twitter.com/francesc/status/563310996845244416