Executing SSH commands on hundreds of servers using Go

What is the article about

In this article, we will write a simple Go program (in 100 lines) that can execute commands through the SSH protocol on hundreds of servers, doing this quite efficiently. The program will be implemented using go.crypto / ssh - the implementation of the SSH protocol by the authors of Go.

A more “advanced” version of the program written in this article is available on a github called GoSSHa (Go SSH agent).

Introduction

The company I work with has a little more than 1 server, and to work effectively with our number of servers using the SSH protocol, the libpssh library based on libssh2 was written . This library was written in C using libevent many years ago, and still does its job well, but is very difficult to maintain. Also, the Go language from Google began to gain popularity, including within our company, so I decided to try to replace libpssh with Go and fix some of its shortcomings, at the same time greatly simplifying the code and the complexity of the support.

To get started, we need a Go compiler (available for download at golang.org ) and a working hg command to download go.crypto / ssh using “go get”.

Beginning of work

Create the “main.go” file in some directory, preferably empty. Now let's write the “framework” of our program, and then we will implement the missing functions during the course of the article:

package main
import (
	"code.google.com/p/go.crypto/ssh"
	// ...
)
// ...
func main() {
	cmd := os.Args[1] // первый аргумент - команда, которую мы исполним на всех серверах
	hosts := os.Args[2:] // остальные аргументы (начиная со второго) - список серверов
	results := make(chan string, 10) // будем записывать результаты в буферизированный канал строк
	timeout := time.After(5 * time.Second) // через 5 секунд в канал timeout придет сообщение
	// инициализируем структуру с конфигурацией для пакета ssh. Функцию makeKeyring() напишем позднее
	config := &ssh.ClientConfig{
		User: os.Getenv("LOGNAME"),
		Auth: []ssh.ClientAuth{makeKeyring()},
	}
	// запустим по одной goroutine (легковесный аналог OS thread) на сервер, функцию executeCmd() напишем позднее
	for _, hostname := range hosts {
		go func(hostname string) {
			results <- executeCmd(cmd, hostname, config)
		}(hostname)
	}
	// соберем результаты со всех серверов, или напишем "Timed out", если общее время исполнения истекло
	for i := 0; i < len(hosts); i++ {
		select {
		case res := <-results:
			fmt.Print(res)
		case <-timeout:
			fmt.Println("Timed out!")
			return
		}
	}
}

Apart from the fact that we need to write makeKeyring () and executeCmd () functions, our program is ready! Thanks to the “Magic of Go,” we will establish a connection to all servers in parallel and execute the specified command on them, and in any case, end in 5 seconds by printing the results from all the servers that managed to be executed on the screen. Such a simple way to implement a common timeout for all parallel operations is possible due to the concept of channels and the presence of a select construct that allows communication between several channels simultaneously: as soon as at least one of the constructs in a case can be executed, the corresponding code block will be executed.

Initializing data structures for go.crypto / ssh

We have not written makeKeyring () and executeCmd () yet, but most likely you will not see anything very interesting here. We will only be authorized using SSH keys, and we will assume that the keys are located in .ssh / id_rsa or .ssh / id_dsa:

type SignerContainer struct {
	signers []ssh.Signer
}
func (t *SignerContainer) Key(i int) (key ssh.PublicKey, err error) {
	if i >= len(t.signers) {
		return
	}
	key = t.signers[i].PublicKey()
	return
}
func (t *SignerContainer) Sign(i int, rand io.Reader, data []byte) (sig []byte, err error) {
	if i >= len(t.signers) {
		return
	}
	sig, err = t.signers[i].Sign(rand, data)
	return
}
func makeSigner(keyname string) (signer ssh.Signer, err error) {
	fp, err := os.Open(keyname)
	if err != nil {
		return
	}
	defer fp.Close()
	buf, _ := ioutil.ReadAll(fp)
	signer, _ = ssh.ParsePrivateKey(buf)
	return
}
func makeKeyring() ssh.ClientAuth {
	signers := []ssh.Signer{}
	keys := []string{os.Getenv("HOME") + "/.ssh/id_rsa", os.Getenv("HOME") + "/.ssh/id_dsa"}
	for _, keyname := range keys {
		signer, err := makeSigner(keyname)
		if err == nil {
			signers = append(signers, signer)
		}
	}
	return ssh.ClientAuthKeyring(&SignerContainer{signers})
}

As you can see, we are returning the ssh.ClientAuth interface, which has the necessary methods for authorizing on the server. For brevity, error handling is almost completely absent; in production mode, the amount of code will be 1.5 times larger.

To execute the command on the server, the code is also very trivial (error handling is thrown out for brevity):

func executeCmd(cmd, hostname string, config *ssh.ClientConfig) string {
	conn, _ := ssh.Dial("tcp", hostname+":22", config)
	session, _ := conn.NewSession()
	defer session.Close()
	var stdoutBuf bytes.Buffer
	session.Stdout = &stdoutBuf
	session.Run(cmd)
	return hostname + ": " + stdoutBuf.String()
}

For simplicity and brevity, we always use the current username for authorization on servers, as well as port 22 by default.

Our program is ready! The full source code of the program is under the spoiler:

Launch our application:

$ vim main.go # напишем программу :)
$ go get # скачаем все зависимости
$ time go run main.go 'hostname -f; sleep 4.7' localhost srv1 srv2
localhost: localhost
srv1: srv1
Timed out!
real	0m5.543s

Works! The localhost, srv1 and srv2 servers had only 0.3 seconds to execute all the commands, and the slow srv2 did not succeed. Together with compiling the program “on the fly” from the source, the program took 5.5 seconds to complete, of which 5 seconds was our default timeout to execute the command.

Conclusion

The article was short, but at the same time we wrote a very useful application that can be safely used in production. We tested a more advanced version of this application in the production environment and it showed excellent results.

References:

1. The language of the Go: golang.org
2. Library go.crypto : code.google.com/p/go/source/checkout?repo=crypto
3. GoSSHa (the SSH-proxy to communicate with the outside world through the JSON): github. com / YuriyNasretdinov / GoSSHa