Cross-Cloud Programming with Go Cloud

Introduction

Today (approx. Translator 24-07-2018), the Go team at Google released the new Open Source project Go Cloud , a library and tools for developing in the open cloud . With this project, we aim to ensure that developers choose the Go language to create cross-cloud applications.

This post explains why we started this project, the details of how Go Cloud works, and how to get involved and start using it.

Why cross-cloud programming? Why now?

According to our estimates, there are more than one million developers in the world using Go. Go serves many of the most critical cloud-oriented projects, including Kubernetes, Istio and Docker. Companies such as Lyft, Capital One, Netflix and many others depend on Go in production. Over the years, we found that developers love using Go for cloud development because of its efficiency, performance, built-in concurrency and low latency.

As part of our work to support the rapid development of Go, we interviewed teams who work with Go and understand how they use the language and how the Go ecosystem can be improved in the future. One of the main voiced themes among many organizations surveyed is the need for cross-cloud development. These teams (respondents) want to be able to deploy their applications in multi-cloud and hybrid cloud environments, and distribute the load between cloud providers without significant changes in the application code.

To achieve this goal, some teams are trying to untie their applications from a provider-specific API in order to write simpler and more portable (cross-cloud) code. However, short-term functional delivery requirements mean that teams have to sacrifice long-term cross-cloud requirements. As a result, most Go applications running in the cloud are tightly tied to the initially selected cloud provider.

Alternatively, teams can use Go Cloud, a set of open public cloud APIs, to program simpler and more portable cloud applications. Go Cloud also represents a foundation for the portable cloud library ecosystem. Go Cloud allows teams to focus on the functionality during application development, while still retaining long-term flexibility to deploy applications in multi-cloud and hybrid-cloud architectures. Go Cloud applications can also be migrated to the cloud provider that best meets the needs of the application.

What is Go Cloud?

We identified common services used by cloud applications and created a common API to work between cloud providers. Today, Go Cloud can work with a blob repository, MySQL database, runtime settings (configuration), and an HTTP server configured with logging requests, monitoring and health checking. Go Cloud works with the Google Cloud Platform (GCP) and Amazon Web Services (AWS). We will continue to work with partners in the cloud industry and the Go community to add support for other cloud providers in the near future.

Go Cloud aims to develop a vendor-neutral general API for most of the services used, for example, a simple and easy deployment of Go applications to another cloud. Go Cloud can also be used as the basis for developing other open source libraries for work between cloud providers. Feedback from all types of developers and at all levels of development will influence the priority of implementing / adding this or that functionality in future versions of the Go Cloud API.

How it works?

Go Cloud is based on a set of common APIs for cross-cloud programming. Let's look at an example of using blob storage. You can use the generic * blob.Storage type to copy a file from a local disk to the cloud. Let's start by opening S3 storage using the bundled s3blob package :

// setupBucket opens an AWS bucket.funcsetupBucket(ctx context.Context)(*blob.Bucket, error) {
    // Obtain AWS credentials.
    sess, err := session.NewSession(&aws.Config{
        Region: aws.String("us-east-2"),
    })
    if err != nil {
        returnnil, err
    }
    // Open a handle to s3://go-cloud-bucket.return s3blob.OpenBucket(ctx, sess, "go-cloud-bucket")
}

From the moment * blob.Bucket appears in the application , you get the opportunity to create * blob.Writer , which in turn implements the io.Writer interface . From now on, an application (program) can use * blob.Writer to write data to the cloud storage, checking that Close does not return an error.

ctx := context.Background()
b, err := setupBucket(ctx)
if err != nil {
    log.Fatalf("Failed to open bucket: %v", err)
}
data, err := ioutil.ReadFile("gopher.png")
if err != nil {
    log.Fatalf("Failed to read file: %v", err)
}
w, err := b.NewWriter(ctx, "gopher.png", nil)
if err != nil {
    log.Fatalf("Failed to obtain writer: %v", err)
}
_, err = w.Write(data)
if err != nil {
    log.Fatalf("Failed to write to bucket: %v", err)
}
if err := w.Close(); err != nil {
    log.Fatalf("Failed to close: %v", err)
}

Note that the logic of working with the storage (bucket) does not refer to the specifics of AWS S3 operation. Go Cloud turns the replacement of cloud storage, essentially a replacement for the function used to open * blob.Bucket . An application can easily switch to using Google Cloud Storage by creating an * blob.Bucket instance using gcsblob.OpenBucket without changing the code that copies the file:

// setupBucket opens a GCS bucket.funcsetupBucket(ctx context.Context)(*blob.Bucket, error) {
    // Open GCS bucket.
    creds, err := gcp.DefaultCredentials(ctx)
    if err != nil {
        returnnil, err
    }
    c, err := gcp.NewHTTPClient(gcp.DefaultTransport(), gcp.CredentialsTokenSource(creds))
    if err != nil {
        returnnil, err
    }
    // Open a handle to gs://go-cloud-bucket.return gcsblob.OpenBucket(ctx, "go-cloud-bucket", c)
}

So far, however, different steps are required to access the storage for different cloud providers, the final type used by the application remains the same * blob.Bucket . Thus, the application code remains isolated from cloud-specific code. To increase compatibility with existing libraries, Go, Go Cloud uses existing interfaces supplied in the standard library Go, such as io.Writer , io.Reader and * sql.DB .

Code required to access cloud services ( setupBucket () functionfrom the example above) follows the following pattern: higher abstractions are constructed using more basic (lower) abstractions. While you can write such code with pens, Go Cloud automates this with Wire , a tool that generates cloud-specific code for you. The Wire documentation explains how to install and use it, and examples show Wire in action.

How to get involved in the project and learn more?

To begin with, we recommend the following guide , and then we recommend to try building the application using Go Cloud yourself. If you are already using AWS or GCP, you can try migrating parts of existing applications to use Go Cloud. If you use other cloud providers or an on-premise subscription service, you can expand Go Cloud to support this by implementing driver interfaces (for example, driver.Bucket ).

We will appreciate any and all sorts of experiences with Go Cloud. Go Cloud development is managed on GitHub. We will be glad to make any contribution to the project, including the pool rekuest. Create issueto tell us what you think should be improved or which APIs the library should support first. To follow updates and news, join the project mailing list .

The project requires that contributors sign the same Contributor License Agreement adopted in the Go project. Read the contributor's manual for more details.

Thank you for your time spent exploring Go Cloud, we are happy to work with you to make the Go language the choice of developers for building cross-cloud (portable) applications.