Article written in February 2018.

Go needs to add package versioning.

More precisely, you need to add the concept of versioning to the Go developers’s working dictionary and tools so that everyone uses the same version numbers when they mention which program to build, run or analyze. The team gomust say exactly which versions of which packages are in a particular assembly.

Version numbering allows you to make reproducible assemblies: if I post the latest version of my program, you will get not only the latest version of my code, but also the exact same versions of all the packages that my code depends on, so that we will create completely equivalent binary files.

Versioning also ensures that tomorrow the program builds exactly the same as it does today. Even if new versions of dependencies are released, gohe will not use them without a special command.

Although you need to add version control, you should not give up the main advantages of the team go: it is simplicity, speed and understandability. Today, many programmers do not pay attention to the versions, and everything works fine. If you make the right model, then programmers will still not pay attention to version numbers, just everything will work better and become clearer. Existing workflows will barely change. The release of new versions is very simple. In general, version control should go by the wayside and not take away the attention of the developer.

In short, you need to add package version control, but not break it go get. In this article, we suggest how to do this, and also demonstrate a prototype that you can try now, and which, I hope, will become the basis for possible integration go. I hope this article will be the beginning of a productive discussion about what works and what doesn't. Based on this discussion, I will make adjustments both to my proposal and to the prototype, and then I will present the official proposal for adding an optional function to Go 1.11.

This proposal retains all the benefits go get, but adds reproducible assemblies, supports semantic version control, eliminates vendorization, removes GOPATH in favor of a project-based workflow and provides a smooth departure fromdepand his predecessors. However, this proposal is still at an early stage. If the details are not correct, we will fix them before the work gets into the main Go distribution.

General situation

Before examining the proposal, let's look at the current situation and how we ended up in it. This section may be a little too big, but history brings important lessons and helps to understand why we want to change something. If this is not interesting to you, you can immediately go to the offer or read the accompanying blog article with an example .

Makefile, goinstallandgo get

In November 2009, the compiler, linker, and several libraries were released with the initial version of Go. To compile and link the programs, it was necessary to run 6gand 6l, and we included sample makefiles in the kit. A minimal shell gobuildcould compile one package and write the corresponding makefile (in most cases). There was no established way to share the code with others. We knew that this was not enough - but we released what we had, planning to develop the rest together with the community.

In February 2010, we proposed goinstall , a simple command to download packages from repositories of version control systems such as Bitbucket and GitHub.Goinstallintroduced conventions on import routes, which are now considered generally accepted. But at that time, no code followed these conventions, goinstallat first it worked only with packages that did not import anything except the standard library. But developers quickly moved to a single agreement that we know today, and the set of published Go packages has grown into a holistic ecosystem.

Goinstall also fixed the Makefiles, and with them the complexity of custom build options. Although it is sometimes inconvenient that package authors cannot generate code during each build, this simplification is incredibly important for users.package: they don’t need to worry about installing the same toolbox that the author used. This simplification is also crucial for the operation of the tools. Makefile is a required step-by-step recipe for compiling a package; and applying another tool like go vetor completion to the same package can be quite difficult. Even getting dependencies correctly, to rebuild packages if necessary and only if necessary, is much more complicated with arbitrary Makefiles. Although at that time some people objected that they were being deprived of flexibility, but looking back, it becomes clear that abandoning the Makefile was the right step: the benefits far outweigh the inconvenience.

In December 2011, in preparation for Go 1, we introduced the go team , which replacedgoinstallon go get.

In general, he go getintroduced significant changes: he allowed Go developers to exchange source code and use each other's work. He also isolated parts of the build system within the team go, so that significant automation with the help of tools became possible. But the go getconcept of version control is lacking. In the very first discussions of goinstall, it became clear: you need to do something with version control. Unfortunately, it was not clear what exactly to do. At least we in the Go team did not understand this clearly. Whengo getIf it requests a package, it always receives the latest copy, delegating the download and update operations to a version control system such as Git or Mercurial. Such "blind work" has led to at least two significant flaws.

Versioning and API Stability

The first significant drawback go getis that without the concept of version control, it cannot tell the user anything about what changes to expect in this update.

In November 2013, a version of Go 1.2 added a FAQ entry with such advice regarding versioning (the text has not changed to version Go 1.10):

Packages for general use should maintain backward compatibility as they evolve. Go 1 compatibility recommendations are relevant here : do not delete exported names, encourage tagging of compound literals, and so on. If new functionality is required, add a new name instead of changing the old one. In the case of a fundamental change, create a new package with a new import path.

In March 2014, Gustavo Niemeyer launched gopkg.in under the guise of "stable APIs for the Go language." The domain - GitHub-redirect considering version allows you to import a way kind gopkg.in/yaml.v1and gopkg.in/yaml.v2for different commit (perhaps in different branches) a Git repository. According to semantic versioning, authors should, when making critical changes, release a new major version. Thus, later versions of the import path v1replace the previous ones, and v2may give completely different APIs.

In August 2015, Dave Cheney submitted a proposal for semantic versioning. Over the next few months, this caused an interesting discussion: everyone seemed to agree that semantic tagging of versions is a great idea, but no one knew how the tools should work with these versions.

Any arguments for semantic versioning will inevitably be criticized with reference to Hyrum's law :

The contract of your API becomes unimportant with a sufficient number of users. Someone depends on any observed behavior of the system.

Although Hyrum's law is empirically correct, semantic version control is still a useful way of generating expectations about the relationship between releases. An upgrade from 1.2.3 to 1.2.4 should not break your code, and an upgrade from 1.2.3 to 2.0.0 may very well. If the code stops working after updating to 1.2.4, then the author will most likely accept a bug report and fix the error in version 1.2.5. If the code stopped working (or even compiled) after updating to 2.0.0, then this change was much more likely to be intentional and, accordingly, it is unlikely that something will be fixed in 2.0.1.

I do not want to conclude from Hiram’s law that semantic versioning is impossible. Instead, I believe that assemblies should be used carefully, using exactly the same versions of each dependency as the author. That is, the default assembly should be as reproducible as possible.

Vending and reproducible assemblies

The second significant drawback go getis that without the concept of version control, the team cannot provide and even express the idea of ​​a reproducible assembly. You cannot be sure that users are compiling the same version of the code dependencies as you. In November 2013, the following FAQ was added to the FAQ for Go 1.2:

If you use an external package and are afraid that it might change unexpectedly, the easiest solution is to copy it to the local repository (this approach is used by Google). Save a copy with a new import path that identifies it as a local copy. For example, you can copy original.com/pkgto you.com/external/original.com/pkg. One tool for this procedure is govenKeith Rerik.

Keith Rarik started this project in March 2012. The utility govencopies the dependency to the local repository and updates all import paths to reflect the new location. Such source code changes are necessary, but unpleasant. They make it difficult to compare and include new copies, and also require updating other copied code using this dependency.

In September 2013, Keith introduced godep , "a new tool for fixing package dependencies." The main achievement godepwas what we now call vendoring, that is, copying dependencies into the project without changing the source files, without direct support for the tools, through a certain configuration of GOPATH.

In October 2014, Keith proposed adding Go to the tools.Support for “external packages” so that tools better understand projects using this convention. By that time, several utilities in style had already appeared godep. Matt Farina published a post “Traveling the Sea of ​​Go Package Managers”, comparing it godepwith newcomers, especially glide.

In April 2015, Dave Cheney introduced gb , “a project-based build tool ... with repeat builds through source vending,” again without rewriting import paths (another motivation for creating gb was to avoid the requirement to store code in specific directories in GOPATH which is not always convenient).

That spring, Jason Buberlie examined the situation with Go package management systems, including multiple duplication of efforts and the vain work on similar utilities. His survey made it clear to developers that support for vending without rewriting import paths must be added to the team go. At the same time, Daniel Theofanes began to prepare specifications for a file format that describes the exact origin and version of the code in the vendor's directory. In June 2015, we accepted Keith’s proposal as an experiment on vending in Go 1.5 , which was included by default in Go 1.6. We encouraged the authors of all vending tools to work with Daniel to adopt a single metadata file format.

Introducing the concept of vending in Go has enabled tools likevetanalyze the programs more competently, and today it has been used by a dozen or two package managers or vending tools. On the other hand, since everyone has different metadata formats, they do not interact and cannot easily exchange dependency information.

More fundamentally, vending is an incomplete solution to the problem of version control. It provides only reproducibility of the assembly, but does not help to understand the package versions and decide which one to use. Package managers like glideand depimplicitly added to the Go concept of version control, setting up a certain way the vendor directory. As a result, many tools in the Go ecosystem may not be able to get the correct version information. It is clear that Go needs direct support for package versions.

Official Package Management Experiment

At GopherCon 2016 on Hack Day (now Community Day), a group of Go activists gathered to broadly discuss package management issues . One of the results was the formation of a committee and an advisory group to conduct a range of activities with the goal of creating a new package management tool . The idea was to have a unified tool replace existing ones, although it would still be implemented outside Go's direct toolkit using vendor catalogs. The committee included Andrew Gerrand, Ed Muller, Jesse Frazel and Sam Boyer, led by Peter Burgon. They prepared a draft specification , and then Sam and his assistants implemented dep . For an understanding of the general situation, see Sam's February 2016 article.“So you want to write a package manager,” his December 2016 post “Dependency Management Saga at Go,” and his July 2017 appearance at GopherCon, “A New Era of Package Management at Go . ”

Depperforms many tasks: this is an important improvement over current practices. This is an important step towards a future solution, and at the same time an experiment - we call it an “official experiment” - which helps us better understand the needs of developers. But depit is not a direct prototype of the possible integration of teamsgointo package versioning. This is a powerful, flexible, almost universal way to explore the space of design decisions. It is similar to the makefiles we fought at the very beginning. But as soon as we better understand the space of design decisions and be able to narrow it down to a few key functions that should be supported, this will help the Go ecosystem to remove other functions, reduce expressiveness, and adopt binding conventions that make Go code bases more consistent and easier to understand.

This article is the beginning of the next step after dep: the first prototype of the final integration with the team go, the package equivalent goinstall. A prototype is a separate team that we call vgo: replacementgowith support for package versioning. This is a new experiment, and we will see what comes of it. As well as during the announcement goinstall, some projects and code are now compatible with vgo, while others need changes. We will remove some control and expressiveness, just as the makefiles were removed in order to simplify the system and eliminate the complexity for users. Most importantly, we are looking for pioneers who will help experiment with vgoto get as many reviews as possible.

Starting an experiment with vgodoes not mean ending support dep: it will remain available until we achieve full and generally accessible integration with go. We will also try to make the final transition from depintegration withgoas smooth as possible, in whatever form this integration takes place. Projects that have not yet been converted to depcan still benefit from this conversion (note that godepthey have glidestopped active development and encouraged migration to dep). Perhaps some projects will want to switch immediately to vgoif this meets their needs.

Sentence

The proposal to add version control to the team goconsists of four steps. First, accept the import compatibility rule , which is indicated by the FAQ and gopkg.in: newer versions of the package with the specified import path must be backward compatible with older versions. Secondly, adopt a simple new algorithm, known as choosing the minimum version to determine which versions of the package are used in this assembly. Third, introduce the concept of the Go module : groups of packages that are versioned as a whole and declare the minimum requirements that must be satisfied by their dependencies. Fourth, determine how to embed it all in an existing teamgoso that the main work processes do not change significantly from today. In the rest of this article, we look at each of these steps. They are discussed in more detail in other blog articles .

Import Compatibility Rule

The main problem with package management systems is attempts to resolve incompatibilities. For example, most systems allow package B to declare that it needs package D of version 6 or later, and then allow package C to declare that it requires D version 2, 3, or 4, but not version 5 or later. Thus, if you want to use B and C in your package, then you are out of luck: you cannot select any version of D that satisfies both conditions, and you can do nothing.

Instead of a system that inevitably blocks the assembly of large programs, our proposal introduces an import compatibility rule for package authors:

If the old and new packages have the same import path, the new package must be backward compatible with the old package.

The rule repeats the FAQ mentioned earlier. That text ended with the words: "In the event of a radical change, create a new package with a new import path." Today, for such a major change, developers are counting on semantic version control, so we integrate it into our proposal. In particular, the number of the second and subsequent major versions can be directly included in the path:

import "github.com/go-yaml/yaml/v2"

In semantic version control, version 2.0.0 means a radical change, so a new package is created with a new import path. Since each major version has a different import path, a specific Go executable may contain one of the major versions. This is expected and desirable. Such a system supports the assembly of programs and allows parts of a very large program to independently upgrade from v1 to v2 independently.

The authors' compliance with the import compatibility rule eliminates attempts to resolve incompatibilities by exponentially simplifying the overall system and reducing the fragmentation of the package ecosystem. Of course, in practice, despite all the efforts of the authors, updates within the same main version sometimes break user packages. Therefore, you should not update too often. This brings us to the next step.

Minimum Version Selection

Today, almost all package managers, including depand cargo, use the latest allowed version of packages in the assembly . I believe this default behavior is wrong for two reasons. Firstly, the number of the “last authorized version” may change due to external events, namely due to the publication of new versions. Maybe someone tonight will introduce a new version of some dependency, and tomorrow the same sequence of commands that you executed today will give a different result. Secondly, to override this default, developers spend their time pointing to the package manager "No, it is not necessary to use X», and then the package manager takes the time to find a way not to use the X .

Our proposal uses a different approach, which I call the selection of the minimum version . By default, the oldest permitted version of each package is used. This decision will not change tomorrow, because it is impossible to publish an older version. Even better, it’s trivial for a package manager to determine which version to use. I call this the choice of the minimum version, because the selected version numbers are minimal, and also because the system as a whole is probably also minimal, avoiding almost all the complexity of existing systems.

Choosing the minimum version allows modules to specify only the minimum requirements for dependencies. These are clearly defined, unique answers for both updates and downgrade operations, and these operations are really effective. This principle allows the author of the entire module to indicate the version of the dependencies that he wants to exclude, or to indicate the replacement of a specific dependency with its fork, which is either located in local storage or published as a separate module. These exceptions and replacements do not apply when a module is built as a dependency of some other module. This gives users complete control over how their own programs are assembled, but not over strangers.

Selecting the minimum version provides reproducible assemblies by default without a lock file.

Import compatibility is key to making the minimum version easy to select. Users can no longer say “no, this is too new a version”, they can only say “no, it is too old”. In this case, the solution is clear: use the (minimum) newer version. And newer versions by convention are acceptable replacements for older ones.

Defining Go Modules

A Go module is a collection of packages with a common import path prefix known as a module path. A module is a version control unit, and versions are written as semantic strings. When developing with Git, developers define a new semantic version of the module by adding a tag to the Git repository of the module. Although it is strongly recommended that you specify semantic versions, links to specific commits are also supported.

In the new file, the go.modmodule defines the minimum version requirements for other modules on which it depends. For example, here is a simple file go.mod:

// My hello, world.
module "rsc.io/hello"
require (
	"golang.org/x/text" v0.0.0-20180208041248-4e4a3210bb54
	"rsc.io/quote" v1.5.2
)

This file defines a module that is identified by the path rsc.io/hello, and it depends on two other modules: golang.org/x/textand rsc.io/quote. The module assembly itself will always use certain versions of the necessary dependencies listed in the file go.mod. As part of a larger assembly, it can only use a newer version if some other part of the assembly requires it.

Authors mark their releases with semantic versions, and vgorecommends using marked versions, rather than arbitrary commits. The module rsc.io/quotethat comes with github.com/rsc/quotehas marked versions, including 1.5.2. However, the module has no golang.org/x/textflagged versions yet. To name untagged commits, the pseudo version v0.0.0-yyyymmddhhmmss-commitdefines a specific commit on a given date. In semantic versioning, this line corresponds to the v0.0.0 prerelease with the identifier yyyymmddhhmmss-commit . The semantic rules of version precedence recognize such pre-releases as earlier than version v0.0.0, and perform string comparisons. The date order in the pseudo version ensures that the string comparison matches the date comparison.

In addition to these requirements, files go.modmay indicate the exceptions and replacements mentioned in the previous section, but again they only apply when building an isolated module, and not when building as part of a larger program. All this is demonstrated in the examples .

Goinstalland oldgo getversion control tools, such as gitand hg, are called to download the code , which leads to many problems, including fragmentation. For example, users without bzrcannot download the code from the Bazaar repositories. Unlike this system, Go modules are always issued via HTTP in the form of zip archives. Previously, there go getwere special teams for popular code hosting sites. Now they have vgospecial API procedures for receiving archives from these sites.

The uniform presentation of modules in the form of zip archives allows trivially implementing a protocol and a proxy server for loading modules. Companies and individual users have different reasons for starting such proxy servers, including security and the desire to work with cached copies in case of originals being deleted. With a proxy in place, to ensure accessibility and go.modto determine which code to use, vendor directories are no longer needed.

Command go

To work with modules, the command goneeds to be updated. One of the major changes is that the usual build commands such as go build, go install, go runand go test, will allow for new dependencies demand. To use golang.org/x/textin a completely new module, just add the import to the Go source code and build.

But the most significant change is farewell to GOPATH as a place to write code. Since the file go.modincludes the full path to the module, and also determines the version of each dependency used, the directory with the file go.modmarks the root of the directory tree, which serves as a stand-alone workspace, separately from any other such directories. Now you're just doing git clone,cd, and start writing. Everywhere. No gopath.

What's next?

I also published a Go Versioning Tour with a demo of how to use it vgo. That article tells you how to download and start using today vgo. Other information in other articles . I will be glad to comment.

Please try vgo . Start tagging versions in repositories with semantic tags. Create files go.mod. Note that if an empty file in the repository go.mod, but there are dep, glide, glock, godep, godeps, govend, govendoror the configuration file gvt, then vgouses them to populate the file go.mod.

I am glad that Go is taking this long overdue step in version support. Some of the most common problems faced by Go developers are the lack of reproducible builds, the complete ignoring of release tags from the outside go get, the inability of GOPATH to recognize different versions of the package, and the inability to work in directories outside of GOPATH. The design offered here eliminates all these problems and more.

But I’m probably mistaken in some details. I hope readers can help improve it by testing the prototype vgoand participating in productive discussions. I would like Go 1.11 to come with preliminary support for Go modules, as a kind of demo, and then Go 1.12 came out with official support. In later versions, we will remove support for the old, non -versiongo get. But this is an aggressive plan, and if for the correct functionality you need to wait for later releases, so be it.

I am very concerned about the transition from the old go getand countless vending tools to the new modular system. This process is just as important to me as the right functionality. If a successful transition means waiting for later releases, then again, so be it.